Update to upstream tmux

[WSLUser]

I'm unable to resolve the conflicts myself but thought I'd throw this here as I'm interested in using sixel-tmux but not interested in the rest of tmux being severely outdated. Please keep this fork updated from upstream. Thanks for making Sixel work with tmux and introducing a fall-back.

[csdvrx]

First, thanks a lot for appreciating this work and the effort it took!

Keeping this sixel-tmux "updated from upstream" is just not possible for me in real time - only maybe, say, once every year?

Because given the active development (I see you want to merge 1519 commits), what you are asking for would put on me the responsibility and the burden of resolving all the cosmetic fixes every week... on my free time... and that's not really possible, sorry.

Still, if you believe that in sixel-tmux the rest of tmux is "severely outdated", and if you've found incompatibilities, ​please tell me which important feature you are missing, and I'll do my best to cherry pick and backport it. Depending on the free time I have, I may even do a few extras.

If you want both every single new feature + sixel support, it would seem to me much more logical to integrate sixel-support upstream : it is just a few changes, with almost 0 consequences for anything else (including memory use) since the sixels are not kept as large blobs, but converted (with derasterize) to be stored as text in memory, and used as such in any situation where we can no longer pass through the sixels as-is (ex: after scrolling).

You may want to ask the tmux maintainer, however, don't get your hopes up: the decision to fork was made after noticing a few disturbing things: I invite you to read my https://github.com/csdvrx/sixel-tmux/blob/main/RANTS.md and especially this comment by the maintainer of tmux: tmux#44 (comment)

"I am not interested in graphics in the terminal and there are few practical uses for them anyway. Even if you come up with a diff, unless it is very small it would be very difficult to convince me it'd be worth it. But I wouldn't discourage you from working on it, just don't expect it to end up in tmux, it'd have to be a fork."

This fork is now several years old: check the history to see who else has put some work into it, for a few years...

Maybe I'm reading too much into that situation or @nicm comment, but it seems to me that quite a few people have an axe to grind against graphics in the terminal in general, or against sixel support in particular.

It's a war that I can't fight alone, so sixel-tmux was made instead as a work around the current situation, so that whatever official maintainers try to do (like holding users of VTE-based terminals hostage for over 6 years now, by delaying again and again sixel support), people would have a working solution with sixels shown regardless of the terminal they use.

I would love to create a community around sixel-tmux, to not just play catch up with tmux by backporting its features every year, but to innovate, by supporting say iterm and kitty formats too. I've tried asking @yatli, @saitoha, @HPA ... but nothing has happened so far.

There is a very small community working very hard to make sixel support mainstream, unfortunately, for reason I can't really understand, it seems we've had far more luck with software like mlterm or mintty than with the mainstream Linux community. You can thank the excellent @ThomasDickey that you have at least xterm as an option!!

This is one of the many reasons I have strong hopes in Windows Terminal: I believe WT will have a working sixel support in the default (stable) branch way before that happens in VTE / gnome-terminal or whatever becomes the new default on most Linux desktops.

Ubuntu running on WSL2 with Windows Terminal will then provide the best terminal emulation... which I find quite funny (-:

[csdvrx]

Anyway, if you or someone else want to do the backport, please do, I'm very interested in starting a sixel community around sixel-tmux, and this is the kind of action (code) that speaks louder than words!

[WSLUser]

I think the biggest hurdle is the fact it's over a year old. Conflicts can typically be handled quickly and easily by those experienced with the code if done right away. Unfortunately there's been alot of code creep so now it appears as a daunting project. Hopefully you or somebody else can figure out what exact conflicts need to be resolved to get this fork updated. I too would love to see this project thrive with a community and does end up using other graphic protocols too. @christianparpart is actually trying to create the "Good Image Protocol" laid out by egmontkob in the Terminal Working Group specs. I still think doing some adaption to Kitty's protocol would be better than trying to create yet another protocol but it's his choice and time to use.

[christianparpart]

@christianparpart is actually trying to create the "Good Image Protocol" laid out by egmontkob in the Terminal Working Group specs. I still think doing some adaption to Kitty's protocol would be better than trying to create yet another protocol but it's his choice and time to use.

Hey. I first finish the unicode VT spec. Then i resume. Kitty's image protocol, phew. It is clear that we need something to satisfy today's interests. That won't be sixel and i don't see kitty there either. But having a closer look, the more modern one share some subset. Getting the biggest subset with a focus on simplicity and adaptability alive is my goal here.

[nicm]

My recollection is that last time we (@csdvrx) talked about this, I did a fair bit of work on parsing sixels for tmux and then we discussed how in order to support them fully we would need the ability to convert them into a text representation that could be used on terminals without support and when moving them into the history. You said you were going to work on that, but then IIRC never returned with anything suitable to be used in tmux, so after a while I assumed you had lost interest, stopped maintaining the branch and then later deleted it (although I think I still have it somewhere). It now appears you have taken my parsing work and done your own thing, which I suppose is fair enough if you like, but it seems a bit counterproductive to still be quoting comments from 6 years ago at this point...

SIXEL is far from the best image protocol for tmux because it is very slow to send all the image data to the terminal all the time. Ideally a protocol would have these features:

• Cell based, so size and position is specified in cells;

• Size and position is given by application and resizing, scaling the image or fitting it to cells is the responsibility of the terminal;

• Support for clipping the image (specified in cells) to deal with partially-visible images (scrolled off screen or moved);

• Caching of images, so they can be uploaded once and thereafter referred to by an id.

But I do not know if a perfect protocol will appear and gain widespread support. There is a lot of disagreement about how an image protocol should look and where the various areas of complexity should be, as well as generally how terminals should be developed. I think getting everyone to agree will be next to impossible.

[csdvrx]

Hello @christianparpart

Kitty's image protocol, phew.

1. exists 2) is the 3rd most popular terminal image format IIRC.

It is clear that we need something to satisfy today's interests.

No, it is not clear AT ALL to me. Sixels work beautifully in mintty, with resizing, scrolling, hicolor support etc.

Also, I agree with @HPA technical analysis: there is no clear drawback with ssh compression and given the high quality of libsixel.

That won't be sixel and i don't see kitty there either.

I strongly disagree. Existing formats have been widely tested, supported and implemented.

Before reinventing the wheel, maybe supporting the existing stuff would be a good start? Otherwise, say gnuplot (my #1 usecase) won't be able to plot in your terminal. Maybe it will, in a few years, but why tolerate a regression?

But having a closer look, the more modern one share some subset. Getting the biggest subset with a focus on simplicity and adaptability alive is my goal here.

I mean, why not, but to me, that's NIH syndrome. What's the most likely to happen is that it will add yet another standard: https://xkcd.com/927/

I don't know what to say: "the perfect is the enemy of the good", " nothing is perfect"... yet nothing seems to stick.

Maybe a little pragmatism would help?

I use graphics in my terminal every day for inspecting data, plotting residues for regressions, checking the picture content (lsix), for monitoring remote systems (histograms)...

Let's look at the situation for an average Linux user (say using Ubuntu default software): after @saitoha initial work, 6 years later, there is still 0 support for terminal graphics on the most widely used Linux terminal emulators and software. w3m and libsixel give a minimal experience if you spend some time configuring xterm or use mlterm (the font setup is a bit tricky on both)

What you have on Linux is plenty of discussions and zero tangible results. Meanwhile, I can do about everything on windows, with minimal effort.

At one point, maybe having something to show is better than nothing at all?

Implementing support for the 2 most popular (sixels and iTerm) is on my list, with interconversion so that the format and the terminal is abstracted away. Adding kitty (3rd most popular) will be the next logical step.

And so on, because what matters is USERS and THEIR EXISTING SOFTWARE and what they do with it!

Imagine if gif, jpg and png each had their own incompatible ecosystem, with people saying things like "gif let you make animations! you should support it" "no, it should be jpg because it lets you compress better", "wait you guys are wrong, png is THE format, because it supports transparency", ...

All these arguments are technically true, but the small differences pale in comparison of the common points: all that is now achieved with libraries, file detections, and eventually a drop-down list at the save step. GIMP, Paint, etc. all support every format. You can load AND save in any format - so it turns that into the user choice, even if some format are pushed through as the default for technical reasons: people generally know what they are doing and take the best format for the job.

Now think about how it would be it the situation was like with terminals: you'd have 4 programs each supporting only their own format, unable to import (or display) images without conversion through an external program like ImageMagick... and what's the most common (say, web browsing) wouldn't be possible due to the fragmented ecosystem.

Does it sounds crazy? To me, it is, and unfortunately, that's the current situation. If you want to add a new format, go for it, but maybe you should support the existing first, as ultimately, FORMATS DO NOT MATTER! It's what we do with them!!

[csdvrx]

My recollection is that last time we (@csdvrx) talked about this, I did a fair bit of work on parsing sixels for tmux and then we discussed how in order to support them fully we would need the ability to convert them into a text representation that could be used on terminals without support and when moving them into the history.

In my memory, we indeed had a discussion, but in the end you decided this wasn't going to be in tmux, even if it was done as you describe, and with a compatible license (as tmux is part of BSD), because you didn't like the sixel format.

You said you were going to work on that, but then IIRC never returned with anything suitable to be used in tmux, so after a while I assumed you had lost interest, stopped maintaining the branch and then later deleted it (although I think I still have it somewhere).

I work more slowly now, because I want what I do to be useful / have an impact, instead of just wasting electrons on github.

For example, very high on my TODO list, I have adding sixel support to nnn, my new favorite terminal file manager: 1 or 2 years ago, I carefully discussed everything with the maintainers, because I didn't want to be in situation that sixel-tmux is - a fork. It's still not done. But it will eventually happen.

It now appears you have taken my parsing work and done your own thing, which I suppose is fair enough if you like, but it seems a bit counterproductive to still be quoting comments from 6 years ago at this point...

Please accept my apologies if you have changed your mind.

Would you like me to submit a PR to include sixel support in tmux?

I would prefer you to have a look at the diff: you'll see it's mostly self contained, with little consequences on the rest of tmux (except sixel-tmux tries very hard to avoid invalidating/redrawing the terminal, as it would cause the sixels to be replaced by their derasterized version)

SIXEL is far from the best image protocol for tmux because it is very slow to send all the image data to the terminal all the time.

Modern hardware is crazy fast: I scroll in mintty, with pages of history full of sixels, and do not notice any slowdown!

Also, we're far from being limited to a few MB of memory anymore: most modern systems have a minimum of 4GB, meaning we can really keep a few sixels in RAM for redraw, even without converting them to text: this fallback should be reserved to terminals that don't support sixels. If you're worried about performance, simple techniques (ex: memoize based on the sha256 of the sixel sequence) could help, but I think this would be premature optimization. It's fast. Some people play videos with sixels!

tmux could have a sixel scrollback buffer, with a compile-time option defaulting to a few MB: this way, when changing tabs or scrolling with a terminal that support sixels, it could resubmit the same sixel sequence to the terminal. The derasterized version would only be kept for when the sixel buffer is full or to serve to non sixel terminals.

If you are worried about memory use, only do that for what's currently visible (no sixel history scroll buffer), but again that seems like a premature optimization. Still, a bitmap the size of each tmux tab would cost next to nothing in RAM, while that would enable one big usecase: displaying sixels in different overlapping tabs, and redrawing the full sixel instead of the derasterized version.

The only difficulty I see is 1) keeping the (row,col} and pixel geometry (OSC14t or TIOCGWINSZ) with each sixel in case of change in size (but maybe I'm overthinking it)

But I do not know if a perfect protocol will appear and gain widespread support. There is a lot of disagreement about how an image protocol should look and where the various areas of complexity should be, as well as generally how terminals should be developed. I think getting everyone to agree will be next to impossible.

I fully agree with you: it's not just next to impossible, the last 6 years prove it won't happen at all.

We have to show that formats do not matter: a terminal that implement support for one graphical format will have little difficulty adding more formats. The coexistence of formats on a given page may cause some issues at first, but nothing that can't be resolved with simple rules, say for example, the last picture partially overwrites the others if they overlap?

I mean, for sixel-tmux, decoding iTerm or kitty graphics and storing their derasterized output will be quick ; the difficulty will be in handling the passthrough mode.

As for your 4 points, above asking the terminal geometry in pixels (OSC 14t or TIOCGWINSZ) can let applications create sixels of the right size given the text format, making them fit in the cells, while scaling is the responsibility of the terminal when it detects it's being resized (ex: increasing the font), likewise for clipping when scrolling in the history ; caching is often done in memory (modern systems have plenty!) but some terminals (like mlterm) support caching sixels to disk.

I would encourage you to try mintty on Windows 10 ; it's a pleasure to use with sixels: try to display a few images, configure a large scrollback buffer then scroll: you won't notice slowdowns. Increase the font (like with ctrl + but the default shortcut may be different?) and you will see everything is properly resized with the sixels still aligned to your text!.

Then try it with sixel-tmux and you'll see what a great terminal mintty is!

To be honest, the closest I could achieve on linux with a heavily configured xterm still pales in comparison.

Anyway, if you are here, maybe you are open to discussion?

Then, let's leave the past behind.

Have a look at sixel-tmux features (passthrough and derasterize), check derasterize performance (it can be tuned by changing the set of characters to reduce the exploration step that selects the best glyph and is often the limiting factor) and license (if it is a problem, check basicidea.c which is public domain and just uses halfblocks so it's both simpler and faster)

Can that which already exists become part of tmux?

What would you see in the future?

Ideally, I believe the input formats should not be too restricted (sixel, iterm, kitty... no need to only support one!), the output format should however match what the terminal says it support but should support overrides (as some terminals may have bugs), with derasterize as the ultimate override (for terminals that only support text)

In my vision of the future, a good start could be 2 input formats (I'd suggest adding iterm as many people seem to use MacOS) and 3 output formats (adding basicidea halfblocks for say playing video, which may be too fast for derasterize)

[nicm]

The SIXEL format isn't very good but I don't remember saying it won't be in tmux, although perhaps that is what came across. SIXEL really doesn't work very well with the terminals I have available, even with xterm. I don't see SIXEL in tmux by default but it could certainly be made available under a --enable-sixel flag. The difficulty is in making it work acceptably, there is no point in doing something that doesn't really work.

The way I had seen it moving forward was that we display SIXEL as an image immediately, and then when tmux needs to do anything complex with it (I am not sure what that would cover), it is converted to a text version. Looking back, you had a little program that did this but it needed to be tidied and changed to use tmux's struct sixel_image for input and struct grid as output. It would be better if this was something built in to tmux but if it was to use an external library I could be persuaded, if it is going under a configure flag.

If you link me to the code you are using, I will take a look.

At one point I had tmux using a struct image wrapper to leave open using other formats in future, but I don't remember if I left that in or decided it was unnecessary at this point. I would say one thing at a time, and SIXEL is the only protocol supported by multiple terminals at the moment.

[nicm]

OK I assume you are talking about the code here https://github.com/csdvrx/sixel-tmux/blob/main/sixel.c.

So this seems reasonable enough to me, although I am not going to pretend to fully understand how it works on a quick look. I will try to test it out in the next week.

Some initial comments, mostly about portability and code style:

• Perhaps it would be better converting HLS to RGB in sixel_parse_colour? Or would that lose colour resolution when displaying the SIXEL as image?

• Because the rasterization code is large and self-contained, I would move it into its own file maybe sixel-rasterize.c or sixel-convert.c or something.

• I think UTF-8 character constants (u'█') are too new and begging for portability problems, but it seems like they should be easily replaced by integer constants.

• Likewise I tend to minimize use of wchar_t because it is not well defined. There seems no reason not to simply use uint32_t here since you are bit manipulating the values anyway.

• __int128 is definitely not portable. You will need use uint64_t in some fashion (directly or a wrapper struct).

• TODO and FIXME comments need to be resolved (fixed or removed) as does code that is commented out.

• I would like to see more head-of-file and head-of-function comments explaining what this does, since there is a lot of complex code.

• I am on the fence about use of inline assembly and would be inclined to remove it entirely unless there is a measurable benefit with the size of typical SIXEL images and the frequency tmux will be converting them.

• It needs a bit of tidying to fit with the tmux code style. The brace and indentation is inconsistent; there is use of unsigned (should use u_int, and preferably u_long and u_char as well); {}s around single line compound statements; declarations should go at the head of functions not inside blocks; there is some crazy cpp stuff that needs to be cleaned up (#if -__STRICT_ANSI__ + !!(__GNUC__ + 0) && (__i386__ + __x86_64__ + 0), FLOAT_C(X)); lines are too long; and so on. I will tidy up most issues myself before commit but this needs at least a sweep to get it closer to the right shape. You can see https://man.openbsd.org/style and the other tmux .c files.

[WSLUser]

I do want to bring up that libsixel has been forked and under new maintenance. Many issues have been resolved and slowly getting improved. Here's the new fork https://github.com/libsixel/libsixel

[csdvrx]

I don't see SIXEL in tmux by default but it could certainly be made available under a --enable-sixel flag.

If you mean the sixel passthrough (or the transcoding of other formats like iTerm/kitty into sixel and vice-versa), that would be a very acceptable compromise, as some of that may depend on libsixel.

But could derasterize be a feature that is on by default, after having been carefully tested?

The difficulty is in making it work acceptably, there is no point in doing something that doesn't really work.

sixel-tmux works really well, with no known issue when all you do is convert sixels and never show them as sixels. The feature doesn't depend on any external library or dependency.

If you are concerned about performance, a simpler mode (ex: using only half blocks) could be offered as an alternative, for those who need high performance conversion (ex: to stream sixel videos)

So this seems reasonable enough to me, although I am not going to pretend to fully understand how it works on a quick look. I will try to test it out in the next week.

Great ; please have a look and you'll see it doesn't impact much.

Some initial comments, mostly about portability and code style:
(...)
I will tidy up most issues myself before commit but this needs at least a sweep to get it closer to the right shape. You can see https://man.openbsd.org/style and the other tmux .c files.

All that can be done ; I will post an update when it's ready

[csdvrx]

I do want to bring up that libsixel has been forked and under new maintenance. Many issues have been resolved and slowly getting improved. Here's the new fork https://github.com/libsixel/libsixel

Excellent ; I'll let @nicm see if the license is acceptable to tmux for the --enable-sixel flag

If not maybe @saitoha will be open to relicensing under a tmux-compatible license?

[csdvrx]

Unrelated, but yall may be interested in the HN discussion on https://news.ycombinator.com/item?id=28756701

[christianparpart]

Hello @christianparpart

@csdvrx I know your stance on this, as we both talked about it about a year ago or so. I am very well aware of that. I was just about to reply to the post I was tagged in and won't interfere here any further. Given your very strong feelings and writing style, I am not here to debate anything that is already so heated without even having started. ;)

[csdvrx]

I was just about to reply to the post I was tagged in and won't interfere here any further. Given your very strong feelings and writing style, I am not here to debate anything that is already so heated without even having started. ;)

I'm sorry if my answer came off as heated. Indeed, we had a very interesting private discussion, and I thought I had been able to show you why the few things that could be reproached to sixels ultimately didn't matter much, using mintty behavior as an example: in practice, using sixels, it's possible to built TUIs that will work as well as any GUI tool. Try jexer or see the wonderful examples from https://jexer.sourceforge.io/sixel.html

This is why I was so surprised when I found in the PR this unexpected anti-sixel stance: it was so traditional in its tone and handwaiving of technical issues that I was surprised it didn't come from the VTE team in the first place!!

I'm also very sorry if you felt singled out. This was not my intention. What can I say, I messed up. The current situation is driving me crazy, so much that I see with suspicion brand new HN accounts posting anti-sixel comments (I have showdead enabled). I guess I'm slowly sliding in conspiracy theory mode (but you know, you're not crazy if THEY'RE REALLY THERE TO GET YOU!) seeing anti-sixel people trying to sabotage the efforts everywhere 😅

If you prefer, we can talk in private. You can email me to my nickname followed by outlook dot com.

[WSLUser]

I can also mention that should tmux adopt notcurses like I really hope it does, notcurses already supports Sixel, Iterm2 protocol, and Kitty protocol so tmux could get that support essentially for free and notcurses does provide a fall back for terminals that don't support any of the graphics protocols. What I don't know is your derasterize approach is better, faster, etc. It may be worth your time to evaluate the notcurses code base to see if your fall back method could be added or replace what's already there but from a visual perspective, the current fall-back notcurses uses isn't as noticeable. you can take a look here: https://github.com/dankamongmen/notcurses

Also, Wezterm can be used for testing as it also supports all 3 protocols.

[Bondrake]

I am primarily a Linux user, but off and on find need to make significant use of Windows. I used to lean heavily on PuTTY. In my cases previously it was mostly because I have gotten very proficient at processing CSV type data (on the order of dozens of columns and thousands of rows) in the command line. PuTTY was very fast and made it easy to do this sort of exploratory data munging between *NIX and Windows (e.g. doing a lot of massaging on data before dropping it into an Excel spreadsheet for final presentation, or copying it HTML-color-formatted staight out of the terminal and pasting it directly into an email).

I have recently had need of doing this sort of thing in Windows again and had started to miss emoji-font support and graphics support. I just happened to come across this thread today and @csdvrx pointing out MSYS2/Mintty is a godsend...

A quick pacman -S openssh ssh-pageant-git in MSYS2/mintty had me replacing PuTTY in just a few minutes. The story around graphical emoji seems a bit of a mess though. It isn't able to just pull the color Windows emojis from Segoe UI, and doesn't have color emoji setup in any way by default. You have to manually download a collection of emojis and manually configure it to use them:

svn export https://github.com/iamcal/emoji-data/trunk/img-apple-64 apple
mv ./apple/ ~/.config/mintty/emojis/apple
echo "Emojis=apple" >> /etc/minttyrc

Then that configuration seems to barf and produce weird widths and messed up emojis for a fair few of the emojis in, e.g., emoji-prompt, and I don't have the patience to dig into this further now.

This might seem like a tangential mini-rant in itself, but I think my point from the above ties into the arguments @csdvrx is making around how formats don't matter and people just want features to work sensibly out of the box. Yes, it might be nice to be able to substitute my emoji set in mintty with, e.g. those from Apple, but why not just use the ones already on my system by default? iTerm2 seems to be the only sensible player in this whole game.

Imagine if terminal emulators were web browsers! None of this behavior would fly. People would jump en-masse from browsers which are missing (now) basic features like emoji and image support, or required arcane steps to get them working sensibly.

[Bondrake]

Oh more on point, I whole-heartedly agree with @csdvrx's assessment that tmux shouldn't quietly discard large outputs with no option to fix that. As mentioned above, I often rely on messing with massive piles of text (my go-to scrollback buffer length is around 100k lines) and anything that breaks that is a no-go in my book.

Some sort of low-bandwidth flag or option would make sense for features like that, but definitely not a default behaviour with no option to undo it.

[Bondrake]

I perused notcurses a bit today and now would agree that pushing for it to be integrated in these platforms is the way forward.

[nicm]

If you mean the sixel passthrough (or the transcoding of other formats like iTerm/kitty into sixel and vice-versa), that would be a very acceptable compromise, as some of that may depend on libsixel.

It is not OK to pass through by default because it will make a mess of the terminal as soon as tmux redraws over the top of the SIXEL.

But could derasterize be a feature that is on by default, after having been carefully tested?

Yes probably.

[nicm]

Oh, turns out I didn't actually delete the branches...

[nicm]

I tried to test your sixel-tmux build but it won't build with clang on macOS and after building with gcc-11 it crashes on startup... I'm not sure why because it appears to be in the term lookup code and I don't see any changes in there, but oh well. I'll try again at some point.

I have updated the sixel and sixel-passthrough branches to master:

sixel-passthrough:

I actually don't think we need a configure flag to enable this. I think it would be enough to require the user to configure it with terminal-overrides or terminal-features. tmux no longer requests primary DA so will not automatically detect SIXEL support, so all that is needed in this branch really is a "sixel" terminal features (tty-features.c), if someone wants to do that.

sixel:

• This branch has a lot of code to support multiple image formats and to redraw, scale SIXELs etc (it never converts them to text), but IIRC this was unusably slow and unreliable and I can't remember what state it is in. So it either needs a lot of work to improve that, or to move back to the old way of turning them into text pretty much immediately (which TBH I prefer).

• The code to redraw all SIXELs now does not work after the link between the tty layer and panes was broken for popups, so I have left it commented out. If we wanted to preseve SIXELs rather than converting them then it would need to be fixed.

• We would want to automatically detect SIXEL, so the code to issue and parse primary DA would need to return.

• It would also need a "sixel" terminal feature.

[csdvrx]

What I don't know is your derasterize approach is better, faster, etc. It may be worth your time to evaluate the notcurses code base to see if your fall back method could be added or replace what's already there but from a visual perspective, the current fall-back notcurses uses isn't as noticeable. you can take a look here: https://github.com/dankamongmen/notcurses

derasterize was state-of-the-art ... a few years ago. Chafa (with to the 0x1FBx block from Øyvind Kolås
cf https://hpjansson.org/blag/2021/09/16/chafa-1-8-terminal-graphics-with-a-side-of-everything/) and Notcurses offer interesting approaches, but they may be too CPU intensive.

Using only U+2580 (upper half block) could be a reasonable default, as it should be fast enough to enable even video playback on low-end hardware with just minimal optimization.

I will try to release a newer derasterize to keep up with the competition. The 0x1FBx block ( https://en.wikipedia.org/wiki/Symbols_for_Legacy_Computing ) may be the simplest addition that will provide the most improvements.

@Bondrake I'm happy my suggestion of mintty could help you! It's one of the hidden gems of msys2, so much that even on Windows 11 with wslg, I think I will stick to msys2!

This might seem like a tangential mini-rant in itself, but I think my point from the above ties into the arguments @csdvrx is making around how formats don't matter and people just want features to work sensibly out of the box. Yes, it might be nice to be able to substitute my emoji set in mintty with, e.g. those from Apple, but why not just use the ones already on my system by default? iTerm2 seems to be the only sensible player in this whole game.

Agreed. Your issues with the emoji may seem inconsequential to some people, but I believe they are worth considering, as we should be providing sane defaults for a wonderful "out of the box" experience.

@Bondrake, what would you think of tweaking IOSevka to add the emojis and (if not already present) the 0x1FBx block?

This is a very nice font providing a lot of advanced features (ex: programming ligatures)

[csdvrx]

If you mean the sixel passthrough (or the transcoding of other formats like iTerm/kitty into sixel and vice-versa), that would be a very acceptable compromise, as some of that may depend on libsixel.

It is not OK to pass through by default because it will make a mess of the terminal as soon as tmux redraws over the top of the SIXEL.

If tmux must redraw on top of the sixels, this will not affect a terminal that is not sixel-aware (which currently is most of them) as they won't have displayed the sixels in the first place.

If you are considering sixel-aware terminals will be messed up, yes, the current passthough mode makes intensive use of inhibiting redraws, whenever possible.

A simple solution would be, before redrawing, first overwriting the sixel by another sixel matching the terminal background. This could be easily achieved by just keeping the sixel geometry, and making a sixel of the same size, but without any picture:

• Terminals that are not sixel-aware will not be impacted by either the first write, or the overwrite :)
• Terminals that are sixel aware will display the sixel, until tmux redraws on top of it with that empty pseudo-sixel.

But if we are doing that, at this point, we might as well go the extra mile: it may be worth keeping the intact sixel instead of just making a sixel mask? The only difference would be the eventual "recutting" of the original sixel to match the content of the currently visible part, and the slightly higher memory requirements (as we would keep the sixel and not just its size)

But could derasterize be a feature that is on by default, after having been carefully tested?

Yes probably.

Great! I will try to prepare something well optimized, providing a good balance between visual quality and CPU optimization. My goal would be to allow video playback, without damaging too much the visual quality when displaying static images.

I'll try again at some point.

I don't have a Mac but I can try preparing a binary for Ubuntu 20 (LTS) if it can help?

For the default branch, I will try to clean up the current code. Until the right balance can be found, the existing derasterize should provide a good enough default choice.

move back to the old way of turning them into text pretty much immediately (which TBH I prefer).

I agree that derasterizing images to text is a safe bet and thus a sane default, because most terminals currently in use don't support any image format. But eventually, they will, so maybe it should be tweakable from tmux.conf without requiring a recompilation?

sixel-passthrough:

I actually don't think we need a configure flag to enable this. I think it would be enough to require the user to configure it with terminal-overrides or terminal-features. tmux no longer requests primary DA so will not automatically detect SIXEL support, so all that is needed in this branch really is a "sixel" terminal features (tty-features.c), if someone wants to do that.

Fair enough! And this will indeed allow users to get a sixel experience when changing terminals after a simple tmux.conf tweak.

sixel:

• This branch has a lot of code to support multiple image formats and to redraw, scale SIXELs etc (it never converts them to text), but IIRC this was unusably slow and unreliable and I can't remember what state it is in.

If the sixel branch needs optimization, I can try to ask friends specializing in that?

You seem to be a Mac user, I'm a Windows user, my best friend is a Linux user - that should be enough to ensure tmux is working great on any modern platform.

I'm making the offer, because we seem to be generally in agreement, so if we are indeed on the same page, we can try fixing the sixel branch, with the ambitious goal of having everything working on most terminals - and I mean everything, from derasterize (default) to sixel passthrough (default or tmux.conf tweak) to format independant recoding say for example to convert sixels to something iTerm can display, and vice versa (in the worst case through another tmux.conf tweak if we can't reliably detect terminal features, as it's highly unlikely a given user will have a mix of iTerm and mintty)

I'd just like to be sure that it's ok, because that would be a non-trivial effort that would require a bit of coordination on my side like taking some time off to focus on tmux, same for my friends.

But if it's for the greater good, and can reach every tmux user, I'm fully in! I'd even arrange to borrow a Mac and setup any MacOS version we may want to test given the compilation issues you report.

[nicm]

There is no point in getting complicated with passthough since it would not happen if tmux understood SIXEL itself. It just can't happen by default.

SIXEL is slow because there is no way to move, clip or otherwise modify one without completely resending it to the terminal. It will not be possible to avoid this, particularly with large images.

I would not like to try to do too much at once. If you get a version of your code to convert a SIXEL to text tidied up then I think we can initially go with just converting images to text immediately, so they only last as an image until tmux needs to redraw them. Anything else can be built on that.

Note though that it needs to build and work on all the platforms tmux works on (you don't need to test it on them all but best to avoid anything too new or egregiously unportable).

[WSLUser]

it's highly unlikely a given user will have a mix of iTerm and mintty

I have Windows VM on Mac so I use Iterm for Mac and Windows Terminal (though I do occasionally test out other available terminals like Contour). Sharing the same conf would be the ideal.

[eylles]

kinda off topic

For example, very high on my TODO list, I have adding sixel support to nnn, my new favorite terminal file manager: 1 or 2 years ago, I carefully discussed everything with the maintainers, because I didn't want to be in situation that sixel-tmux is - a fork. It's still not done. But it will eventually happen.

@csdvrx dunno how are you doing with nnn, but if nnn can do a "preview panel" like vifm can do, then perhaps take a look at this repo i started last month, https://github.com/eylles/vifm-sixel-preview it is still under work have been doing some testings and more adding some code from lsix and converting it to bash, i hope pushing the changes in the following days, depending on how previews are done in nnn adding an nnn flag may be doable.

[MarioRicalde]

I loved reading all posts from @csdvrx and @nicm on this subject, so much passion!

It was quite a blessing to just keep on finding little gems here and there as I dove in to figure out how to recommend a feature @nicm would not instantly close (I was a commentor on a feature, wondered if the channel was wrong).

The more I kept on reading the more I got hooked. Interactions between everyone involved were just gold. I am not sure about the state of these pull request or if this is going to get merged ever; however, I'm very thankful with having had the opportunity to read all of this.

Cheers!

[justuswilhelm]

I appreciate the respectful tone. Despite having an appreciation for the immense technical issues of bringing Sixels to tmux, I wish it was there as a 'only use this if you know what you are doing' --flag. Is there any bounty on this issue? The original on the tmux repo looks like it has been locked for a few years.

Honestly it seems like one of those "now that I know about it, I can't believe it's not being used more often" kind of features that would make the open source terminal world look even more attractive to a future generation of developers.