Support non-wlroots compositors using Pipewire

[CorvetteCole]

I know Gnome is not wlroot based (or at least that is what I think), but I was hoping to get a little insight in to what work I'd have to do to get this working with Gnome. I hate the autobrightness on Gnome so that is what prompted this.

I know you can get the screen using pipewire on Gnome, and I think there is an experimental setting for enabling DMA buffer sharing, but I'm not entirely sure how much of the code I would have to change to enable this.

Thanks for any insight in advance. I understand you are probably busy but if you would like to help me implement that it would be greatly appreciated as well :)

[maximbaz]

Hello! It's exciting to hear that you are interested!

To be honest it's hard for me to maintain this in C so I have a half-baked rewrite in Rust, where I want to keep all the parts modular, such that for example ALS sensor is one part (and you can choose whichever implementation fits your device), screen contents detection is another part (which is what you are asking, wlroots implementation will be one, but Gnome could be added too), and finally "ideal brightness" calculation is yet another part.

It's been a while since I touched it, but let me finish it up to a "workable" state and I'll push and ping you and maybe we could try to get this working?

[CorvetteCole]

Sure, sounds good. I love the idea of this project and would be more than willing to help :)

[egormanga]

Me too! Great idea, always wanted something like this. Unfortunately, I got zero experience with Rust, but maybe could help with the Gnome integration part.

[maximbaz]

Hey guys, let's start investigating what needs to be done for Gnome while I finish preparation up.

1. I assume you are using Wayland, not Xorg? The current code is written for Wayland in mind, and although it is possible to add Xorg support, I remember also seeing a bunch of similar projects for X, so it's possible that you can find something decent faster.
2. Getting a frame of the current screen contents - for wlroots I'm using a wlroots-specific API to get DMA-BUF (see all frame_* functions in main.c), so we need to find what is the alternative for that. The idea with PipeWire sounds very cool, to be honest I dont know myself where to look for, but if you want to already start experimenting, take the current C code and what you need to find is a way to rewrite register_frame_listener() function to use PipeWire (on the one hand I know myself, experimenting in C is not easy if you don't know the language already, but on the other hand if there is any kind of API, it's most likely available in C). It better be DMA-BUF, so that memory is not copied anywhere, because doing that for every frame would be laggy and really hard on your battery life. Finally, if all fails, it will also be possible to simply make wluma not react on screen contents (only use ALS sensor like phones do), but in my mind it's not as cool 😄
3. Once frame is available, there is a bunch of code that detects "average color" and the derives "lightness" of it - so a screen with a lot of white color would be considered "light" and a screen with a lot of darker colors would be considered "dark". This is done on GPU using Vulkan API to (again) save battery and have instant reaction time, and I'm pretty pretty sure all this code will be reusable on any platform, as long as you can get a DMA-BUF with an image to feed to Vulkan API.
4. At this point if Gnome can be supported, it probably will be! There are other challenges in this project (supporting different kinds of ALS sensors, supporting laptops with no such sensors, supporting external screens and finally having smarter algorithms that use all of the available data to decide the best possible screen brightness value), but we will tackle them separately as we go 🙂

[egormanga]

1. I'm on Gnome/X11.
2. The key feature & whole point of using wluma is using screen content to determine best perceived brightness, imo.

[maximbaz]

Agree on the second point 🙂 It might be that PipeWire approach would be agnostic of Wayland/X11, but I know too little about it, let's see if you guys are able to find any examples or proof! I'll try to get us some framework done soon, so that it's easier to plug in various implementations, but I'll definitely have to rely on you guys and the community to figure out and contribute alternative implementations like the idea with PipeWire.

[CorvetteCole]

hey there, I've got some answers as well! I use Gnome/Wayland currently. I don't have a ton of experience with the Pipewire API, but I have been looking in to it and totally think it's possible. I'm not sure if Pipewire would be distro agnostic completely, but it would definitely be a step in the right direction. I think we can use DMA-BUF with Pipewire on gnome as an experimental thing, will look in to it!

[maximbaz]

A quick update on my end, I finally managed to finalize the rust rewrite - it's still rough, but it already works. It's pushed to the main branch.

You can see in main.rs an idea of what I have in mind when I say I want to make wluma modular - some things are easy to auto-detect (e.g. a type of ambient light sensor), other things are easier to ask users to define in config.toml.

Specifically to the topic of this issue, you can see that all GPU magics are extracted in vulkan.rs, you only need to feed it a simple struct describing the frame (width, height, and a single dma-buf fd - multiplanar frames are not supported yet), and it will give you a "luminocity percentage" later used in algorithm calculation.

So for pipewire, I would imagine you'd only need to implement an alternative to wlroots.rs - it should implement an event loop (it's really not wlroots specific, we can/should extract it in wayland.rs perhaps), where it calls vulkan::luma_percent() to process the frame and controller::adjust() to actually predict the best brightness and apply it.

Feel free to experiment, let me know how it goes, hopefully you'll have as much fun as I do 😁 PRs are highly appreciated, smaller more frequent PRs are preferred over large ones, as I still haven't exactly hashed out in my head how I want to structure things, so I dont want to introduce too much code conflicts for you.

[CorvetteCole]

Great, thanks! I am in between apartments right now but will be working on this in the coming weeks. Very exciting!

[CorvetteCole]

Update: been taking a look at this recently. Hope to be able to open a PR this weekend with preliminary Pipewire support. It is my understanding that other compositors which implement support for the method pipewire uses to grab screens (all of them?) could use this, so it will definitely need a config option and some documentation so people can choose which works best for their setup

[maximbaz]

Awesome news! Looking forward! It was also my understanding that pipewire support would be compositor-independent... Let's see! As for config, I agree, I was thinking we could use this parameter, the value being wlroots or pipewire:

wluma/config.toml

Line 3 in 0875510

capturer = "wlroots"

Don't polish too much, if you have a rough thing, lets get it looked at to hash out "design/layout" decisions and test, you can then take your time to beautify the code and tell me when it's ready for a final review and merge 🙂

[CorvetteCole]

@maximbaz I'm wondering how we should handle multiple monitors and/or backlights. In particular we will likely request the user to grant access to a monitor to capture screen contents for WLUMA. I'm not sure I can record both at the same time, but am looking in to it. We will probably end up relying on xdg-desktop-portal for this (which already has different implementations for kde, wlroots, and gnome handled)

[maximbaz]

I'd suggest you go with the simplest route for now, whatever it is (take one monitor at random?), and we tackle multiple screens and multiple backlights separately. This problem is also present in wlroots-based code (where I already am taking an output at random 😁 ).

For the proper implementation, I thought we'd let the users configure this, an example which I had in mind can be seen in config.toml: we can define output by name, where (a) if it's defined then we want wluma to control its brightness, (b) per output we configure how it should be controlled (e.g. backlight or ddcutil or whatever), and (c) whether the contents of this screen should be accounted by the algorithm or not (this I didn't completely think through, my idea was that you have a main screen, and contents of it decide brightness of all available screens, but maybe it's not such a good idea).

Maybe we should move this discussion in another thread, brainstorm possibilities, and then implement it similarly for pipewire and wlroots.

Totally ok with depending on xdg-desktop-portal btw.

[CorvetteCole]

(references for me or others):

• https://gitlab.gnome.org/feaneron/obs-xdg-portal/-/blob/master/pipewire.c : plugin for obs that used the exact things we want to use for screen capture
• https://github.com/SeaDve/Kooha : screen recording tool, written in rust and implements what we will

[CorvetteCole]

that link I sent above implements what I will probably implement (or similar). What actually happens is it asks you which screen you would like to record, so the user gets to pick which one brightness is based off of. I don't know if it is important to average multiple monitors, at least on laptops, but on my desktop it would be cool to have a similar setup (both my monitors have their backlights exposed via a ddcci dkms driver).

[CorvetteCole]

I'll build out the basic implementation for one display capture first, then I think it would be a good idea to have a separate issue and think over how this should work long-term. Thanks for being so open to working on this with me!

[CorvetteCole]

p.s. maybe we just capture both monitors as one BIG screen (or combine multiple screens), then feed that in to the rest of the program. Would be a handy config option that would probably solve the above for some people's specific hw setups. I'll look in to that after the basic pipewire implementation

[maximbaz]

My current bet is that averaging would probably work not great (having a dark terminal on one screen and white browser on another would probably set too dim for the screen with terminal and too bright for the screen with browser), but let's leave this problem for the future selves 😁

[CorvetteCole]

I'm BACK, sorry for the delay. Should we still handle multiple monitors the same way? I see now that wluma supports multi-monitor configurations so I'm a little unsure. Thanks!

[maximbaz]

Welcome back 😁

I suggest to start how ever is simpler to get a prototype going, and we will get it polished and properly architected along the way. We did indeed make a lot of progress in terms of proper monitor selection and multi-monitor support in general, as well as prediction itself.

We will not go the road of averaging multiple screens, it produces bad predictions and I spent a significant amount of time trying to make the predictions as good as possible 😅, instead I found that we get the best results by treating each screen independently: each screen has its own training, its own data points (which also makes sense because each screen is just different, 50% brightness on one monitor can look totally different to 50% on another).

So the current architecture is this:

• each output has its own capturer config, which can now be wlroots or none, where you will add pipewire as the third option.
• each output also has a name config, the value is used to pick the correct screen when working with various APIs. You will probably need to make use of it in pipewire as well.

Hope that clarifies some things? And again, dont get too discouraged by the complexity, start simple, lets get it working and then make it pretty 😉

[CorvetteCole]

sure, sounds great. Working through my backlog haha. Finished a major PR elsewhere already so it is go time here. I'll do a little research and hopefully will have some sort of prototype next weekend-ish? don't quote me on that haha

[CorvetteCole]

Wow, I feel terrible seeing this. Update for anybody stumbling upon this: I am SUPER busy. If anyone else has interest in implementing this I welcome it. I will try to get to it eventually™. I apologize!

[egormanga]

@CorvetteCole Don't worry! Everybody have their own lives, it is normal to. This is a pet project, after all.

[CorvetteCole]

Part of the problem with this as I've explored an initial implementation (at least with my approach), is that it involves basically "sharing" the screen to the application. At least in Gnome, this results in a bright yellow "screen is being shared" indicator in the top right which is quite annoying. I am still exploring alternatives

[obj-obj]

Yeah it might be easier (and better) to get dmabuf support on the desktop environments themselves, because screensharing with pipewire will always show that recording indicator

[maximbaz]

A new Wayland protocol has just been merged, that would potentially allow wluma to work on any Wayland desktops, not only wlroots-based ones. This is all very new and exciting and not supported anywhere yet, but subscribe to #121 for the news 🙂

[obj-obj]

Could also just use KMS like https://git.dec05eba.com/gpu-screen-recorder/about/ does. Would require root (or modifying capabilities) though