Layer masks

[dsizzle]

ArtPaint would be a lot more powerful with layer masks. This entails:

• Grayscale bitmap added to a layer, where black is "hidden" pixels (aka alpha 0) and white is shown (alpha 100%). Rendering would need to apply the mask as it's compositing the layers

• Layer window needs a second thumbnail to show the mask

• Mask can be turned on or off with a checkbox

• Mask needs to be paintable (but only in grayscale) with all tools.

• Mask painting needs two modes: one where the mask takes over the canvas so the user is painting directly on a grayscale image but can't see the effect of the mask:
  

• and one where the mask is not directly visible but the painting shows the effect of the mask immediately:
  

[humdingerb]

I've never really used masks, I have to admit... Only thinking about it abstractly, would it maybe make sense to be able to seemlessly blend between those two modes with an Alpha-like slider? I can imagine that I may want to see the image a little while drawing in the mask.

[humdingerb]

Idea: Could the whole masking business be just a new Blend mode that only works on the layer below it?

WonderBrush does it kinda this way with its modes "Alpha" and "Inverse alpha", only those apply to all layers below it. Which is probably not what's expected from a mask.

[dsizzle]

Yeah, I had thought of that before - like a blend mode of Mask or something, but then if you rearrange layers you have to make sure you move your mask layer too, and like you mentioned does it affect all layers below? Does it mask via alpha of the layer? Might be confusing.

[humdingerb]

Yeah, not totally straight forward...
Maybe if layers had the feature to "link" together (useful for moving around together or a "Merge linked layers"), the Mask blend mode could auto-link the mask layer with the one below. The mask layer could affect all layers linked to it. Masking via alpha of the layer sounds intuitive to me, more so that e.g. black&white. It's just a convention that's easy for the user to learn, similar as rgb(0,0,0,0) or rgb(255,255,255,0) being transparent pixels.

[dsizzle]

I've never really used masks

one of the common uses I’ve seen is to duplicate a layer, apply an effect, and use the mask to only show parts of the layer with the effect. For example, you can take a portrait of someone’s face, duplicate it, sharpen it, and then paint a mask so that only the eyes are sharpened. It makes the portrait “pop” more, and it’s way easier to use a mask vs trying to select the eyes.

Masks are also the easiest way to fade one layer into another, just by slapping a grayscale gradient in a mask on the top layer. Like this wallpaper; the reflection would be a good candidate for a gradient mask:

[humdingerb]

Maybe, once you have something masky to test, I'll have to work through some mask-related online tutorials...

[dsizzle]

The challenge of alpha is that if you think of a mask as a piece of paper you place over a layer, then the cutouts (aka transparent parts) are where you see the layer, and everything else is covered. With an alpha layer you might think the transparent parts of the alpha layer should be the transparent parts of your masked layer, but it’s actually the opposite. Idk, I will think about it more.

and unfortunately I am still super busy with work & life stuff so I added these multiple tickets just to note things I’ve thought about but hadn’t written down yet. I’m not sure when I’ll actually work on this.

[humdingerb]

With an alpha layer you might think the transparent parts of the alpha layer should be the transparent parts of your masked layer, but it’s actually the opposite.

I don't think it matters too much which way around it's done. The user quickly finds out when trying to use it...

Anyway, no hurry. ArtPaint is very usable already!
In case you want to take a break or think RL slows you down too much, we can do another release to at least get the many improvements out there. Or do that anyway... You decide. :)