As editor selection effect, I had a white overlay color on all hovered and selected entity , so pretty difficult to see the material beneath. Also, I needed to have outlines for interactive objects in the game, so it was time to add them.

I had multiple constraints as well; respecting the mesh silhouette, being able to merge silhouettes together when more than one is selected, and finally being able to detect which part of the silhouette is occluded, in order to change slightly the color of its outline.

I tried multiple things to get a good result, my first attempt was the “inverted hull” approach where we inflate the mesh by pushing its vertices across the inverted normal axis, then test it against the depth buffer to get what’s left, outlines. The result is far from good, inflating the mesh also dislocate it a bit and the outline width is varying depending of the view angle, not great...

I ended up having two small render passes for outlines, where the first constructs a mask texture which contains the unified shape, and the second one detects and draws edges from it.

For the first pass. I attach a color texture which holds one hardcoded value. The fragment shader writes this value for all passed fragments, so that I’ll get one unified shape. I also attach a depth texture to fill up, this turned out to be very useful later to get which part of the silhouette is occluded.

The second pass is a fullscreen render pass where we sample the two textures from our previous mask pass, and the main scene depth texture. In the fragment shader, we simply detect if our fragment is an edge or not by sampling the surrounding pixels (up/down/left/right) and checking if it’s equal or not to our central current fragment value. If not we’re on the edge, otherwise we discard.

Once we know our fragment is the edge, we can return our outline color and stop there, but we won’t see parts that are occluded. For that, we sample the outline mask depth texture, and our main scene depth texture. If our outline depth value is less than or equal to the main scene one, it’s not occluded. We can base our output color on this information!