Masking Unity Sprites with Stencils

The dream of every Flash artist coming to Unity3d art production is to get familiar tools and practices. One of the major workflows in Flash is masking images with other images. This is something that Unity missed outside of UI images. To mask 2D Sprites, it was left to creating custom shaders or downloading tool packages from the asset store.

In this post I am going to explain the use of stencil shaders to create masking effects in Unity sprites. Here is the output we are going to target.

Mask

So what is this stencil buffer?

The stencil buffer is an area in RAM dedicated to store per pixel integer values. In simplest terms, we can write a specific value to this buffer and compare the value in further operations to do what we want. The stencil operations are separated in a “Stencil {}” block within a “Pass {}” block of our shader.

How does this stencil thing works ?

I am trying to write in as simplified terms as possible, so some technical details might be diluted here. We have three necessary tasks to do per stencil operation. Get a Reference value, compare it to the current value in the buffer, perform the task we need to do with this pixel. Here is a sample stencil block.

 

Reference value

The reference value is an integer in the range of 0-255. This value will be compared to the values already in the buffer. If we need to write to the buffer, this is the value that gets written to the buffer. This value should not be confused with any color or alpha information. This is strictly used for stencil operations. This is specified with the “Ref” keyword.

Comparison

This is a very important step in stencil operations. Here we compare our current value to the previous one in the buffer. There are 8 comparisons available to us –

Greater, GEqual, Less, LEqual, Equal, NotEqual, Always, Never.

This comparison gives us a result of pass or fail for the current pixel. Based on the result of the comparison, we perform the next action. This comes after the comp keyword.

Stencil Operation

This is the step where we decide what we want to do with the current pixel. We can keep it on screen, decide to hide it or change the values in the buffer. This is either after the “Pass” or the “Fail” keyword.

Meanwhile back at the ranch ..

If you are still with me on this, we still need to create a masking shader! So let’s take the default Unity Sprite shader and apply our learning to it. Off we go.

SpriteMask

This is the shader that will mask the other sprites. Only the sprites overlapping this sprite will be visible. We need to add the following section to the “Pass {}” block of the shader

This will write the value of 2 in the space that our shader is rendering. The value 2 is arbitrary and can easily be 5,6,10 etc. Our comparison function is “Always”, so the value will be written to the buffer regardless of the current value. Create a new material and assign it to the cloud sprite.

This is the result of our shader

But it looks the same!

Of course it does, since we do not have any other shaders reading the stencil buffer. Now comes the time to do our magic. Copy the previous shader and replace the stencil block with the following code. Create a new material and assign it to the duck sprite.

 

Et Voila ! here is the masking.

Now the duck sprite will only be rendered in the pixels where the stencil buffer has the value of 2. This is the value written by our previous shader.

But it looks to be glitching. The duck is rendered slightly outside the cloud as well!

The reason is that Unity creates a mesh for each sprite that it renders.

In places where the image has no color, the alpha value is still written. This causes the values to be written to the stencil buffer as well. There are a few solutions for this, we can either create custom meshes for sprites, or discard pixels based on some criteria. For this example I am going to use the second approach. Add this section of code to the frag function of our mask shader.

(Since both the if and the discard operations cost GPU cycles, I would prefer to limit the masking effect to custom meshes or straight edges.)

This will skip any pixels with an alpha value below 0.1. Now the mask works better.

So we created a shader for masking Unity sprites ! Tinker with the shader ,change the stencil comparisons, try new stuff. If you get into any difficulties, comment below and I will try to help.

Update

After writing this, I had a few requests to modify the shader so that the mask image is not shown. This is done easily enough by adding one line to the shader set up section. The new shader is added to the Github repo mentioned below with the name – SpriteMask_ColorOff.shader

Here is the modification

The colormask is used to modify any color information that the shader is going to write. By using a value of 0, we cause it to skip writing any color or alpha information to the buffer. Here is the modified output after applying this change.

Image mask without color

Hope this helps to achieve your goals. Please put in your comments below if you need more explanation of these shader. Keep making games awesome !

The three shaders we created are uploaded to my github repo for this purpose.

GIFs captured by the excellent ScreenToGif tool.

Art assets provided by Unity – Unity 2D asset Pack.

 

 

8 thoughts on “Masking Unity Sprites with Stencils”

  1. Hello!
    Thanks for great tutorial!
    Since I never tried to write shaders so I need some little help to change the script for my case.
    In my case I want the sprite to be transparent only when it gets into the mask (the inverse effect), and also I will use only a square form for mask and the sprites which will be hidden will not have any semi-transparent pixels, so I don’t want this to be verified in shader, since it costs GPU.

    Any help appreciated!!
    Thanks!!!

    1. Hi Gleb, thanks for your comment. If you want an inverse mask effect, you may just replace the line
      Comp equal
      to
      Comp notEqual
      This will cause the sprite to only be rendered outside the masked area. For your second request you may just remove the discard section altogether. Hope that helps !

  2. The pixel transparency part helped me… I applied this so instead of being masked, it creates a hole on the cloud when the duck makes contact, and if you reduce transparency the cloud generates a hole with the duck’s form, but how can I make it so the duck its at half and it is only visible when hitting the cloud and creates the hole at the same time, do i need to passes? one that renders the hole and other that renders the duck inside the cloud?

Leave a Reply

Your email address will not be published. Required fields are marked *