Rim Shader
When creating expressionistic artwork, artists will only create the broadest strokes of detail. Ambiguous objects allow the viewer to engage the work on a more emotional level as they fill in the gaps within the picture. To recreate this affect using shaders, I adjusted the transparency of objects so that they were only visible towards the edges. This allowed the gestural nature of the objects to be captured without disclosing their entire form.
First, I experimented with extracting the edge of an object. This can be done using the dot product of the normal with the incident ray. When the surface is facing the camera, the value of the dot product will be 1. When the surface is facing perpindicular to the camera, the dot product will be 0. By checking for very small ranges of the dot product, we can extract the edges of an object.
The code for this simple edge detection involves normalizing the surface normal and the incident ray and then generating a dot product and using that in a control structure operated by a smoothstep. Rim_width is a parameter which controls how wide the edge will appear to be.
To make the design more interesting, instead of basing the illumination off of I, the incident vector, we can instead use an arbitrary vector to create the illusion of a light source. Now, the vector is no longer pointing directly towards the camera and the rim has the appearance of being part of a shadowed object.
Next, to make things more interesting, I added some noise to the rim_width parameter. This way, the edges of the filled area appear more ragged and less uniform.
Because each layer is only a single color, currently the rim shader will only produce a monochromatic image. Color would make things more expressive so I decided to add a second layer to the shader containing another rim pattern. This extra rim has its own color and direction vector, allowing for independent positioning. Thus, the artist could use one layer as a fill light and another as a kicker or rim. Using contrasting colors on the two layers also helps better define the shape and imply depth.
Below is the complete shader listing:
