Cell Shading

In 3D graphics cell shading (also called cartoon shading or toon shading) means a material / rendering method that produces to a result that looks like a hand drawn picture. The purpose of cell shading is to create a rendering that looks like cartoon or comic book style.

In this cell shading tutorial I cover a method of doing cell shading in 3DS MAX with the scanline renderer. Of course cell shading can be done by using Mental Ray's Ink'n Paint material but in this tutorial I concentrate on a method that is very fast and works with both scanline renderer and Mental Ray.

Default Rendering in 3DS MAX

In picture 1 you see a standard rendering in 3D Studio MAX. All material settings have default values except for the diffuse value which has been adjusted to get some color. The model is illuminated with one Omni light. So if you are trying cell shading according to this tutorial, please do the following preparations:

1. Open a simple 3D model
2. Apply materials (adjust only the diffuse color)
3. Create one Omni light with default settings
4. Change background color to pure white

Remove shading

The first thing to do is to get rid of the shading. At the moment the model is clearly shaded. There are a lot of color variation that depicts the levels of darkness. Shading can be removed from material by changing Self-Illumination to 100 (Blinn Basic Parameters > Self Illumination: 100). The result is shown in Picture 2. (If you are familiar with 3DS MAX you might wonder why we even created the Omni light. At the moment the light has no effect at all but in the end it will have.)

Add Composite Map

We will be using two separate maps to achieve cartoon rendering:

1. Fallof Map for the cell shading
2. Mask Map for the black edges

We will use Composite map to combine these two maps into one material. So first add Composite as Diffuse Color Map according to picture 3.

Add Cell Shading

The actual cell shading is achieved by using Falloff map. Click on Composite map and add Falloff as Map 1 according to picture 4. Click on Falloff map and make the following adjustments to it:

A Select colors for your model (picture 5). The upper color is the color on the shadow side and the lower color is the color on the side of the light source. So the both colors should have the same hue but upper color should be darker.

B Change the Falloff Type to Shadow / Light (picture 5)

C Adjust the Mix Curve according to picture 6. Add two new control points to the curve and place them according to the picture. (The staircase falloff curve combined with the Self Illumination is the key to cell shaded graphics.)

Now is a good time to make a test rendering with the cartoon shader. At the moment the cell shading should look like in picture 7.

Cell Shading with Borders

Add Mask

Let's add black edges to our cell shader. Go back to the Composite Map and add Mask as Map 2 according to picture 8. (In rendering the Map 2 is displayed on top of Map 1. The purpose of Map 2 is only to produce back edges. Mask is needed because we want to show Map 2 only partly (edges)).

Add Noise Map

Next click on Mask. There are two slots: One for Map and one for Mask. First we need to place black color to the Map slot. Let's do that with a Noise map. Choose Noise as Map according to picture 9.

Click on Noise and change the white color to black according to picture 10. This way the output of this Noise map is just black color. (We could have achieved the same output also with several other maps)

Add Falloff Map

Go back to Mask and add Falloff as Mask according to picture 11. We'll use Falloff map to mask unwanted parts away.

Click on Falloff and adjust the Mix Curve according to picture 12. Add two new control points and place them according to the picture. (The placement of the stair controls the thickness of the black edges.)

Cell Shaded Rendering

Render to test your cartoon shader. The rendering style should look similar to picture 13. If you render a large picture you will see many jagged edges. To fix this just turn Super Sampling on. If you gave several materials you might want do that globally in Renderer's settings. Picture 13 is rendered with Super Sampling. (The effect of Super Sampling is hardly visible in such a small rendering but in large renderings the effect is clearly visible).

Cell Shading With Three Colors

Usually two color shades is suitable for cell shading but sometimes you might want more. If you need more color shades just add one or more steps to the Falloff Map's Mix Curve. Just make sure you adjust the right Falloff map. It's the one with color.