A quick tessellation trick

4 minute read

Let’s talk a little bit about tessellation in Unreal Engine 4. I’ve been experimenting with this recently because the usual techniques involving normal maps to add surface detail to an object simply don’t hold up well in virtual reality, and my development is focused on VR. What tessellation does is to subdivide a 3D model’s geometry to add more detail, and it does this entirely on the GPU. Consider this standard template cube model that comes with the engine:

With tessellation enabled, those faces will be subdivided dynamically, based on the tessellation multiplier, like so:

Tessellated Cube

Now, the most useful aspect of tessellation is that you can use that dynamically created geometry and offset it from the base mesh, creating not just the illusion of detail as you would normally do with normal maps, but add actual geometric detail. This is done through the use of height maps, monochrome textures that tell the renderer which parts of the texture are elevated and which ones are not. The tradeoff for this is performance – the added tessellation detail comes at a rendering cost, and you should always take care to check whether a given mesh really needs tessellation or not – so don’t simply use it everywhere. Older graphics cards without full DirectX 11 support might not even support it, so take that into account as well.

So, with that disclaimer out of the way, let’s set up a tessellation material! It’s pretty straightforward: the first thing we have to do is to enable tessellation in the material properties, by setting the D3D11 Tessellation mode to either Flat Tessellation or PN Triangles. The difference is explained here. That will enable the World Displacement and Tessellation Multiplier output pins in the material, and allow you to connect things to it. Here’s a very basic tessellation material setup:



The way this works is that for every pixel this shader renders, it takes the mesh’s vertex normal which, in case you don’t know, is a normal vector pointing outwards from the surfaces of the mesh, linearly interpolated between mesh vertices according to smoothing groups and face orientation. It then uses a height map aligned to the mesh’s UV coordinates, multiplied by the Tessellation Height Factor, which basically just tells the shader how far in world units the surface should be displaced from their original position, and scales the normal vector by that amount.

Now, the problem with this simple approach is that it only displaces a surface outward, so the more you increase the Height Factor, the bigger your mesh’s silhouette gets.


As you can imagine, this complicates things. One of the problems with this setup is that since tessellation doesn’t apply to collision, even if you have complex per-poly collision enabled for your mesh, it’s possible for the player’s viewpoint to clip into the mesh if it is heavily tessellated.

Thankfully there is a simple workaround to adjust that, which only involves a tiny little bit of maths. What we did before simply adds a positive offset to the mesh’s surface. Instead, we should average it so that the displacement goes inwards and outwards in equal amounts. This is quite simple! Just replace this part of your material:

Standard setup

…with this:

Adjusted setup

In the example above, we add anything between 0-50 displacement to the mesh, depending on whether the height map is more black (=0) or more white (=1), which then gets multiplied by 50. To correct for this, in the second example we simply subtract 25 from the result (the “Tessellation Scale” parameter divided by 2), before multiplying it with the vertex normals. That means that now, the blacks in our heightmap are at -25, and the whites are at +25 offset from the mesh. Simple, right? 🙂

Here’s what the adjusted setup looks like compared to the unadjusted one. As you can see, the mesh is displaced both inwards and outwards:

Don’t get confused by the huge Tessellation Height Factors, that’s one huge mesh I’m using there.

Alright, that’s it for this simple technique. Hope you learned something! Feel free to contact me on Twitter or comment on this post if you have any questions or suggestions for other tutorials you would like to see!