Unity Tutorial – Tile Blending / Edge Smoothing

In this tutorial we will learn how to create a TileMap in Unity using a custom mesh, define terrain types and blend terrains textures (edge smoothing). You can find the source on GitHub. Please share and consider supporting me on Patreon.

1. Creating a Map of Tiles

Each tile has a terrain type

Setting up map getters and enumerator.

2. Creating a MeshData / Mesh

3. Adding material / shader

5. Blending / Edge Smoothing

Tile neighbours

Creating the new mesh

7. Video Tutorial

6. This tutorial on Github

 

 

1. Creating a Map  of Tiles (or a grid of points)

First let’s create a Tile class, for each tile we just want a poisition and a terrain type (we will just use an enum here).

public enum TerrainType {
	Water, Dirt, Grass, Rocks
}

public class Tile 
{	
  public TerrainType terrainType;
  public Vector2Int position;
  public Map map; // We will create the map class just after this.

  public Tile(Vector2Int position, TerrainType terrainType, Map map) {
    this.position = position;
    this.terrainType = terrainType;
    this.map = map
  }
}

And sometimes it’s anoying to use a Vector2Int for the position, so let’s just add an other constructor with an integer for x and y.

Tile(int x, int y, TerrainType terrainType, Map map) 
: this(new Vector2Int(x, y), terrainType, map) {}

Now let’s just add an empty Map class with a width and height (represented by a Vector2Int width = vector.x and height = vector.y) and an array of tiles.

// Map.cs
public class Map {
  public Tile[] tiles;
  public Vector2Int size;

  public Map(Vector2Int size) {
    this.size = size;
    this.tiles = new Tile[this.size.x * this.size.y];
  }
}

We want to be able to get a tile like this:  Tile tile = map[x, y]; or Tile tile = map[Vector2Int.zero]. Just a quick note here, we are using a one dimentional array and it’s just a personal preference but let’s explain this. For our tile array we want a list of all the tiles in the map, let’s say our map is 3×3 so we want a 1dArray, we can just say the indice is x + y * width.

 

public class Map {
  //...

  public Tile this[int x, int y] {
    get {
      if (x >= 0 && y >= 0 && x < this.size.x && y < this.size.y) {
        return this.tiles[x + y * this.size.x];
      }
      return null;
    }
  }

  public Tile this[Vector2Int v2] {
    get {
      return this[v2.x, v2.y];
    }
  }
}
  

We also want to be able to foreach on our map object like this foreach (Tile tile in map){} so let’s add an enumerator.

public class Map {
  //...
  public IEnumerator<Tile> GetEnumerator() {
    for (int x = 0; x < this.size.x; x++) {
      for (int y = 0; y < this.size.y; y++) {
        yield return this[x, y];
      }
    }
  }
}

Now we need to initialize our tile array, and it would be usefull to be able to set a rectangle in our map to a different terrain type.

public class Map {
  public Map(Vector2Int size) {
    //...
    for (int x = 0; x < this.size.x; x++) {
      for (int y = 0; y < this.size.y; y++) {
        this.tiles[x + y * this.size.x] = new Tile(x, y, TerrainType.Water, this);
      }
    }

    // Let's add some random stuff just to test this out. We will pretend our map is big enouth.
    // Obvisouly in a real game you should be able to load and save this, use procedural generation, etc...
    this[2,2].terrainType = TerrainType.Dirt;
    this.SetRect(5, 5, 12, 12, TerrainType.Dirt);
    this.SetRect(5, 5, 5, 5, TerrainType.Grass);
    this.SetRect(9, 9, 9, 9, TerrainType.Rocks);
  }

  // We just set a rectangle to a terrainType value.
  public void SetRect(int startX, int startY, int width, int height, TerrainType terrainType) {
    for (int x = startX; x < startX+width; x++) {
      for (int y = startY; y < startY+height; y++) {
        this[x, y].terrainType = terrainType;
      }
    }
  }
}

Now we have our map and our tiles but now way to test this because we don’t have a renderer but what we can do is use Gizmos in Unity, so let’s create a MapRenderer class, Add our renderer to a game object and create a new map.

public class MapRenderer : MonoBehaviour
{
  public Map map;
  public bool ready = false;

  public void Start() {
    this.map = new Map(new Vector2Int(40,40));
    this.ready = true;
  }

   void OnDrawGizmosSelected() {
   	if (this.ready) {
     	foreach (Tile tile in this.map) {
     		if (tile.terrainType == TerrainType.Water) {
     			Gizmos.color = Color.blue;
     		} else if (tile.terrainType == TerrainType.Dirt) {
     			Gizmos.color = Color.yellow;
     		} else if (tile.terrainType == TerrainType.Grass) {
     			Gizmos.color = Color.green;
     		} else {
     			Gizmos.color = Color.grey;
     		}
     		// DrawCube(v3 center, v3 size) we position from the center, so we need to add .5f;
     		Gizmos.DrawCube(new Vector3(tile.position.x+.5f, tile.position.y+.5f), new Vector3(1, 1, 0.1f));
     	}
   	}
   }
}

2. Creating a MeshData / Mesh

Now we need to create a mesh

For each tile we will have 4 vertices, and 2 triangles. It’s quite simple you can see on the image the vertices for the tile 0,0, now we just need to to this for every single tile (and add the tile position).

But before that we will create a MeshData class to handle all things related to meshes, this is just a good habit. For example in my game I use some methodes like meshData.Update(flags) (flags can indicate update UV/Colors, etc…) or just some helpers methods like AddTriangle(vIndex, a,b,c).

public class MeshData {
  public List<Vector3> vertices;
  public List<int> triangles;
  public List<Color> colors;
  public Mesh mesh;

  // You can use multiple constructor, for example one to specify the size of each list, or just use arrays in a real game
  // Because we already know our vertice size (4 vertices per tile, so 4*width*height) and the same goes for triangles/indices (6*width*height).
  public MeshData() {
    this.vertices = new List<Vector3>();
    this.triangles = new List<int>();
    this.colors = new List<Color>();
    this.mesh = new Mesh();
  }

  public void AddTriangle(int vi, int a, int b, int c) {
    this.triangles.Add(vi+a);
    this.triangles.Add(vi+b);
    this.triangles.Add(vi+c);
  }

  public void NewMesh() {
    UnityEngine.Object.Destroy(this.mesh);
    this.mesh = new Mesh();
  }

  public void Clear() {
    this.vertices.Clear();
    this.triangles.Clear();
    this.colors.Clear();
    this.NewMesh();
  }

  public void Build() {
    this.mesh.SetVertices(this.vertices);
    this.mesh.SetTriangles(this.triangles, 0);
    if (this.colors.Count > 0) {
      this.mesh.SetColors(this.colors);
    }
  }
}

Now in our system we want one mesh per terrain type (4 meshes, WaterMesh, DirtMesh, GrassMesh, RockMesh). We can just create a MapMesh class to handle this. The MapMesh class will contain a dictionary of TerrainType/Mesh and a GenerateMesh method. Let’s also do a GetMesh method to get a mesh from a TerrainType (if the dictionary has a mesh for this terrain type we will return else we will create then return).

public class MapMesh 
{
  public Map map;
  public Dictionary<TerrainType, MeshData> meshes;

  public MapMesh(Map map) {
    this.map = map;
    this.meshes = new Dictionary<TerrainType, MeshData>();
    this.GenerateMesh();
  }

  public MeshData GetMesh(TerrainType terrainType) {
    if (this.meshes.ContainsKey(terrainType)) { // We already know this terrain type, let's return the mesh
      return this.meshes[terrainType];
    }
    this.meshes.Add(terrainType, new MeshData()); // It's a new terrain type, let's create a new MeshData object.
    return this.meshes[terrainType];
  }

  public void GenerateMesh() {

  }
}

And now let’s generate the mesh for each terrain type.

public class MapMesh 
{
  //...
  public void GenerateMesh() {
    foreach (Tile tile in this.map) {
      MeshData meshData = this.GetMesh(tile.terrainType); // Get or create a new mesh

      // Get the verticeIndex, it's just the vertice count in our MeshData object.
      int verticeIndex = meshData.vertices.Count; 

      meshData.vertices.Add(new Vector3(tile.position.x, tile.position.y)); // Vertice 0 (check image)
      meshData.vertices.Add(new Vector3(tile.position.x, tile.position.y+1)); // Vertice 1
      meshData.vertices.Add(new Vector3(tile.position.x+1, tile.position.y+1)); // Vertice 2
      meshData.vertices.Add(new Vector3(tile.position.x+1, tile.position.y)); // Vertice 3
      
      /*
        That's the first triangle (or TriangleA if you check the image).
        But we can just use our AddTriangle method in ou MeshData object.
        meshData.triangles.Add(verticeIndex);
        meshData.triangles.Add(verticeIndex+1);
        meshData.triangles.Add(verticeIndex+2);
      */
      meshData.AddTriangle(verticeIndex, 0, 1, 2); // TriangleA
      meshData.AddTriangle(verticeIndex, 0, 2, 3); // TriangleB
    }

    foreach (MeshData meshData in this.meshes.Values) { // Don't forget to call build for each mesh.
      meshData.Build();
    }
  }
}

To test this we need to change our MapRenderer.cs. I will use GameObjects with MeshRenderer in this example but you could also use Graphics.DrawMesh.

public class MapRenderer : MonoBehaviour
{
  public void Start() {
    //...
    MapMesh mapMesh = new MapMesh(this.map);
    foreach (KeyValuePair<TerrainType, MeshData> kv in mapMesh.meshes) {
      MeshData meshData = kv.Value; // It's just easier to read, you don't need to do this.
      TerrainType terrainType = kv.Key; // It's just easier to read, you don't need to do this.


      GameObject go = new GameObject("Mesh for "+terrainType.ToString());
      go.transform.SetParent(this.transform);
      
      // In our TerrainType enum Water=0, Dirt=1, Grass=2, Rocks=3
      // We always want to draw Rocks over Grass, Grass over Dirt, Dirt over Water
      // So we can just use the negative integer value as the Z position for the GameObject.
      go.transform.localPosition = new Vector3(0, 0, -(int)terrainType); 

      // Add a mesh filter and set the mesh to our mesh.
      MeshFilter mf = go.AddComponent<MeshFilter>();
      mf.mesh = meshData.mesh;
    }
    this.ready = true;
  }
}

If we check the inspector for our new game objects (Mesh for Water, Dirt, Grass, Rocks)  we can see the mesh in the MeshFilter component.

Obvisouly,  now we need to add a Material and a Shader and a MeshRenderer to each GameObject.

 

3. Adding material / shader

I will not explain the Shader in details, but it’s a simple one, the only difference with the basic Unlit Texture is that we handle transparency, multiply the texture color by the vertex color alpha, and use vertex as the texture coordinate.

Shader "Game/TerrainTiles" 
{
  Properties 
  {
    _MainTex ("Main texture", 2D) = "white" {}
  }

  SubShader 
  {
    ZWrite Off
    Tags { "Queue" = "Transparent" }
    Blend One OneMinusSrcAlpha 

    Pass {
      CGPROGRAM

      #pragma vertex vert
      #pragma fragment frag

      sampler2D _MainTex;

      struct appdata {
        float4 vertex: POSITION;
        fixed4 color: COLOR;
      };

      struct v2f {
        float4 pos : SV_POSITION;
        float2 uv : TEXCOORD0;
        fixed4 color: COLOR; 
      };

      v2f vert(appdata v) {
        v2f o;
        o.pos = UnityObjectToClipPos(v.vertex);
        o.color = v.color;

        // This is the tiling or the texture resolution (pixels per tile) if you want, for example:
        // if you textures are 512x512 and you do o.uv = v.vertex.xy you will have in each tile full texture.
        // But if you do v.vertex.xy/8 a 512x512 texture will cover 8x8 tiles.
        // If you use seemless textures, this is nice for rendering tile map grounds. 
        o.uv = v.vertex.xy/8; 

        return o;
      }

      half4 frag (v2f i) : COLOR {

        // We just multiply the texture color by the vertex color alpha this will be use for the blending.
        half4 texcol = tex2D(_MainTex, i.uv.xy) * i.color.a; 

        return texcol;
      }

      ENDCG
    }
  }
}

Let’s Add 4 materials to our project, one for each terrain type using this shader.

And will also do a quick static class Res to have access to our materials (you could also just add a Material List to the MeshRenderer) .

public static class Res {
  public static Dictionary<string, Material> mats;

  public static void LoadMats() {
    Material[] mats = Resources.LoadAll<Material>("Materials/");
    Res.mats = new Dictionary<string, Material>();
    foreach (Material mat in mats) {
      Res.mats.Add(mat.name, mat);
    }
  }
}

And let’s add the MeshRenderer, and material to our MeshRenderer class.

public class MapRenderer : MonoBehaviour
{
  public void Start() {
    Res.LoadMats();
    //...

    foreach (KeyValuePair<TerrainType, MeshData> kv in mapMesh.meshes) {
      //...

      MeshRenderer mr = go.AddComponent<MeshRenderer>();
      mr.material = Res.mats[terrainType.ToString()];
      
    }
  }
}

And now you should have somthing like this when you press Play:

5. Blending / Edge Smoothing

We will first need to know the neighbours for each of our tiles (8 max) and what mesh we use in our neighbours.

We will also need to know the terrain type of each neighbours.

If the Tile T of terrain type Dirt has a South neighbours of rocks, we will add a new rectangle to the Rocks mesh, and subdivide it in 4 small rectangles.

So first let’s make a Direction enum and some utils to get our neighbours.

public enum Direction : ushort {
  S, SW, W, NW, N, NE, E, SE
}

public static class DirectionExtensions {
  public static Vector2Int Position(this Direction direction) {
    switch (direction) {
      case Direction.S:
        return new Vector2Int(0, -1);
      case Direction.SW:
        return new Vector2Int(-1, -1);
      case Direction.W:
        return new Vector2Int(-1, 0);
      case Direction.NW:
        return new Vector2Int(-1, 1);
      case Direction.N:
        return new Vector2Int(0, 1);
      case Direction.NE:
        return new Vector2Int(1, 1);
      case Direction.E:
        return new Vector2Int(1, 0);
      case Direction.SE:
        return new Vector2Int(1, -1);
      default:
        return Vector2Int.zero;
    }
  }
}

Now let’s edit our GenerateMesh method, and add terrainList and neighboursTerrains.

public class MapMesh 
{
  //...
  public void GenerateMesh() {
    List<TerrainType> neighboursTerrainList = new List<TerrainType>();
    TerrainType[] neighboursTerrain = new TerrainType[8];
    foreach (Tile tile in this.map) {
      neighboursTerrainList.Clear();

      // [...] Previous code for getting the mesh adding vertices and triangles.

      for (int i = 0; i < 8; i++) { // We have a max of 8 neighbours.
        // Let's get the neighbours with our Direction utility.
        Tile neighbour = this.map[tile.position+((Direction)i).Position()]; 

        if (neighbour != null) { 
          // If the neighbour is not null, we will set neighboursTerrain to the neighbour terrain type.
          neighboursTerrain[i] = neighbour.terrainType;

          if (
            neighbour.terrainType != tile.terrainType && // We add only if its different than current tile.
            !neighboursTerrainList.Contains(neighbour.terrainType) && // And if it's not in the list.
            (int)neighbour.terrainType >= (int)tile.terrainType // And we only blend when we're on top.
          ) {
            neighboursTerrainList.Add(neighbour.terrainType);
          }
        } else {
          // If its null we will pretend it's the same as the current tile (we don't need to blend).
          neighboursTerrain[i] = tile.terrainType;
        }
      }
    }
  }
}

We can do a foreach on neighboursTerrainList, get the meshData and subdivide in 4 rectangles (8 triangles, 9 vertices).

public class MapMesh 
{
  //...
  public void GenerateMesh() {
    //...
    foreach (Tile tile in this.map) {
      //...
      foreach (TerrainType terrainType in neighboursTerrainList) {
        meshData = this.GetMesh(terrainType); // Get the new meshData (rocks for this example)
        verticeIndex = meshData.vertices.Count; // Get the meshData verticeIndx

        meshData.vertices.Add(new Vector3(tile.position.x+.5f, tile.position.y)); // 0
        meshData.vertices.Add(new Vector3(tile.position.x, tile.position.y)); // 1
        meshData.vertices.Add(new Vector3(tile.position.x, tile.position.y+.5f)); // 2
        meshData.vertices.Add(new Vector3(tile.position.x, tile.position.y+1)); // 3
        meshData.vertices.Add(new Vector3(tile.position.x+.5f, tile.position.y+1)); // 4
        meshData.vertices.Add(new Vector3(tile.position.x+1, tile.position.y+1)); // 5
        meshData.vertices.Add(new Vector3(tile.position.x+1, tile.position.y+.5f)); // 6
        meshData.vertices.Add(new Vector3(tile.position.x+1, tile.position.y)); // 7
        meshData.vertices.Add(new Vector3(tile.position.x+.5f, tile.position.y+.5f)); //8

        meshData.AddTriangle(verticeIndex, 0, 8, 6);
        meshData.AddTriangle(verticeIndex, 0, 6, 7);
        meshData.AddTriangle(verticeIndex, 1, 8, 0);
        meshData.AddTriangle(verticeIndex, 1, 2, 8);
        meshData.AddTriangle(verticeIndex, 2, 4, 8);
        meshData.AddTriangle(verticeIndex, 2, 3, 4);
        meshData.AddTriangle(verticeIndex, 8, 5, 6);
        meshData.AddTriangle(verticeIndex, 8, 4, 5);
      }
    }	
  }
}

And now if we press play in Unity and Move some stuff around we can see this:

We’re just missing the vertex color. So we will need to add 4 colors for the tile vertices and 9 for our overlapping rectangles.

public class MapMesh 
{
  //...
  public void GenerateMesh() {
    //...
    Color[] _cols = new Color[9];
    foreach (Tile tile in this.map) {
      // [...] Getting mesh, setting up vertices, verticeIndex

      meshData.colors.Add(Color.white);
      meshData.colors.Add(Color.white);
      meshData.colors.Add(Color.white);
      meshData.colors.Add(Color.white);

      foreach (TerrainType terrainType in neighboursTerrainList) {
        // [...] getting terrainType mesh, verticeIndex, settting up vertices/triangles

        for (int i = 0; i < _cols.Length; i++) {
          _cols[i] = Color.clear;
        }
        meshData.colors.AddRange(_cols);
      }
    }
  }
}

Because we set the overlapping to clear now we should see something like this in unity.

Don’t forget foreach vertex our shader multiply the texture color by the alpha of the vertex color. So now we just need to set every vertex color of the overlapping rectangles.

To do this, we will use a switch and a cool trick.

 

Because we use the same indice order for the neighbours and the vertices of our sub-rectangles, the angles are always odd indices (1,3,6,7) we can just do something like if the overlapping mesh terrainType is equal to the neighbour terrain type at index INDEX set the verticeColor at this index to white. For the remaining vertex colors we will just use a switch.

          • For South (or 0): 1, 0, 7 to white.
          • For West (or 2): 1, 2, 3 to white.
          • For North (or 0): 3, 4, 5 to white.
          • For East (or 0): 5, 6, 7 to white.

And that’s it, it should work after that.

Let’s implement this just after we clear all the colors in _cols.

 

public class MapMesh 
{
  //...

  public MapMesh(Map map) {
    this.map = map;
    this.meshes = new Dictionary<TerrainType, MeshData>();
    this.GenerateMesh();
  }

  public void GenerateMesh() {
    //...
    foreach (Tile tile in this.map) {
      //...
      foreach (TerrainType terrainType in neighboursTerrainList) {
        // Get mesh set vertices/triangles/vertices index.
        for (int i = 0; i < _cols.Length; i++) {
          _cols[i] = Color.clear;
        }
        for (int i = 0; i < 8; i++) {
          if (i % 2 != 0) { // If it's odd.
            if (terrainType == neighboursTerrain[i]) {
              _cols[i] = Color.white; // Odd indices trick
            }
          } else {
            if (terrainType == neighboursTerrain[i]) {
              switch (i) {
                case 0: // South
                  _cols[1] = Color.white;
                  _cols[0] = Color.white;
                  _cols[7] = Color.white;
                  break;
                case 2:  // West
                  _cols[1] = Color.white;
                  _cols[2] = Color.white;
                  _cols[3] = Color.white;
                  break;
                case 4: // North
                  _cols[3] = Color.white;
                  _cols[4] = Color.white;
                  _cols[5] = Color.white;
                  break;
                case 6: // East
                  _cols[5] = Color.white;
                  _cols[6] = Color.white;
                  _cols[7] = Color.white;
                  break;
              }
            }
          }
        }
      }
    }
    //...
  }

And now we should have beautiful blending. Thanks for reading this tuturial, you can find the source on GitHub. Please share and consider supporting me on Patreon (I need a lot of time to make tutorials like this).

Obvisouly you’re limited for the map size because Mesh indices are ushort in Unity but in a real game you should divide your map into sections and only draw what’s shown on the camera but that’s for an other tutorial 🙂

 

Be the first to write a comment.

Leave a Reply