We choose to make image segmentation a supervised learning problem meaning to train a segmentation model, we need a training dataset. There is no training dataset online for our specific problem, so we choose to create our own. A segmentation dataset is simple. All we need is a collection of images which come from the game along with their segmentations.

The image on the left is the unsegmented image. The right is the segmented one. The legend is only there for human readability and specifies what classification each color represents. The segmentations are the same size as the original images, but instead of rgb pixels, each pixel stores an integer representing the classification of the corresponding pixel in the original. The original images can be copied straight from the game's screen. The challenge is labeling each pixel to get the segmented images. We can manually segment images but this process is not scalable as each image takes about 15 minutes to segment by hand. And we would need thousands of images to train a network sufficiently.

An alternative approach is to make a program that generates a dataset of synthetic segmented and unsegmented images automatically. We call this program the dataset generator. The unsegmented images will be constructed from images taken from the games Read Only Memory (ROM). These images are broken down into components: background, tilesets, sprites.

We would manually segment these images by hand to create a foundation dataset. Once we have our foundation dataset, our dataset generator stitches them together similar to the way the game renders images on the screen. Our dataset generator will not generate images exactly like they would appear in the game. It only overlaps backgrounds, tilesets and sprites randomly to create a dataset that a neural network can learn from. With this method, we were able to create a synthetic dataset of any size in a short time. Below is an example of what the dataset generator creates.