Ever since college, I've been fascinated with machine learning, specifically, teaching computers to create art, music, and potentially even entire games. Is it possible for a computer to do what we do - be creative? Could an advanced statistical learning model, with enough data, produce art, music, stories, or even design gameplay and write code? If you're worried about machines becoming independent of humans and taking over the world, I can assure you that we're a long way away from that. However, every now and then I come back to this idea, thinking about how cool it would be if I could create a game that generated itself from an "intelligent" computer program. Once I can teach a computer to generate at least one game-element satisfactorily, I plan to build a full game around that ability.

This blog post is about my most recent attempt to teach computers to be creative. The idea was to create a program that would draw graphics by learning the probability distribution of a color given the pixel above it, to the left of it, and the pixel in the previous mipmap level of the input data (graphics made by humans). Creating a new graphic then involves drawing from the distributions from lowest detail mipmap to the highest, left to right, and top to bottom, ensuring that we always have the inputs for the distributions while generating. I liked the idea, because it mimicked what artists do - first creating the basic shape of an object, and then adding in details. Similarly, the program progresses from low detail to high detail with each mipmap level.

The Data

The first 493 pokemon used as input data for the program.

I wanted to generate something fairly retro, and I needed a lot of consistent sprites to teach the model. So, I chose to use sprites from Pokemon as my data set as they were easy to find and there were a lot of them. The question of whether a program taught using pokemon sprites generates copyrighted material or not is tricky, but this was just for fun/research. Since the model depends on the number of colors in the image (the more colors, the harder it is to learn and the more data it needs), I decided to convert the image to just four grayscale colors. I also downsized the pokemon to 32x32 pixels, all to make the data smaller and easier to learn from.

The Results

Ideally the program would be able to learn how to create infinite reasonable looking original pokemon (imagine catching them all!). Instead, the program generated what I've termed "inkblots," some of which don't look anything at all like pokemon, some of which have interesting pokemon-ish shapes if you look for them, and some of which are actually decent (if you squint a little). The quality of the results is completely subjective, so I'll let you be the judge:

36 randomly generated results. Didn't quite have the coherence and structure I had hoped, but not outright terrible either.

You can also test the program yourself on my blog post here: http://david.fancyfishgames.com/2012/10/the-artistic-computer.html

Older Experiments

Back in college, I attempted to teach a computer how to compose music by feeding midi data to a Hierarchical Hidden Markov Model. The results for that were not that great, but I still have one midi file generated by it that I thought sounded nice (from a set of anime midi files): http://fancyfishgames.com/Procedural/sample.mid, and here's a mp3 version for those who have trouble playing midi files: http://fancyfishgames.com/Procedural/sample.mp3. However, this was one result from many that were less coherent, so take it with a grain of salt.

Conclusion

I still haven't made a program that can teach computers to make games, or even acceptable pixel art yet, but that doesn't mean it won't be possible one day. For now, the results are interesting given that this was just a short project, so I figured I would share. Feel free to share any experiments you have done with machine learning and games. Until my next experiment!