What Went Right
1. AI-Assisted Design Process
Our AI-based approach to design paid off in spades. There is no question in our minds that this exceeded our expectations and improved product quality and time-to-market. This was not some high-minded academic fluff but a practical, boots-on-the ground competitive advantage.
The full theory behind our design process is much too complex do it justice in the scope of a postmortem article. We hope to be able to explain it more fully in the future if time permits. But we can touch on two main elements of our design approach that impacted City Conquest.
The first was our optimization-based approach to the core design. The defensive towers and unit types in City Conquest were not based on arbitrary creative decisions: nearly all of the decisions around their functional aspects were guided by an explicit decision modeling process.
We clearly specified the design goals and constraints for all of our towers and units and then constructed a decision model to optimize the features of all the units and defensive towers together as a whole.
We then used an evolutionary optimizer to select the best set of nine towers and nine units that would best work together in concert to satisfy our design goals while remaining within the boundaries of our design constraints.
This approach is broadly similar to the one described in the book Decision-Based Design, although our approach is much simpler and customized for game design decisions.
We firmly believe that this type of optimization-based approach can pay major dividends for game developers. It can help us provide more value to our customers by reducing the complexity of design problems, allowing us to make more optimal decisions more quickly, and in some cases, allowing us to solve problems that are otherwise unsolvable.
The second advantage was Evolver, which we discussed in an earlier interview with AIGameDev.com. Evolver was an automated balancing tool based on coevolutionary genetic algorithms. Every night, it would run a huge number of simulated games between red and blue opponents, with each opponent evolving a "population" of scripts (each script being essentially a fixed build order of buildings within the game).
Evolver would generate random scripts, play them against each other, and assign them a fitness score based on which player won and by how much. It then applied standard evolutionary operators such as crossover and mutation to genetically optimize each population.
This meant that we could wake up every morning, open Evolver, determine which scripts the red and blue players ranked the most highly, and then plug those into the game and watch them play against each other. This instantly told us how our game balancing was working. Were players building too many Rocket Launchers? Not enough Skyscrapers? Were Commandos not useful enough, or were they consistently preferring Crusaders over Gunships?
We could then use this output to tune and refine a few units and buildings every day, tweaking their resource costs, health, speed, damage, rate of fire, and other parameters. It was very much like having an external outsourced testing team that would play the game overnight -- except that it was cheaper and more scalable than a human playtesting team, in addition to being capable of absolute objectivity.
We optimized Evolver by disabling the rendering, adding task-based parallelism, and hand-optimizing the gameplay logic. This allowed us to achieve roughly 1 million simulated games per 12 hours. We later upgraded the genetic algorithm to use an island model and hand-tuned the fitness function to achieve certain outcomes (such as helping the script populations learn to upgrade their Skyscrapers quickly enough to achieve optimal long-term income levels).
This might seem like a lot of work. It wasn't: the work to create, tune, and optimize Evolver was roughly two weeks' worth of development time in total. Considering all the valuable feedback that Evolver gave us, the fact that it gave us better results than we could have achieved by hand, and the fact that doing this initial hand-tuning would have taken far longer than the two weeks we spent on Evolver, we consider this an obvious net win.
It also left us with a system that we could quickly run to test any possible changes to the gameplay parameters to see the ramifications of changing any design changes -- in one case, we were able to quickly identify the issues that would arise from reducing the Roller's cost from 2 crystals to 1 crystal (and reducing the Roller unit's stats accordingly), and Evolver allowed us to immediately identify the problems and abandon this idea before it caused gameplay problems.
We also benefited enormously from having a large number of playtesters in our TestFlight team for eight months leading up to release giving us the invaluable aesthetic, usability, and other subjective feedback that Evolver could not. We eventually invited all of our Kickstarter backers to join us as playtesters.
As a result of all of these factors, the game was fun almost from day one. Every design concept worked.
2. The FBI: Fix Bugs Immediately
I've worked on several projects with 5K+ bug counts in the past. Once the team moves on to the final "bug fixing" phase, there are inevitably dozens of awful discoveries: "Wow, if we'd known that bug was there, we could have saved so much time!" "That bug caused ten other bugs!" "If we'd gotten that bug fixed six months ago, we would have totally changed the way we designed our levels!"
On one memorable project, a producer told his team: "Don't fix bugs -- we don't have time right now! Save them all up until the end!" The project failed catastrophically months later, destroying the studio and nearly pulling the entire franchise down along with it, in no small part due to the overwhelming bug count at launch.
That should never happen. Letting any software defects linger is a risk to the schedule and to the integrity of the code base.
Our approach is to fix bugs immediately. We don't work on any new features or continue any other development until all known bugs are fixed. Design flaws and performance problems count as "bugs." Playtesters' suggestions also usually count as "bugs," especially if more than one playtester reports the same problem or suggests the same change.
Our running bug count remained under 10 at all times throughout development.
Now that we've done it, we can't imagine developing any other way. What's the point of postponing fixes just for the sake of the "schedule"? It's essentially a form of accounting fraud: you're pushing off a problem onto one side of the ledger to pretend that the other side of the ledger is doing fine. You're introducing hidden costs by sacrificing product integrity.
Our playtesters frequently mentioned the relative lack of bugs, and this codebase integrity ensured our testers would be focused on gameplay rather than technology issues.
In our experience, the practice of pushing bugs into a database and waiting to fix them until a later date is a major cause of product delays, developer stress, work-life imbalance, and studio failures. It's a primitive and barbaric practice and it needs to end.