Gamasutra: The Art & Business of Making Gamesspacer
Postmortem: Intelligence Engine Design Systems' City Conquest
View All     RSS
August 23, 2014
arrowPress Releases
August 23, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Postmortem: Intelligence Engine Design Systems' City Conquest

February 6, 2013 Article Start Previous Page 4 of 4
 

3. Kickstarter

There were two City Conquest Kickstarter campaigns. We set the initial goal at a modest $12,000. When it moved too slowly, we canceled it, worked on extensively polishing the game, and returned a few months later with a better pitch, a better product, and an even more modest $10,000 campaign.

The good news is that we reached our funding goal and delivered a much better product than our Kickstarter actually promised.

The bad news is that it probably wasn't worth it. We only got 146 backers and only modestly exceeded our funding goals. The time involved in creating and managing the two Kickstarter campaigns and promotional videos would have been better spent working on the game.

The most problematic issue we faced was our promise to offer our backers the full game for free. At the time, we believed we could use Apple's promo codes to keep this promise, but we unknowingly made it impossible to keep when we changed the plan for our pricing model from separate "lite" and "full" versions to a single combined product with an in-game IAP to unlock the full game.

We'd been assured by several sources that Apple promo codes worked with in-app purchases, so we felt very confident in this approach. We also had a long list of workarounds on hand in case this failed, including hard-coding the recipients' device UDIDs or offering a customized redemption dialog in-game. Unfortunately, promo codes proved not to work for IAPs, and we learned how severely Apple frowns on custom redemption dialogs, the use of UDIDs, and pretty much every single other workaround we'd had in mind.

This was a clear and unfortunate failure of due diligence. We immediately apologized to our backers and offered a refund. The financial impact was minimal since this ended up being only a few dozen pledge refunds at $6 each, but it still should never have happened.

The true value of Kickstarter for us was that we could convert many of our backers into playtesters who provided invaluable feedback. If we use Kickstarter again, we will focus on using it to acquire playtesters and communicate with our audience rather than using it as a fundraising vehicle.

4. Getting Engineering Help Too Late

We realized in May 2012 that our team was stretched too thin to be able to implement and test multiplayer properly. We had a relatively solid first-pass multiplayer implementation in place, but our discovery-driven plan was telling us that multiplayer was a huge risk and required much more attention.

At the same time, multiplayer was non-negotiable: it was an essential part of the game's vision and added far too much value to the project to consider dropping it.

We brought on the Full Cycle Engineering team to replace our proof-of-concept netcode with a serious multiplayer implementation. The multiplayer undoubtedly is a far better experience now than it could ever have been without their contributions. However, we should have recognized the bottleneck and brought the Full Cycle team on board much earlier than we did, and we should have focused on Wi-Fi from the outset rather than experimenting with cellular. The tasks involved were huge and this development effort would have benefited enormously from a much earlier start had we been willing to admit sooner that we needed the help.

5. Due Diligence Failures

Our biggest mistakes were several basic errors in due diligence. None of these ultimately impacted product quality, but we hold ourselves to a higher standard than to be the type of studio that would make these kinds of mistakes.

Other than the aforementioned Kickstarter snafu, the worst due diligence failure was iPod Touch support. For reasons too complex to explain here, the iPod Touch 4 had to be our baseline iOS device, and our discussions with friends and our own research indicated that the iPod Touch 4 was basically the same as an iPhone 4.

This was incorrect. The iPod Touch 4 is a markedly inferior device, and it suffers from much more severe memory constraints. This forced us to bite the bullet and implement a lot of fine-grained memory optimizations. We spent four to five developer-days on time-consuming texture and sound memory usage reductions and major improvements to our asset-loading system and resource management code. Although many of these memory optimizations benefited the overall game, these problems should have been identified and resolved earlier.

We also had problems with a third-party technology we selected for the multiplayer lobby. Very late in the development cycle, we were forced to confront intractable technical issues around this technology, and we had to replace it with a solution based on Amazon Web Services. Rather than selecting this technology ourselves, it would have been better for us to have tasked the Full Cycle team with the due diligence around this and let them pick the best tool for the job.

A third case was our achievements. Some developer friends assured us early in development that there was no limit on the number of Apple GameCenter achievements per game. When we realized that GameCenter actually has a fixed limit of 100 achievements, we needed to drop dozens of achievements to get back under this limit. In the end, this ended up being a good thing, as we only lost a few days' worth of time and we were able to significantly improve the overall quality of our achievements by separating the wheat from the chaff. Although we regret the wasted time and the due diligence failure, the net result was that we ended up with what we feel is truly the best possible set of 100 achievements for the game.

Conclusion

IEDS is a different kind of indie developer, and we are interested in exploring a genuinely different approach to making games. Although we do not claim to have any sort of magic bullet to game development, we are deeply encouraged by what we have seen so far of the results of our approach and the success of our game. We're deeply proud of the final City Conquest game experience and looking forward to evolving it and building on its success with the sequel.


Article Start Previous Page 4 of 4

Related Jobs

Raven Software / Activision
Raven Software / Activision — Madison, Wisconsin, United States
[08.23.14]

Sr. Software Engineer (Gameplay)
AtomJack
AtomJack — Seattle, Washington, United States
[08.22.14]

Level Designer
FarSight Studios
FarSight Studios — Big Bear Lake, California, United States
[08.22.14]

Lead Android Engineer
Churchill Navigation
Churchill Navigation — Boulder, Colorado, United States
[08.22.14]

3D Application Programmer






Comments


Paul Tozour
profile image
I just wanted to note that Gamasutra's tagline on this article, "how automated testing saves time, and how Kickstarter wastes it," is Gamasutra's characterization, not my own, and is not the way I would have characterized this article. I'm working to get this fixed, but in the meantime, Evolver is not exactly an automated testing system, and Kickstarter isn't a waste of time.

Paul Tozour
profile image
EDIT: The Gamasutra editors have tweaked the tagline to avoid confusion. Thanks, Gamasutra!

Don Hogan
profile image
Excellent write-up, Paul. It's always good to hear your take on game development, there's never a shortage of food for thought. Glad to hear the project went well!

Michael DeFazio
profile image
Paul,
Fantastic article--

Wish the Kickstarter video could have been attached it was also awesome (Seeing how you created algorithms to find optimal strategies and had them play against each other was fabulous.)

Love your philosophy about games (problems spaces) and AI... And the amount of times you mentioned "decisions" in the article put a smile on my face (I'm sorta a gameplay first kinda guy, and great games to me always find a way of presenting "interesting decisions").

You completely sold me on this game (one android copy Sold!)... and I will continue to watch for other revelations/advancements you and your company find about making compelling (and balanced) gameplay in the future.

Cheers

Paul Tozour
profile image
Thanks, Michael! For anyone who's interested, the Kickstarter video is here: http://kck.st/GzJ324

I'm also going to be giving a talk on Evolver and some other aspects of my approach at the GDC AI Summit in March (along with Damian Isla and Christian Baekkelund, who will be discussing the role of AI in the design of Moonshot Games' terrific new game Third Eye Crime).

Paul Tozour
profile image
.

GameViewPoint Developer
profile image
I think the AI approach to game testing is definitely interesting but would be a lot of work for a small indie team to implement, perhaps if there were 3rd party tools available it would be a useable solution.

Paul Tozour
profile image
To be clear, the point of Evolver was not as a game testing system. It was designed to help explore the design space and guide the game balancing. It was fundamentally a design tool, NOT a testing tool.

It did help find some bugs, of course, but that was just a nice side-effect. The real point was to help us optimize the balancing between all the different units and towers in the game.

The difficulty of implementing something like this really depends on the game. It's not a magic bullet and it's not an approach that will work for every game. And it's the type of system that has to be carefully designed for the particular game in question, so it's not the kind of thing where you can really create a tool that will work for any game.

In the case of City Conquest, it cost us about 2 weeks' worth of coding and other work, and easily saved us a good 3-4 weeks worth of design time -- while also giving us better results than we would likely have been able to get by hand, AND giving us a system that would give us instant visibility into the ramifications of any given design decision. So, it was clearly a net win just in terms of the time savings alone, even before you consider all of the other benefits.

In any event, as I mentioned in one of the previous comments, I'm going to be speaking about this in more detail at the GDC AI Summit in March. So be sure to stop by if you'll be at GDC.

Denis Timofeev
profile image
Hi! That's a great article, thanks. A lot of inspiration. I'm just wondering how many users you had on TestFlight, how did you get them and how helpful their was.

Paul Tozour
profile image
We had around 50 or so at first, but once we opened the testing up to *all* the Kickstarter backers, we ultimately got to the maximum number of users allowable based on Apple's developer restrictions for the maximum allowable number of devices per app (100). The actual number is below 100 since some of them had multiple devices (iPhone + iPad) so they took up multiple device ID slots.

We started with personal friends and industry contacts, then added backers at the appropriate backing level in Kickstarter, and then, a few months later, opened it up to absolutely all the backers who had iOS devices.

Their feedback was extremely helpful overall. We got a very wide range of feedback from a lot of people at a lot of different skill levels. We had a few industry veterans in there (see the game's Credits for the full list), who provided terrific and detailed feedback, along with a few non-gamers. We also did some diagnostics, such as adding buttons so the testers could send back valuable data on mission completion and achievements earned. So it was nice to see that about 90% of the ones who responded were able to finish the single-player campaign, and every mission was completed by at least one tester at every available difficulty level. Also, they were able to get me invaliable feedback on devices I didn't own / couldn't get my hands on at the time, such as the iPhone 5 and the "new" iPad aka iPad 3.

Louis Gascoigne
profile image
Great article Paul, worth the read just for the tech section.

Bram Stolk
profile image
Impressive stuff, and great article.
Amazing that you could get Computer Aided Game Balancing working in just 2 weeks.
Writing a system like Evolver sounds like more fun than tediously going through manual iterations of game balance.

Jeremy Tate
profile image
@Paul How did you approach actual code testing? It seems like it would be crucial under this model to have technical integrated testers on the team if you were going to maintain a running total of 10 bugs. Otherwise, you are kind of pushing those undiscovered bugs to later in the project.

Paul Tozour
profile image
I definitely agree in principle with having as many testers as possible doing testing from day one. This is a great idea whenever you can afford it, and it worked very nicely for Retro Studios when I worked with them on Metroid Prime 2 and 3 -- the internal elite ninja testing team was super effective.

I think we were able to get away with not having dedicated testers on the City Conquest team thanks to a combination of factors: it was the relatively limited scope of the project, the involvement of our external playtesting team (friends and Kickstarter backers on TestFlight), our defensive programming practices, and our habit of playing through the entire game on a regular basis. Also, the fact that the Evolver tool found a few of the most difficult/subtle bugs on its own just by virtue of the fact that we were running a million simulations overnight and could pretty quickly find anything in the game logic that caused a crash or a hang.

But again, I do agree with you that having dedicated testers throughout the process is the better way to go if at all possible. The earlier you can find bugs, the better.

Paul Tozour
profile image
And when I say "10 bugs," of course I mean 10 *known* bugs. Naturally there's no way to count bugs that you don't know about.

But, yeah -- the earlier you find them, the earlier you can fix them.

Jeremy Tate
profile image
In addition, since a game by its very nature is almost entirely focused on usability vs functionality, so if something has to go, it has to be the latter. It's not like its a piece of medical software where lives depend on the functionality. Which, by focusing on playtesting, was the choice you made.

I'm thinking that a person who would the upkeep of the Evolver scripts, managing the data, disseminating data out and making sure issues are tracked and are acted on would be logical next step though. You could easily scale the concept out and get even more detail.

Paul Tozour
profile image
Absolutely. Evolver is really only scratching the surface of what's possible.

There's also been some very interesting academic work related to this, as well as interesting procedural content generation work, being done by folks like

Alexander Jaffe http://homes.cs.washington.edu/~ajaffe/
Adam Smith http://users.soe.ucsc.edu/~amsmith/
and Gillian Smith http://sokath.com/main/publications/

... and several others whose names escape me at the moment.

James Yee
profile image
Good to see your game came out Paul.

As a writer in the Kickstarter community I recall seeing your project at the time and not being overly impressed by it. (Hence the reset you did) Do you think the Kickstarter would have gone better if it wasn't up to YOU to create it? Basically would it have been more cost effective for you to keep working on the game while letting someone else do the PR/Campaign management of Kickstarter?

Also for future Kickstarter creators how do you avoid the Apple Promo code problem you ran into? Just make sure your full game is a separate app if you plan on a free version?

Paul Tozour
profile image
Hi James -- I'd love to get your thoughts on the Kickstarter and what could have been done better there. Please feel free to send us a direct message via Twitter or e-mail us directly; I always appreciate honest, direct feedback.

Yes, the Kickstarter probably would have been more successful if I'd hired separate PR for it, but I'm not sure it would it have been cost-effective or that it ever would have actually brought in more funding than the cost of hiring the PR firm in the first place.

I didn't bring on a PR firm until the game was ready to launch on iOS; I did consider doing it for the Kickstarter campaign but it seemed like overkill.

As for avoiding the Apple Promo code problem: I don't have a good way to recommend that developers make their app available to backers for free outside of the limited promo codes Apple gives you.

Other developers I've spoken with have recommended briefly making your IAPs free for a very short time frame and telling your audience exactly when they need to download them, or putting hidden features / secret codes in your app (players need to click on a certain pattern on invisible hotspots). But the latter approach especially is risky since it risks incurring the wrath of Apple (and potentially getting banned from the App Store) or becoming open knowledge (and allowing people to pirate your game easily once they know the secret code).

Damian Connolly
profile image
Thanks for the excellent post-mortem, and congrats on the game!

Can you elaborate a bit on how you used discovery driven planning with your game? Did it change the game design, and if so, to what extent? Did you use it mainly in the prototyping stage, or throughout the entire process?

Also, your Evolver tech sounds pretty amazing. Is it specific to the game, or do you see yourself eventually spinning it off as middleware? It seems like an ideal candidate, especially for smaller studios.

Paul Tozour
profile image
Hi Damian -- Thanks for the kind words!

Although I'd love to be able to build middleware someday to help with this aspect of game design, I don't think it will be a case of extending Evolver to do that. The genetic algorithm component of Evolver isn't really anything unique or protectable, and the aspects relating to integration with City Conquest is too game-specific to be able to put it in middleware.

Regarding Discovery-Driven Planning: As luck would have it, I'm working on an article (or two) on DDP for Gamasutra right now. I hope to have them up by the end of the month or maybe early in March.

DDP is really a project planning methodology, so you want to use it throughout the project to make sure you have a good handle on the risks of the project, and ensure that you're prioritizing your efforts to learn as much as you can to reduce uncertainty as cheaply and as quickly as you can.

So on City Conquest, it drove every milestone. It didn't really change the game design directly but it did ensure that we worked on the riskiest aspects of the project first to ensure that we reduced our risks and ensured the smoothest possible path a completed, profitable game.

Paul Tozour
profile image
FYI, Gamasutra has now published my article on applying discovery-driven planning to games:

http://gamasutra.com/view/feature/191523/managing_risk_in_video_g
ame_.php

Josh D
profile image
Hi Paul,

Thanks so much for writing this postmortem, I was hoping you could elaborate a little bit on your monetization decision. You said that in retrospect, you didn't think making a free app with one large IAP was the best choice. I'm wondering why that is and what you think would be more effective? I'm currently involved with an app considering a similar pricing structure, so I'm very curious as to your thoughts and experiences with this. Any feedback would be greatly appreciated, thanks again!

Paul Tozour
profile image
Hi Josh,

What it comes down to is that to really maximize your revenues, you need to be able to do proper price discrimination. And by that, I mean that you need to be able to serve all the customers across the whole curve of different levels of willigness to pay -- the 5% of "whales" who will pay $20+ per month of your app without hesitation, the 10% of "dolphins" who might pay $5 per month, and the "minnows" who will pay maybe $1 a month on average ... in addition to all the non-paying users who just won't pay anything, which will likely be 3-10x your number of paying users.

So, I've come around to the realization that the full-unlock-through-a-single-IAP model is suboptimal because it can only do price discrimination at a single point: the decision of whether or not to pay for that one IAP. So you are getting no revenue at all from the many users who consider your price point too high, and you're getting a lot less revenue than you could from the minority of very wealthy "whales" who would be willing to pay much more to enhance their game experience.

Paul Tozour
profile image
I'd also recommend checking out GamesBrief -- http://www.gamesbrief.com/

They have some interesting papers on this.


none
 
Comment: