|
 Features
Automated Build and Test Systems for Games
 Test System Architecture
Our automated test system was written in C# - it launches each
level in the game in turn (parsed from the game’s own level
list file, so it’s always up to date), executing the game
directly into the level being tested on a designated target console,
and then communicates over the network with the running game executable
on the target console to retrieve report data. The majority of
Nihilistic’s tools today communicate over TCP/IP to the running
game to speed iteration time during content creation. The test
system simply hooks into that as well - a highly recommended approach.
If the game doesn’t respond to the test applications probing,
it means that particular level crashed during load and that is
noted and reported. Assuming it does respond, we know that it has
at least been able to load and enter the main game update loop,
where we begin to sample memory and performance data.
Memory and Performance Metrics
In non-retail executable builds, our game engine is constantly
recording various memory and performance metrics. At the test system’s
request, it can send that data back to the test tool across the
network. All of this data is partitioned by the engine programmers
into specific areas of interest. For example, memory data reports
total heap, heap used, texture memory, model memory, physics memory,
etc. Performance data recorded includes average FPS, number of
objects being rendered, number of lights in effect at the time
of probe, amount of time in milliseconds spent doing post-processing
shader passes, and so on. Other useful metrics can also be recorded,
such as load time, locations of spikes in performance, and memory
leaks. We have budgets set up for the majority of these areas of
interest that the testing tool knows about, so when an area is
reported as over-budget by the game it is noted in the report.
Reporting Test Results
In the end, all of this data is dumped by the testing application into a Microsoft Excel formatted XML file (see references for a simple C# library that helps facilitate this). Each game area or level of interest is a row, with columns for each of the different data types collected. This color coded spreadsheet marks in bright red anything that needs immediate attention such as a level that is currently crashing on load, or an area that is using too much texture memory. These tests run every night, though they can also be requested at any time, dropping off a spreadsheet with all this data in the e-mail box of nearly everyone on the project so that they know exactly what is going on with the game. This is especially useful in the later stages of a project when stability and meeting memory/performance constraints are of paramount importance.
Areas for Future Exploration
Although the build and test systems we have developed at Nihilistic
have worked well, there are areas where it could be expanded. The
build request system should be web based, which would allow for
slightly easier updating and would more easily give remote developers
access to make requests. Another improvement currently under investigation
is allowing the test system to automatically navigate the player
through our ever growing levels to more accurately test the effects
of gameplay variables. How this is done is very specific to the
type of game being developed, so it will be left as an exercise
for the reader. Finally, a recent addition to our engine that displays
crash code callstacks on screen could be included into the test
process as well, to report the full callstack for faster identification
of problems when referencing the latest test report. There are
many useful directions automated build and test systems can be
taken in. The key is to evaluate the costs of developing a specific
feature versus the amount of time it saves the team. In many ways
it still makes sense to have a human do various forms of testing.
Conclusion
Developing a robust automated build and test system has saved
everyone on our team countless hours. The framework was developed
in house and took an insignificant amount of time in comparison
to the savings. If you are working on a project of any kind of
significant complexity, I would highly recommend implementing a
similar system. Everyone on the team will thank you for it and
the saved time can be devoted to improving the quality of your
game.
References:
Perforce http://www.perforce.com/
CVS http://www.nongnu.org/cvs/
Apache Foundation’s ANT http://ant.apache.org/
Perforce ANT tasks http://ant.apache.org/manual/OptionalTasks/perforce.html
CarlosAg C# Excel XML Library http://www.carlosag.net/Tools/ExcelXmlWriter/Default.aspx
|
|
|
|
|
Previous