Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
September 20, 2014
arrowPress Releases
September 20, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Differences between Software Testing and Game Testing
by Johan Hoberg on 07/21/14 05:09:00 am   Expert Blogs   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutraís community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

What is the major difference between testing an application or a software system compared to testing a game? This is the question I asked myself when I switched jobs after ten years of working with software testing and quality assurance within mobile phones, to start working with games. I have read almost everything there is to read about software testing in general, but I have no real knowledge about game testing. So I have bought two books on game testing to get some basic understanding, and worked a few weeks to get some practical hands on experience.

To aid in my own competence development I tried to compile what I have learned so far. 

Software testing is an engineering discipline. This is also true for game testing. Yes, you can play games and find bugs without structure or expert knowledge just as you can test that a mobile phone works, but if you want to exceed you need to become a professional tester.

Being a professional game tester means that you must understand both testing in general, but also the unique discipline of game testing. This is no small feat, as software testing is very complex and multifaceted, and game testing requires very specific skills.

Before we can explore the differences we need to agree on the similarities between software testing and game testing. Basically everything that Cem Kaner has written in his course Black Box Software Testing [1] is in my opinion applicable to both fields. Almost everything Alan Page has written about test automation [2] can also be applied to game testing. Most of what is included in the ISTQB testing certifications [3] should also be valid. Game testing basically inherits almost every aspect of software testing in general. Even the mindset of James Bach and his rapid software testing [4] can basically be tailor made for game testing. How Google and Microsoft works with software testing can teach game testers a lot about how to structure and organize testing in a good way [5][6].

Game testing is every bit as complex as any other testing, and should be treated in just the same way.  It is not something that can be left to untrained laymen. There is a place for live user test, or alpha and beta test, but those are just a few pieces in a much larger puzzle. To reduce game testing to something as simple as just playing the game to see if you find any problems, is to seriously undermine the quality of your game.

So how is game testing unique? What aspects make game testing different from normal software testing? I tried to compile a list of types of testing that makes game testing especially troublesome. [7][8][9]

  • Fun Factor Testing
  • Balance Testing
  • Game Level/World Testing
  • AI Testing
  • Multiplayer/Network Testing
  • Audio Testing
  • Physics Testing
  • Realism Testing
  • Modification API Testing

Usability or user experience is something that is tested for all software, but Fun Factor Testing is something unique to games, since they are an entertainment product. Games are not only supposed to work intuitively and provide a good user experience – they also have to be fun to play. How do you assess if something is fun or not?  How do you know if a certain experience is something that will entice the proposed target group? This requires a unique insight into game design, and vast experience and data about the user group and what that group enjoys.

Creating balance between different options, as well balance of difficulty for different levels, monsters or events, is also something unique to games. Balance Testing can only be done in a good way with a vast knowledge of game design and how the target audience responds to different difficulty levels. It also requires many hours of actual gameplay of the game under test.

One of the most complex aspects of game testing can be the testing of the actual world or level, especially if it is a vast, sprawling, 3D world, such as for modern MMOs. Some parts of this can be automated in interesting ways that are also quite unique to game testing, such as having bots move randomly through the game world to see if they get stuck or find other problems with the world. As the complexity of the task grows, it becomes more and more important to find ways to reduce the complexity with the help of tools. For puzzle games it is important to make sure that all the graphics for every level looks good, but also that each level is passable, and that game mechanics that have previously been tested in isolation, actual work in different level implementations.

Testing that computer-controlled opponents are working correctly, that the artificial intelligence is behaving according to design, can also become very difficult as the complexity of the behavior increases. Chess is a good basic example, and enemies in a first person shooter is a more modern one.  This type of testing requires the tester to understand what triggers different types of behavior, and how these triggers can be confused by different parameters. Understanding of AI and game design is critical to succeed in this field.

Multiplayer testing is a whole other beast in itself. Many players simultaneously interacting with the game world, with computer-controlled opponents, with game servers, with each other. So many things that can go wrong. And it often requires a whole team of testers. Many difficult risk-based decisions to make if you don’t want to spend unlimited amounts of time testing different scenarios. Understanding of multiplayer game design, and how to test efficiently as a team is required knowledge for this type of testing.

Audio testing is common in all software that creates some kind of sound or plays media. However games have a unique aspect that other software does not have to care about to the same extent. Game music has to involve the user in the game and enhance the game play. Not only should the audio play without stuttering or missing elements, it should also add to the gameplay. This requires extensive audio skills and specific understanding of game audio. Very specific expert domain knowledge.

Many modern 3D games have physics engines. Modern shooters such as Battlefield or Crisis and RPGs such as Skyrim, have destructible environment and possibilities to throw objects. Testing if the physics engine works requires an understanding of physics as well as how to implement that in a good way into a game. The complexity of the testing grows with the complexity of the engine.

Especially in simulators or racing games it is very relevant that the game feels real, but many other games also incorporate elements that need to feel real. Testing for realism requires specific domain knowledge needed for the game or game element. An airplane simulator requires an understanding of airplanes. Cars, weapons, human movement, animals. Each poses their own problems. 

A lot of software has open API that can be used by third parties. But there are few other instances where the players will try to exploit these open API to gain unfair gameplay advantages. This requires outside-the-box thinking. How will the modders use the API, and how will they be able to alter the gameplay in significant ways? Being one step ahead of the entire community of modders is a daunting task indeed.

In addition to these different types of testing, categorizing users can most likely also be done in a unique way for games, to help prioritize and select what tests and use cases are most relevant to run. How to do this categorization is another article in itself, but I leave you with an example of how you could think about categorizing different types of gamers. [7][8]

I have tried to cover some of what makes games a unique testing experience. I am sure there is even more, and I hope that I will be able to identify even more aspects as my experience in game testing grows.

One thing is already certain however. Game testing is very complex, and the complexity will grow as games become more and more complex. It is a combinatorial explosion that can only be handled by testing in a very smart way.  Underestimating the competence and effort required to test games is something that can undermine the quality of any game.

Any tester who takes their game testing seriously need to master a large number of techniques, methods and processes, and understand their product in a way which is not required for many other software testing fields.

A daunting task indeed.

 

References

[1] BBST

http://www.testingeducation.org/BBST/

[2]The A Word

https://leanpub.com/TheAWord

[3]ISO29119

http://www.softwaretestingstandard.org/

[4]Context-Driven Testing

http://context-driven-testing.com/

[5] How Google Tests Software

http://www.amazon.com/Google-Tests-Software-James-Whittaker/dp/0321803027

[6]How We Test Software at Microsoft

http://www.amazon.com/How-We-Test-Software-Microsoft/dp/0735624259/

 [7] Game Development Essentials: Game QA & Testing

http://www.amazon.com/Game-Development-Essentials-QA-Testing/dp/1435439473

[8] Game Testing: All on One

http://www.amazon.com/Game-Testing-Second-Charles-Schultz/dp/1936420163/

[9] Artificial Intelligence (Video Games)

http://en.wikipedia.org/wiki/Artificial_intelligence_(video_games)

 


Related Jobs

Infinity Ward / Activision
Infinity Ward / Activision — Woodland Hills, California, United States
[09.20.14]

Senior AI Engineer
Infinity Ward / Activision
Infinity Ward / Activision — Woodland Hills, California, United States
[09.20.14]

Lead Tools Engineer - Infinity Ward
Insomniac Games
Insomniac Games — Burbank , California, United States
[09.19.14]

Senior Engine Programmer
Nexon America, Inc.
Nexon America, Inc. — El Segundo , California, United States
[09.19.14]

Front-End Developer






Comments


Katy Smith
profile image
Hey, an article about QA! And it has sources! Yay! :)

I would caution you on some of your testing ideas. If you get a job testing a game be extremely cautious in bugging "fun factor", "balance" and "emotional impact" of audio. There is a very good chance that it will just piss off the developers.

Most of that stuff is decided months before an alpha or pre-alpha build gets into a tester's hands. I would ask the lead or producer what the scope of testing is. Some might be fine with balance suggestions or design changes and others will get very angry. Things like "feel" and balance are in the realm of design, so unless they are asking about it, you are stepping on somebody else's toes.

In working in both game and non-game software worlds, I would say the biggest differences are reliance on test plans and randomness. Games are inherently random. You can't predict player behavior and when you add AI on top of that, it gets complex very quickly. This randomness means your test plans can't cover as much of the testing, and you have to rely on more ad hoc methods. This leads to longer test cycles and more stress :) still, some games are easier to test than others. I find adventure games are easier than most, and games that have both other players and AI are the hardest.

I would also add compliance testing to your list. There are many TRCs/TCRs/LotChecks/User Interface Guidelines that have to be met in order for your game to see the light of day.

Good luck with your game testing :) let us know how it goes!

Johan Hoberg
profile image
Thank you for the comment :)

Don't you think that game testers should be involved even before alpha/pre-alpha? You could have game testers testing features as soon as they are ready at a very early stage as well? Even be part of the requirements process to make sure the games are testable in a good way, and such.

I agree with you that the types of test you mentioned are very difficult to test, and should not be done without someone explicitly asking you to do so. As you say - stepping on toes. Creating a bug report about something not being fun or emotional can cause more harm than good in some instances, especially if you do not have the knowledge or experience to do the assessment. However I am guessing that games need to go through this kind of testing at some stage, even if the tests are not being done by actual game testers? Someone needs to try the game and assess if it is fun or not? If the music is emotional or not? Maybe someone who didn't actually design or compose that specific part of the game or piece of music? It might be that the domain knowledge required to do the tests is not something that the game testers possess and that it is actually some game designer or music composer who perform the tests?

Compliance tests are indeed relevant in gaming, but there are compliance tests for almost every field of software testing. Is there something unique about compliance tests for games?

Multiplayer and AI definitely seem to add a lot of complexity to games, so from an outside perspective they seem to require more/smarter testing than games which lack these features.

Interesting comment about randomness in game. I will think some more about that.

Thank you again for the comment.


Johan Hoberg


P.S. none of my questions in this comment are rhetorical, so if you have time, please comment again :)

Dusten Sobotta
profile image
"Creating a bug report about something not being fun or emotional can cause more harm than good in some instances, especially if you do not have the knowledge or experience to do the assessment. However I am guessing that games need to go through this kind of testing at some stage, even if the tests are not being done by actual game testers? "

A playtest session with people outside of the studio is usually reserved for this kind of feedback. General gameplay testers should be looking for things that look or act in an unexpected manner. This -can- include sections that are not 'fun', especially in the cases that testers decide a given segment of the game is way too difficult or easy. But generally design decisions are not brought into question by testers. Weird things get bugged up, and designers/engineers/management make decisions on how to improve or resolve the 'bugs'.

David Lejeune
profile image
Former Game Compliance QA guy here:

As I understand it, Software Compliance Testing is more about internal standards set by the company, and those can be waived by whomever. In Game Testing, "Compliance" usually refers to standards set by the console First Parties (Sony, Microsoft, Nintendo).

The biggest issue from a game development and game QA standpoint is that the standards vary significantly between the first parties (and even within them, in Sony's case. Unless they finally fixed that, as was rumoured), so multi-platform development becomes even more complicated.

And since there's actually a round of acceptance testing by the first parties before you can release on their platforms (the submission process), and failing submission means paying a not insignificant sum of money to submit again (though with Microsoft, iirc, you get two submissions included in the price of the concept approval. 3+ starts costing money, and they increase the cost for each additional sub), making sure that the title is compliant is a pretty big deal. Especially if you're a smaller company that can't afford to pay a lot of money to release the game, or you're pressing right up against the release date that Marketing has been spending tons of money touting (slightly less of a big deal with digital being on all systems, but if there's a deal with GameStop or some other brick & mortar it can get messy.)

Joseph Russell
profile image
"What is the major difference between testing an application or a software system compared to testing a game?"

One pays a living wage a person can live on and start a family with, the other one usually does not?
(*ba-dum-bum-KSHH*)

But in all seriousness, the real challenge to the average tester and QA Engineer in the Video Game industry is the creativity required to find major blockers as well as the numerous edge cases that can pop-up in any project. In industries such as E-commerce, Consumer Electronics, Broadcast Media, and Web App Development - the bugs can still be hard to find at times but not to the level of some of the serious edge cases I have seen while I was working in games QA. Many bugs in other industries can be found more easily via established manual test cases and feature suites, whereas in games there are so many possible factors and scenarios to be considered that to craft a full and comprehensive test plan for a project would be inefficient.

David Lejeune
profile image
The other thing is that (in my experience) game QA isn't really empowered to do the risk assessment for those edge cases.

In software QA, risk analysis and assessment is one of QA's biggest responsibilities, so when you find some one-off edge case QA can look at it and say "in a real world end-user scenario, this particular set of inputs is not going to happen, I don't think we need to even write this up."

In Game QA, that call is usually on production or programming. And since employee performance for testers is (again, in my experience) almost always linked to bug quantity as opposed to bug quality, everybody is writing up everything all the time, regardless of severity or even validity (I have seen so so many collision bugs written up that would be literally impossible to find without debug tools, or stability and progression bugs CAUSED by using debug to skip around in games. And god knows I wrote some stupid, silly bugs to try and keep my bug count up back when I was a contract game tester), and this means you end up with testers wasting time re-producing bugs that don't need to be fixed, leads wasting time vetting those bugs, the strong possibility of programmers/level designers/artists wasting time trying to fix those bugs (if they've been written up before somebody's started triaging the database because holy shit there are 250,000 open bugs in the db and we've only got two months to ship), and then a tester wasting time AGAIN regressing it, and potentially starting the cycle over again if their nigh zero visibility edge case bug isn't actually fixed.

Johan Hoberg
profile image
Great comments! Thank you.

Olli Alatalo
profile image
I've been working on my thesis about games for some time now, and this is one on-going theme that keeps popping up: What's the difference between traditional software engineering and game development.

I think you hit the spot pretty neatly, although I failed to see how Multiplayer testing differs from other high intensity network applications, say Live Streaming services or Ebay.

I have never worked on projects with large teams (my teams have always been less than 10 people), but I feel that rather than having separate QA team, it would be more beneficial to have designers & developers educated in QA matters. As certain speaker at a scrum conference put it: "Whenever you find a bug, don't write it down, fix it."


none
 
Comment: