Gamasutra: The Art & Business of Making Gamesspacer
Finding Out What They Think: A Rough Primer To User Research, Part 1
View All     RSS
October 22, 2014
arrowPress Releases
October 22, 2014
PR Newswire
View All

If you enjoy reading this site, you might also want to check out these UBM Tech sites:

Finding Out What They Think: A Rough Primer To User Research, Part 1

April 24, 2012 Article Start Page 1 of 4 Next

[In this first article in a new series, college professor and user research Ben Lewis-Evans takes a look at different methods of game user research, offering up a handy guide to different ways you can collect useful information about your game.]

This article, and its forthcoming followup, is intended to give a rough idea to developers of several different methods that can be used in games user research.

However, many, many books have been written on research methodology and I cannot cover everything. Therefore these two articles cannot be taken as completely comprehensive.

In the first of the articles I will be covering a few general points about Games User Research and then discussing three methods, focus groups, heuristic evaluation and questionnaires in some detail.

What is Games User Research?

So, before getting really started, what is games user research in the first place? Well, let's start by comparing it with Quality Assurance (QA). QA is a well-established part of software development, and is often carried out by professionals within a development team. These folks are, generally speaking, aimed at finding bugs and making sure the game runs smoothly.

However, because those working on a project usually carry out QA, it means they have an investment in it, and are familiar with it. This can cause problems when it comes to evaluating the usability and experience of a game. What seems obvious and fun to someone that has been working on a project may be completely alien and frustrating to a new user.

This is where games user research methods come in, a field that is all about the user and their experience of the game -- in particular, the big question, "is it fun?"

To really (over) simplify things, it could be said that QA and test is about the software, and how it functions when dealing with users, and game user research is about the user and how they function when dealing with the software. Notice I say just it is about how the user functions when dealing with the software, and that it is NOT about testing the user (more on that later).

So how is this done? And what is fun, anyway? Well, there has been plenty written on fun already, so let us just say that fun has quite a few dimensions. Fun can be something that is easy to use, but it can also come along with a struggle and a challenge. It can arise from an engaging experience, a compelling experience, or a relaxing one.

All this means fun is a subjective variable, which changes from person to person, and situation to situation. However, due to the emotional component of fun, what can be said is that if your players are having fun, then it is likely they can tell you about it -- if only you know how to ask.

So How Do We Ask?

Okay -- I'll get to that. But before I get into the fine details, please bear with me as I outline some general principles to keep in mind:

Get the right players

Whatever methods you choose, make sure you get the right sample. For most methods, this means getting representative users, aka the people that you expect will play your game.

If you have the time, perhaps there is some advantage to getting as wide and large a user group as possible (if you really think that everyone will want to play your game), but given that you are likely to be constrained by time (and money) it is generally best to concentrate on getting as close a sample of the target type of player you are after as possible.

The game is being tested, NOT the user

Secondly whenever doing this type of research make sure it is clear to the user that they are not being tested. The research is about improving the game, not the user, so the user shouldn't be made to feel inadequate if they can or cannot do something. In principle it is all valuable information.

This can be hard to do sometimes, as you are the ones designing the game, and it can be uncomfortable to hear others criticising it. But try your best to not be defensive or judgemental (i.e. avoid thinking the problem exists between the keyboard and chair.)

What do you want to know?

Whenever doing research, you should be clear about what you want to know. You will be playing the game yourselves as you work on it, and you should have design documents, so you know how things are supposed to work. So don't just plonk people down with the game and come up with questions on the fly. Work out what areas you think might be problems and know what you want to ask about before it's time to test.

Please note: I am not saying you go out there with preconceptions and already "know" the answers you want; rather I am just making the point that you should be at prepared and know what you're looking for. Otherwise, you get a mass of data that may not be of any use to you at all. At the same time, be open to surprises. You never know what might pop up.

Test early and test often

The next point is considered one of the most important by user researchers, and that is to test as early as you feel it is possible to do so. This can be difficult, as it feels bad letting your baby out into the hands of users before it is 100 percent compete. But really, the sooner you can test the better -- test with paper prototypes, for instance!

The primary reason for this is that it is much easier to change the game if you find an issue early in development rather than late. Once you have made the changes, you should test again. That said, it is also a good idea to make sure that the product isn't too buggy; you want to test the experience of playing the game, not the experience of crashing due to bugs.

One extreme example of this approach is the Rapid Iterative Testing and Evaluation (RITE) method, developed by games user researchers at Microsoft. In this method a test is run (usually via behavioural observation -- which will be covered in the second article in this series), and changes are made to the game as soon as problems or issues are detected, before the next participant arrives. This can occur perhaps even after just one user has been tested.

Listen to and act on problems, but not necessarily the solutions

When dealing with your users, you should be open, and listen to the problems they are raising. You can also listen to the solutions they give to those problems -- but they are likely to be less useful to you. You are the game developers, and you know what is possible with the technology, time, and resources you have. The users won't. So observe, do the research, and treat it seriously when it reveals issues, but take suggestions from users as to how to solve the issues with a grain of salt.

Games user research is just another source of data

Often when articles such as this appear online, there is much gashing of teeth and angst about taking the art out of game design and instituting "design by committee". I can understand this worry; however, much like QA, user research is just another tool to improve your game. It shouldn't dominate your design or suppress your artistic talent; rather, if done correctly, it should augment your talent and give you new insight.

Article Start Page 1 of 4 Next

Related Jobs

Infinity Ward / Activision
Infinity Ward / Activision — Woodland Hills, California, United States

Senior Game Designer - Infinity Ward
Infinity Ward / Activision
Infinity Ward / Activision — Woodland Hills, California, United States

Gameplay Animation Engineer - Infinity Ward
Infinity Ward / Activision
Infinity Ward / Activision — Woodland Hills, California, United States

Producer - Infinity Ward
Infinity Ward / Activision
Infinity Ward / Activision — Woodland Hills, California, United States

Lead Tools Engineer - Infinity Ward


Brian Bartram
profile image
Enjoyed your article, although I was surprised that telemetry/metrics weren't mentioned anywhere. Especially liked the example from MS re: the RITE method.

Ben Lewis-Evans
profile image
Hi Brian. Thanks for the comment. Metrics/telemetry (along with interviews, observational methods and biometrics) will be covered in part 2 of this feature :)

Rolf Moren
profile image
I find when I do this kind of research there are a few caveats you have to watch out for. Most of us do

- It is easy to ask your friends but they will very seldom tell you the truth about your work.
- Don't introduce research subjects to your team before they say their thing about the game. I found that the ones that got to know the developers before they tested the game where 10-25% more positive about your game.
- People like when they are asked questions about what they think about stuff, they will be more positive just because of that.

How should you handle this. Well, this is all dependant on how you ask the questions, which method of research you use and so on but you hallways have to be prepared that the reality is a few points lower than the measurement.

Jorge Diaz
profile image
This is very comprehensive. Thank you sharing it.

Paul Lenoue
profile image
Good article. You covered a lot of ground with good depth. Having had experience in focus groups from both sides of the room I just have two things to add about focus groups.
1) Be sure you get the right group to focus. If you're going for a casual browser game you don't want a room full of hard-core console freaks. Also keep in mind that holding the focus groups in the middle of a weekday means you'll end up with housewives, retirees, college students skipping classes and the unemployed. You'll never get feedback from people who have jobs and want to play fun games after work to relax.
2) You say that discussions of solutions is not wanted. Why not? If a game has a problem you _should_ ask for possible solutions from the focus group because the developers are often too close to the game to see alternate solutions, not to mention the possibility of being blind to the problem in the first place. The belief that only programmers can come up with solutions leads to bad games with fatal flaws.

Ben Lewis-Evans
profile image
Hi Paul.

In terms of solutions it is fine if they are mentioned and people should not ignore them out of hand. However, the risk that I am talking about is that focus groups can have a tendency to get fixated on providing solutions (which may or may not be worthwhile) to just a few initial issues that come up and time runs out before other (potentially more important) issues can be raised. There is also the risk that they fixate on providing solutions to a problem that is simply impossible to fix with the money, technology, or time available.

So the problem is not so much that they talk about solutions (that is fine) but more that focus groups have a tendency to do this at the expense of other information. Or as I say in the article "too many solutions, not enough issues".

If you have a good person running the focus group they can recognize when this is happening and try and guide conversation back to more fertile grounds but it is still a risk that is associated with this method simply because of the natural tendencies of groups and the relatively lower experimental control that this method provides.