[In this first article in a new series, college professor and user research Ben Lewis-Evans takes a look at different methods of game user research, offering up a handy guide to different ways you can collect useful information about your game.]
This article, and its forthcoming followup, is intended to give a rough idea to developers of several different methods that can be used in games user research.
However, many, many books have been written on research methodology and I cannot cover everything. Therefore these two articles cannot be taken as completely comprehensive.
In the first of the articles I will be covering a few general points about Games User Research and then discussing three methods, focus groups, heuristic evaluation and questionnaires in some detail.
What is Games User Research?
So, before getting really started, what is games user research in the first place? Well, let's start by comparing it with Quality Assurance (QA). QA is a well-established part of software development, and is often carried out by professionals within a development team. These folks are, generally speaking, aimed at finding bugs and making sure the game runs smoothly.
However, because those working on a project usually carry out QA, it means they have an investment in it, and are familiar with it. This can cause problems when it comes to evaluating the usability and experience of a game. What seems obvious and fun to someone that has been working on a project may be completely alien and frustrating to a new user.
This is where games user research methods come in, a field that is all about the user and their experience of the game -- in particular, the big question, "is it fun?"
To really (over) simplify things, it could be said that QA and test is about the software, and how it functions when dealing with users, and game user research is about the user and how they function when dealing with the software. Notice I say just it is about how the user functions when dealing with the software, and that it is NOT about testing the user (more on that later).
So how is this done? And what is fun, anyway? Well, there has been plenty written on fun already, so let us just say that fun has quite a few dimensions. Fun can be something that is easy to use, but it can also come along with a struggle and a challenge. It can arise from an engaging experience, a compelling experience, or a relaxing one.
All this means fun is a subjective variable, which changes from person to person, and situation to situation. However, due to the emotional component of fun, what can be said is that if your players are having fun, then it is likely they can tell you about it -- if only you know how to ask.
So How Do We Ask?
Okay -- I'll get to that. But before I get into the fine details, please bear with me as I outline some general principles to keep in mind:
Get the right players
Whatever methods you choose, make sure you get the right sample. For most methods, this means getting representative users, aka the people that you expect will play your game.
If you have the time, perhaps there is some advantage to getting as wide and large a user group as possible (if you really think that everyone will want to play your game), but given that you are likely to be constrained by time (and money) it is generally best to concentrate on getting as close a sample of the target type of player you are after as possible.
The game is being tested, NOT the user
Secondly whenever doing this type of research make sure it is clear to the user that they are not being tested. The research is about improving the game, not the user, so the user shouldn't be made to feel inadequate if they can or cannot do something. In principle it is all valuable information.
This can be hard to do sometimes, as you are the ones designing the game, and it can be uncomfortable to hear others criticising it. But try your best to not be defensive or judgemental (i.e. avoid thinking the problem exists between the keyboard and chair.)
What do you want to know?
Whenever doing research, you should be clear about what you want to know. You will be playing the game yourselves as you work on it, and you should have design documents, so you know how things are supposed to work. So don't just plonk people down with the game and come up with questions on the fly. Work out what areas you think might be problems and know what you want to ask about before it's time to test.
Please note: I am not saying you go out there with preconceptions and already "know" the answers you want; rather I am just making the point that you should be at prepared and know what you're looking for. Otherwise, you get a mass of data that may not be of any use to you at all. At the same time, be open to surprises. You never know what might pop up.
Test early and test often
The next point is considered one of the most important by user researchers, and that is to test as early as you feel it is possible to do so. This can be difficult, as it feels bad letting your baby out into the hands of users before it is 100 percent compete. But really, the sooner you can test the better -- test with paper prototypes, for instance!
The primary reason for this is that it is much easier to change the game if you find an issue early in development rather than late. Once you have made the changes, you should test again. That said, it is also a good idea to make sure that the product isn't too buggy; you want to test the experience of playing the game, not the experience of crashing due to bugs.
One extreme example of this approach is the Rapid Iterative Testing and Evaluation (RITE) method, developed by games user researchers at Microsoft. In this method a test is run (usually via behavioural observation -- which will be covered in the second article in this series), and changes are made to the game as soon as problems or issues are detected, before the next participant arrives. This can occur perhaps even after just one user has been tested.
Listen to and act on problems, but not necessarily the solutions
When dealing with your users, you should be open, and listen to the problems they are raising. You can also listen to the solutions they give to those problems -- but they are likely to be less useful to you. You are the game developers, and you know what is possible with the technology, time, and resources you have. The users won't. So observe, do the research, and treat it seriously when it reveals issues, but take suggestions from users as to how to solve the issues with a grain of salt.
Games user research is just another source of data
Often when articles such as this appear online, there is much gashing of teeth and angst about taking the art out of game design and instituting "design by committee". I can understand this worry; however, much like QA, user research is just another tool to improve your game. It shouldn't dominate your design or suppress your artistic talent; rather, if done correctly, it should augment your talent and give you new insight.