Gamasutra: The Art & Business of Making Gamesspacer
Best Practices: Five Tips for Better Playtesting
View All     RSS
October 21, 2014
arrowPress Releases
October 21, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Best Practices: Five Tips for Better Playtesting

January 23, 2013 Article Start Previous Page 2 of 3 Next
 

2. Test Your Test Before You Test!

Before your testers arrive, set aside some time for everyone on your team involved in the playtest to do a run through. Testing your playtest will ensure things run as smoothly as possible, by ironing out the kinks ahead of time and getting the whole team on the same page.

  • Test your player's experience from the moment they enter your door. Your playtesters should be greeted by someone when they enter or find a sign-in sheet. Every detail matters to create a comfortable experience.
  • You may have a state-of-the-art usability lab and great refreshments, but it won't matter if the building's front door is locked! Is all of your technology working? Check your speakers, mouse, keyboard and internet. You don't want to have to call IT during a playtest because you forgot to install a plugin that your game requires to run.
  • Brief all playtesting helpers on your process. At Arkadium, the whole team can get involved in playtesting. Everybody involved in the process is briefed that day on the goals of this playtest, what we hope to learn and the process we're using.

3. Take the Pressure Off

The concept of playtesting is still new to some game developers; for the average human being, it is completely foreign. Your playtesters may be unsure of what to expect, and that uncertainty can have a negative effect on your test results. Make your playtesters comfortable so they can focus on playing your game and you'll get much better feedback.

Be sure to tell your moderators to segregate themselves from the game during the test. We assure players, "We're testing the game, not you," and that we are hoping to find areas where they struggle or get confused, because we want to improve the game. We don't want our players to be afraid of giving honest (negative) feedback, so we always stress that we are observers, not designers, and are here to listen to any and all criticisms.

Our observers sit back and let the playtester play the game with very little interference. However, we find it helpful to encourage the playtester to think out loud during their experience, to help us understand their point of view.

Recording the test for the rest of the game's team to watch later is also helpful. Because most people don't play video games in front of a note taker and a video camera, we find it's important to put a player's mind at ease about these elements and explain to them why they are helpful. Players usually enter our office not knowing what to expect, but leave understanding what a playtest is and why we conduct them.


Article Start Previous Page 2 of 3 Next

Related Jobs

Filament Games LLC
Filament Games LLC — Madison, Wisconsin, United States
[10.20.14]

Quality Assurance Associate
InnoGames GmbH
InnoGames GmbH — Hamburg, Germany
[10.20.14]

Team Lead Online Marketing - TV (m/f)
Yoh
Yoh — Vancouver, British Columbia, Canada
[10.17.14]

Build & Test Engineer
Nix Hydra
Nix Hydra — Los Angeles, California, United States
[10.16.14]

Programmer






Comments


Andre Gagne
profile image
I agree with your best practices as they are mostly from the ISO standard on usability testing. I do have several other questions and comments though:

Observation: It appears that you are combining survey/attitude methods and observational studies (having a survey but then using a think-aloud protocol?). Doing both at the same time adds confounds that threaten the validity of your results.

Question: Also, do you ask questions about the UI in your surveys? I suspect that you are getting most of that data from the observers.

Question: How many participants are you running at a time? it would seem that you are either taking a long time to run a study or have too few participants for any statistics ran on the surveys to have reasonable margins of error.

For anyone interested in this, I would suggest another Gamasutra article that was written a while back by a very good researcher in collaboration with the Games User Researcher SIG.

Part 1:
http://www.gamasutra.com/view/feature/169069/finding_out_what_the
y_think_a_.php
Part 2:
http://www.gamasutra.com/view/feature/170332/finding_out_what_the
y_think_a_.php

Good Luck!

Vin St John
profile image
Great points/questions, Andre. I'll try to address each one:
1) On combining survey and observational methods for collecting information - we use both or only one or the other depending on the situation. Often times a survey is not needed at all, since most of the questions we have can more easily be answered just through observation of player behavior. There are many cases where a survey question will not produce helpful results, as well. When we need to rely on a survey, it is generally only presented to the user after they have played the game and all of our observational notes are recorded. There is still some room for improvement here, but we find that this works for our purposes.

2) Most UI questions are answered through observation. Sometimes we will confirm our observations by presenting the user with a screenshot (in a survey) of the UI and pointing to different elements, asking for a description of what that element is for. This helps us understand if the player knew what they were doing because of the great UI, or if they figured it out without the UI's help. It also helps identify ways that our UI is misleading, i.e. when a player sees a running clock and thinks it represents "time left" instead of "time elapsed".

3) In a single session it is rare for us to exceed 10 playtesters. Because our games are intended to be played for short session lengths, each player usually only plays for about 30 minutes. The data we collect is useful for identifying patterns, but not statistics. (For statistically significant data, we rely on post-release analytics in a live environment with many thousands of players). We consider playtesting to be part of the design process - playtesting requires the intuition of game designers in order to determine whether the problems identified are significant and how to best solve them.

Thanks for sharing the additional resources, it's a great read! We're constantly trying to improve our process, so if you have any criticisms or suggestions I would welcome them. Thanks!

Trevor Cuthbertson
profile image
"We also have a policy against recruiting friends of our employees for playtesting."

This is the greatest advice of all -- the gold of playtesting! Don't hire your friends and family.

Thiago Appella
profile image
Great Article, Vin. Congratz.
Just would like to reinforce 2 of your tips as they are really important in my opinion, but sometimes not followed correctly or without the proper attention.

1) Recruit the right target.
Every product has your own target, sometimes the audience is really broad but still there are some shared elements and requirements that you need to fit while recruiting for playtesting.

2)Group your data.
Don't change your game based on only one play test/feedback. As a more in deep analysis based on aggregated data could show you it was not a pattern across that session - sometimes you just got a person that was not actually on the target you were looking for.

Vijay Srinivasan
profile image
Very good read, thanks a bunch !

Ian Hamilton
profile image
There's something critical missing from the 'always consider when recruiting' bit -

Disabilities.

Regardless of what segment you're looking at, people with disabilities (visual, motor, cognitive and hearing impairments) will account for a huge chunk, so you need to represent them in your recruitment profile.

Numbers depend for the most part on age. At the usual target audience age range for games it's around 15% for what's commonly regarded at disability, with another 8% of males who are colourblind, and 14% who have an adult reading age of below 11 years old.

And that's just amongst the general population, disability is actually more common (20% Vs 15% in PopCap's research) amongst gamers than in the general population.. PWDs have all the same reasons to be gamers as anyone else, plus extra reasons such as limited recreation and social opportunities, as an alternative to pain relief medication, and so on.

If you're testing with kids the numbers are smaller, with visual and hearing impairments in particular pretty rare in children (they're more commonly caused by deterioration accident or disease, none of which have had much chance to happen by the time you're 5 years old), so it's more motor and cognitive impairment that you want to test with for them.

If you're aiming at people who are older it increases pretty rapidly, the 15% becoming 50% by the time you hit 65.

Really helpful conditions to recruit for are colorblindness, dyspraxia and dyslexia, but really if you can just manage to recruit even one person from each of those four top level groups (motor/cognitive/hearing/visual) then you'll get some incredibly useful feedback.

Vin St John
profile image
Ian, these are some really great points. Thanks for sharing (and for the statistics to back it up).

Ian Hamilton
profile image
I'd be happy to chat more about in person if you're interested, will you be at the GUR summit?

Vin St John
profile image
Not sure yet, will let you know! My Gmail is "vinstjohn" if you would like to get in touch about it before then.

Ian Hamilton
profile image
Also I completely agree with the answers to the questions.

You have to test interfaces primarily through observation, as what people say and remember can be very different to what they actually do. There are some things that it can be helpful for asking questions on, if you're specifically looking for feedback for lasting impressions or emotional engagement, but you can't really ask questions after the event about how usabable specific elements of areas of the interface were.

Again in agreement the only way to get statistical significance is to run analytics post-launch, but that doesn't help you when you're in the early stages, small observational studies have lots of value for that.

The statistical significance thing is something that's really critical and often not understood, with people mistakenly believing that what they've seen in a small sample testing session is 'proof'. You need to be aware that not only are there small sample sizes but also an incredible number of uncontrolled variables, it's about as inexact a science as you can find. So instead it needs to be treated for what it is, a way to gather anecdotal suggestions of areas that could be worth looking into.

The way to mitigate against the inaccuracy is simple enough - test early and often.

Virgile Delporte
profile image
Very interesting article, with great tips. I would add something complementary: "Playtest early". Indeed, while playtesting prior to release is essential, providing playable prototypes as early as possible to a carefully targeted group should validate / invalidate / orientate future milestones within the game development process. Same methodologies as listed in your article - also one step ahead.


none
 
Comment: