Gamasutra: The Art & Business of Making Gamesspacer
Part 1: The End of the Dark Ages for QA in game development
Printer-Friendly VersionPrinter-Friendly Version
arrowPress Releases
April 18, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM TechWeb sites:


 
Part 1: The End of the Dark Ages for QA in game development
by Tulay Tetiker McNally on 04/09/13 03:17:00 pm   Expert Blogs   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

Teams that blame QA for not finding certain bugs, live in the dark ages of development, because they don't understand that quality (or the lack of it) is the responsibility of the entire team and that QA is only one of the groups contributing to this effort. 

One of the roles of QA is to lead the road to quality by influencing the whole team to put quality at a higher level and by making sure the right tools and processes are in place to enforce quality throughout those processes. 

To be in a position to do so, you need to be empowered by your studio’s leadership and development culture and have the trust and backing from developers and producers.

In the very early days of game development, typically debugging was a developer’s responsibility. About 15/20 years ago it started to become common practice to bring on one or two testers who would test games from the end-user’s perspective.

With the introduction of gaming consoles, game development began to become more complex. Publishers realized that they needed to invest in a larger pool of testers which then became known as Quality Assurance.

Today, a lot of publishers have large pools of testers whereas many developers still keep a smaller team of highly skilled development testers around that are both technical and game savvy. 

Testing was romanticized as “playing games all day and getting paid for it”. It is often seen as a “foot in the door” job. Unfortunately this has at times given QA a bit of a negative reputation within some areas of the games industry.

Fortunately we’re working towards changing the perception of QA and the current industry trend of focusing on fewer but higher quality games helps with that. 

Changing Hiring Standards might also influence this change of perception. I think, or I hope, that those days of advertisements about “playing games all day and getting paid for it” will be history soon.

I just recently read an Interview with Zappo’s CEO about how bad hiring decisions cost his company nearly $100 Million. It makes no difference how large or small an organization is or if you are hiring an entry-level employee or an executive, in Art, Programming or QA etc. 

Our hiring standards at BioWare are high. Our development disciplines attract top talent to work on genre-defining AAA-titles. In QA we set ourselves the same goal. If you want AAA-QA, you need to hire the right people with the right mind-set and the right skill-set throughout the ranks.

Even for our temporary/entry level positions every candidate has to go through a rigorous pre-selection and interview process that have helped us to identify good QA talent. 

I believe that the games industry is undervaluing or even underusing QA in many scenarios. This often leads to frustration among QA professionals who are taking testing seriously, but also feel powerless that their insights into the game are not heard; 

QA traditionally has a high turnover in staff, because it is usually not seen as a career destination within games development. This inevitably leads to loss of knowledge and skill set. High turn-over of staff doesn’t help QA as a department or discipline to mature.   

I also believe that there are still enough folks out there who are under the impression that you can just hire testers off the street, put a controller in their hand and pay them peanuts for "playing games all day" and are then surprised when the output is not as valuable as they expected.

You also often end up with people working on your product who have no personal investment in its success. QA is not easy; but it is rewarding, fun, creative and analytical!

In a good organization, testing is involved in the development process from the outset - from vision & concept over to pre-production, full-production, finaling and post-release support.


Quality Management is done at every point in the game development cycle                                                                                       

At BioWare we involve QA early on to consult with the project teams on the testability of features, identify things that might be hard (or impossible) to test, help develop test approaches in advance of anything being built, raising potential risks and providing feedback from a consumer perspective. 

Quality Assurance at BioWare means working embedded with developers during all phases of the development cycle as an integral part of game development;
Mind you, being part of the scrum team doesn’t mean our QA team is being managed or tasked by the scrum team. This happens at the QA lead level.

Over the last 13 years we have developed a system that supports development in a very efficient way. We gained the trust and respect of the development teams by proving our value. But it was not always like that...

Probably up until Baldur’s Gate II, most QA work was reactionary. The core QA team, thought it would help them to better understand how to test the game if they understood how it was made. 

So they asked designers to teach them and they learned how the toolsets worked and used their friendly relationships with developers to learn more about development and to have more visibility into the development process. Neverwinter Nights was probably the first big turning point for BioWare.

The QA team at this point had started to make test levels using the NWN toolset to test content such as animations, art, assets, etc. - they were testing in a controlled environment and the QA team started to earn the development team’s trust and the perception of QA in the studio started to change. 

During Knights of the Old Republic, the QA Lead at that time (who incidentally is still with BioWare), had started to attend the designer and programmer meetings to get information pro-actively and share it with the test-team and it was during KotoR that the QA team was split into two disciplines - Tech (which generally supports programmers) and Design (which at that time was supporting mostly designers and writers).

The reason for the split was primarily for tasking purposes: the QA staff wanted to split up the developer support work amongst each other and went by their preference/talent for one or the other. 

Jade Empire had established the QA department as part of development and for the first time, QA was invited to planning meetings from the start…. and the rest, as they say, is history :)

Now I understand that not every studio has such a long tradition of working closely together with QA. Remember we’ve been doing this for over a decade. Establishing trust and building relationships with development and production and delivering results had an impact as well on something so basic as our company’s core values and our studio culture. 

At BioWare, we are fortunate that our Core Values and our Studio Culture enable and foster team collaboration where every discipline is seen as equal. QA has a voice, QA is embedded, QA is a truly integral part of development.


Related Jobs

Hasbro
Hasbro — Pawtucket, Rhode Island, United States
[04.18.14]

Sr. Designer/Producer, Integrated Play
2K
2K — Novato, California, United States
[04.16.14]

Web Producer
Treyarch / Activision
Treyarch / Activision — Santa Monica, California, United States
[04.16.14]

Associate Producer - Treyarch
Treyarch / Activision
Treyarch / Activision — Santa Monica, California, United States
[04.16.14]

Production Coordinator (temporary) - Treyarch






Comments


Addison Martinez
profile image
So in other words, QA should have an open line of communication with designers and programmers throughout the entire development.

Also, do you know where I can find the article you recently read interviewing Zappo’s CEO.

Thanks.

Look forward to the next entry.

Eric McVinney
profile image
This. All of this is what I've been thinking and talking about ever since I got into QA.

Great article :)

Tulay Tetiker McNally
profile image
Thanks everyone!

@Addison: The video with the interview is here
http://www.inc.com/allison-fass/tony-hsieh-hiring-mistakes-cost-z
appos-100-million.html?goback=%2Egde_4349457_member_197319636

I will publish the next article with the title "QA as a career destination? Why not!?" on April 15th! Stay tuned :)

Addison Martinez
profile image
Thanks Tulay.

Dan Johnson
profile image
"QA traditionally has a high turnover in staff, because it is usually not seen as a career destination within games development."

It's not seen as a career destination because it's not compensated as a career destination. QA are paid $10-15/hr with no benefits and laid off at the end of the project. That's not the way to build a competent, invested team, but it's the route taken by all but a very rare few studios. And it's a shame, because by the time a tester becomes truly competent, they've either learned enough to get out or they simply get laid off.

Bob McCabe
profile image
BioWare, of course, continues to be the exception on this front. I worked at BioWare from 200 through 2007, spending a great deal of time in QA -- and while there I received full benefits, a friendly salary, and some significant bonuses. But, in general, you're right. And as everyone who's in agreement with the article should know, that's a shame.

Kenneth Banadyga
profile image
Wonderful article!

I have worked at/with companies that are truly in the Dark Age, and I honestly can't see myself ever going back to those times.

It will be interesting to see with the rise of the mobile gaming sector just how many young companies learn and adapt their Q.A. teams within development. Especially now with Mobile Games having such quick and rapid development cycles that demand Q.A. to be on the ball before and after the game goes live.

With Mobile Games needing to have such quick and rapid development cycles that demand Q.A. to be on the ball at every turn, I truly believe that traditional/dark age practices will not work in mobile. The communication between disciplines need to be so in tune with each other that Q.A. HAS to be involved.

Now that some of the bigger consoles/AAA publishers and developers are entering mobile development, it will be exciting to see how they adapt and keep up with other established mobile developers.

Cheers and here is to the future of Q.A.!

Harry Debelius
profile image
"I also believe that there are still enough folks out there who are under the impression that you can just hire testers off the street, put a controller in their hand and pay them peanuts for "playing games all day" and are then surprised when the output is not as valuable as they expected."

Sadly, that is still the case in many QA placements. I don't think, as Dan Johnson mentions, that pay is a definitive factor for this. At least for me, the main problem for a professional QA is the struggle for a permanent job instead of continuous temporal contracts.

I've had the possibility to work closely with developers and it's a huge improvement. The developing team look at QA as a useful asset and the QA themselves feel more involved and influential. Dedicated QA teams that only contact developers via bug reports generally provoke some disdain in the development team. Of course, that is also caused by the mentality in the quote.

Very interesting read.

Adam Bishop
profile image
I think the pay is a huge factor. Unless you're very young and have no obligations, QA pay isn't enough to live on. Certainly not enough to live on long enough to plan for it to be a career.

Kieran Wasylyshyn
profile image
Great article. I was hired just yesterday for my first official games industry job as a QA Tester, and I'm looking forward to seeing how things stack up to this article. Looking forward to the next one as well as my first day at work. :)

Thomas Petersen
profile image
"To be in a position to do so, you need to be empowered by your studio’s leadership and development culture and have the trust and backing from developers and producers."

Totally agree and it is a catch 22. When management don't know anything about "real" QA, they are not going to appreciate a professional manager coming and making demands for a big budget. Good quality will be a big post in your budget, but it is much cheaper than having a bad and bug ridden product in the market. This is the reason why I always do as much interviewing of a potential employer as they interview me for their position, because I can't succeed without the proper management climate behind me.

I made a blog post about developer-to-tester ratio and how I ended up with the department size we have in Unity. For anyone with a lingering senior management, you might want to give them this link: http://www.datalord.dk/developer-to-tester-ratio-part-1.html

Jason Cuffley
profile image
I have been apart of a QA team and actually about to start that again. I believe QA is an integral part but this article hits the nail on the head info. on how QA should be. GREAT STUFF!

Stuart Crocker
profile image
Excellent read thank you!

This is exactly the approach I've been building up within our organisation and in previous ones. Integrating Test or QA throughout the software/games development lifecycle allows the team to, in some cases prevent issues, provide greater analytical evaluation of all aspects of the game in a timely manner and also build up a relationship of trust with the development and production teams because they are seeing real value being added - no longer are we seen as an unwelcome cost to the project.

As was said, it's so important that you hire excellent people to fill these types of roles in order for this approach to be successful.

Looking forward to part II!

Fernando Nicolino
profile image
It is good to see things are changing on QA. At least in some studios.

Even working as a DevTester, right next to the programmers I would see some of them giving a simple check alone, as if they didn't needed QA help.

Phillip Derosa
profile image
Good article, thanks for sharing these experiences and helping dispel misconceptions around the profession and value of software testing.

Here is a excerpt from Code Complete by Steve McConnell, "Testing by itself does not improve software quality. Test results are an indicator of quality, but in and of themselves, don't improve it. Trying to improve software quality by increasing the amount of testing is like trying to lose weight by weighing yourself more often. If you want to improve your software, don't test more; develop better."

Randen Dunlap
profile image
Great article, having personally worked in polar opposite QA environments, very similar to what you described, I can personally attest to the disparity between the two. A large reason for the persistence of the negative stigma about QA being "Get paid to play games all day!" is mostly due to the 3rd party staffing agencies, tasked by large publishers to corral fresh recruits, whom spew this message repeatedly like a mantra (or something very similar).

The staffing agencies have no personal vested interest in the product, publisher, or the industry so they'll use whatever methods they can to get the cheapest labor possible, as that is their bread and butter. When developers/publishers do the recruiting/hiring themselves, the difference is a tangible one. Just do some research and take a look around, the studios that break down the walls between QA and development, while intermingling the two, see vast improvements to company culture and efficiency.

Thanks for the article.

Matthew Buxton
profile image
It's really good to see people who value all the disciplines equally!

I do think some of the disdain may come from the fact that Developers sometimes get wary of talented testers that move into dev, I've seen that they don't get the same pay if they go that route as opposed to starting in Dev. It's a bit of a problem when you have an entire team wanting to get out of that team!

Robert Grant Stanton Sr
profile image
If enough time passes, everything is new again. Back in the 60's and 70's, the electronics industry began embedding QA professionals with product development teams and the results were amazing improvements in manufacturing and process. These new professionals became what was called Value Engineers. Very apt labeling, don't you think?

Dan Robinson
profile image
Thanks for sharing your experiences in QA! I am looking forward to learning more about how QA plays a role in the game development process. Out of curiosity, do you see people entering QA in game development after spending some time in QA at other industries (i.e. computer system validation)?

Matt Cratty
profile image
The biggest shift in attitude that needs to happen is that developers need to look at QA as part of the development team and not irritants. Few simple rules:

1. THANK them when they find a bug that you're going to have to stay late to fix.

2. Ask them how you can make THEIR job easier. They complain about us nearly as much as we complain about them.

3. Once you've done 1 and 2, you can successfully get them to help you make your job easier.

Chris Buzon
profile image
@Dan - some of the QA at BioWare have experience in QA/QC outside of gaming, and others (like myself) come from fields that are completely unrelated to software/gaming/QC. Your work background is largely irrelevant - what we look for are people who are able to break down problems into manageable pieces, people who can work well in teams and on their own, people who are willing to learn new skills (many are very specialized and can't be learned in advance), and people who have the natural desire to ask "What happens if I _____?"

Tulay Tetiker McNally
profile image
I'm happy to see that this article kicked off good discussions already. I hope that by sharing best practices and our BioWare QA philosophy we can help initiate change and you can hopefully take this as an inspiration.

Some points that were made here about salary and QA as a career will be addressed in future articles of the series. We have equal and competitive salaries in QA. We had people in the past who switched department within the studio from Programming to QA as a career choice for example.

Here is a preview to future articles of the series:
Part 2: QA as a career destination (April 15)
Part 3: BioWare QA a culture of integrity, problem solving and fun (April 22)
Part 4: “Adopt, Adapt and Improve” – Agile QA BioWare Style (April 29)
Part 5: QA An Integral Partner in Development (May 6)

Seokjun Jin
profile image
This is a great article which I've never met. Also rest of this series would reach my expectation as well. I would like to introduce your articles for our QA fellows in Korea. May I get permission to translate into Korean and posting on my own blog? So far I've posted several translated articles about QA and software testing for introducing qualified information to our QA guys. It will be strictly informational, not a commercial purpose.

Thanks,
JIN

Jason Guyan
profile image
This is a great article. I can only hope that the practices mentioned here become more slandered throughout the industry.

Victor Romero
profile image
Good article, Tulay. I attended your GDC talk as well. Having worked in several test groups—including EARS—I agree with your belief that empowerment and integration into the development team are essential to making game test a career destination and not just a stop on a road to elsewhere. I'm seeing the changes as the years pass. It's not a trek without its bumps and bruises, but it's getting there. New testers are coming into environments which are much improved over that which I first experienced many years ago. Organizations that still cling to the old ways are, as you say, in the dark ages.

I also agree with comments about wage as an important factor in game test as a career. I look forward to the article in which this is discussed.

Gary Masnica
profile image
This is a great start to what looks to be a really interesting series. We talked a year or so ago at CAST about how you try to set up QA at Bioware. I just had to talk to you though, partially because I was shocked to see someone from the game industry actually at a testing conference. I hope that shock disappears, and it becomes more normal to see people making software (games) and testing for a living actually taking an interest in the vast (and really awesome) testing community that is out there.

Jason Chen
profile image
Great article! that is definitely a better way to enhance QA effectiveness

I currently lead a group of young QA testers, i have no luxury of hiring experienced QA. One thing I learn to make QA more effective is to let them ask the designers/engineers on gameplays/ level designs that they do not understand, establishing a communication with the core team is very important. now working to get to right window for bug reports.

Thomas Petersen
profile image
Why do you not have that "luxury"? Management won't give you enough budget?

You need to change the attitude towards QA by making demands fit for an equal discipline. The first place change starts is with you, because no one is going to come and offer you the budget for a group of high cost seniors. Brutal reality.

Carlos De La Torre Jr
profile image
I love the QA discussion going on here and I look forward to reading the rest of article series Tulay.

The two main reasons IMO why it is difficult to have an overall effective QA department is Pay rate and the ability to get a perm position in a QA department.
The low pay rate results in seasoned QA testers leaving QA testing for postions outside the department or leaving the game industry all together. And even if you do get a perm position the pay rate is still low. I've had friends leave the game industry just for that reason. Why do QA in games when I can be making twice as much in software field testing.

I really wished more developers would value QA as it seems Bio Ware does.

Christopher Thigpen
profile image
As a Producer of games both single and multiplayer, I see QA as the lifeblood of the product. I welcome QA into all phases of development. The insight is very valuable. Unfortunately, as stated above, sometimes QA is seen as the bastard of production. QA and analytics should be a focus of all developers from the onset of their product phases.

To me, it is complete common sense.

Great article. I am looking forward to sharing it with my teams.

Thanks again.

Paul Boyle
profile image
I guess I'll ruffle a few feathers here, and say that while I find some of this article just fine, particularly things like trying to improve QA processes, having better screening & hiring practices for QA, the opener of this article is really blowing smoke up people's butts, and I'm a little dubious about subsequent articles.

"Teams that blame QA for not finding certain bugs, live in the dark ages of development, because they don't understand that quality (or the lack of it) is the responsibility of the entire team and that QA is only one of the groups contributing to this effort. "

Yes, game quality is the responsibility of the whole team. But that's not connected to not finding key defects. If key defects are not found, that's QA's failure. WHY that failure happened, through lack of resources or whatever, may be because of lack of funds or training or staff, but THAT it happened, is because the QA department didn't accomplish their primary responsibility - to find and fix defects.

"One of the roles of QA is to lead the road to quality by influencing the whole team to put quality at a higher level and by making sure the right tools and processes are in place to enforce quality throughout those processes. "

Down this path lies QA that's gotten too big for its britches and doesn't want to do the job they are intended to do (see first statement). QA's job is not to evangelize quality. That's up to the developers and the publisher, since QA isn't involved in a lot of the decisions that determine quality - cost, release date, staffing, design, art direction, etc. And there really doesn't need to be another cook at that table. If your production team or your publisher doesn't believe that higher game quality is what the project needs, then you can whinge at them, but noone's going to hire you to whinge about it.

I know it has always been QA's wish that they get to say "this can ship with this bug or not". I'm sorry, but that's just never going to happen, nor should it.

If you actually want QA to perform better, and be compensated better for it, then try to give more concrete evidence that better training and salaried people genuinely do produce a more efficient QA department. I'd agree with you, but you're not backing anything up here. And Jade Empire is clearly not a good example - it was not one of Biowares better titles, and while I don't remember it being buggy, the major thing it was dinged for was game length, and therefore it's a case where resources should have been spent on developing a longer game, not polishing a short one. It's exactly the example you DON'T want to use.

Artur Foxander
profile image
"Down this path lies QA that's gotten too big for its britches and doesn't want to do the job they are intended to do"

Sorry if I am offending you (I really don't mean to), but you seem to have quite narrow definition of QA. QA can be used for bughunting only, but they shoud be used as Quality Assurance. Finding and fixing balance, learnability issues, memorability issues, efficiency issues and flawed interface WILL ALSO improve the quality of the game.

"since QA isn't involved in a lot of the decisions that determine quality - cost, release date, staffing, design, art direction, etc."

I Agree. But QA testers are not asking for this. It's producers responsibility. Experienced QA testers are just asking to develop tools and routines for improving the Quality of the game, not planning or redesigning it.

"QA's job is not to evangelize quality."

Agree again. But that's not what we are talking about. Programmers' job is not to evangelize programming either. (And they usually don't do that)

Johnny Skwirut
profile image
This is fantastic, I look forward to reading more :)

Joshua Peizer
profile image
Thank you Tulay Tetiker McNally,

This is helped solidify my own thoughts about how testers and developers should interact with each other. I am currently attempting to get my company to inherit a lot of what you mentioned above. I cannot stress enough how important it is to have a mutual understanding of the application from a development standpoint and a testing standpoint to get the best results.

I will personally attempt to work a few of your words into my everyday work environment and with any luck a seed will be planted and will flourish with a new idea of working together as one.

Again, thank you and keep up the great work!

Cheers,
Josh

Shawn Stevens
profile image
It is a great article with valid points. However do not delude your thinking in regards to the kind of people that want to get into QA. For example there are plenty of people young and old that would like to get into QA but have a different background in QA than computer software. Albeit there are also those of us, like myself, that have been Beta-Testers for many years that would love to get into QA as a career, however it seems that most Gaming Companies would rather stick with the part-time testers than those who are wanting a full time job with experience.

I know I personally would love to work in the computer game industry as a full-time QA Analyst because I actually enjoy looking for bugs and glitches in the game. Because for me personally I want to make sure that the game is the very best it can be when it ships to the consumers.


none
 
Comment: