Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
October 26, 2014
arrowPress Releases
October 26, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Big data and games - part 2 - algorithms and experts
by Nick Lim on 08/15/13 02:21:00 pm   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

In part 1 of the big data and games blog series, we described the characteristics of big data, and explained how big data would challenge the need for causation.  In this second installment, we will discuss how big data will affect the role of subject matter experts.  As before, we will draw on the experiences of Sonamine and the excellent book by Viktor Mayer-Schonberger and Kanneth Cukier.

Human subject matter experts?

Doctors are experts: they spend many years studying to gain a large amount of knowledge.  In their daily practice, they apply their knowledge to decisions in diagnoses and treatment, the outcome of which continue to hone their knowledge.  They have to keep up with new research and studies in order to refine their understanding of human body and diseases.

In their book,  Mayer-Schonberger and Cukier describe the work of Dr. Carolyn McGregor at the University of Ontario's Institute of Technology, studying the care of premature babies.  There, care of premees has taken on a big data like feel.  Patient data is captured in real time over 16 different streams such as heart rate, temperature and blood pressure, resulting in over 1260 data points per second per patient. 

Using big data algorithms, they have been able to predict the onset of infection 24 hours before any overt symptoms are observed.  And flying in the face of conventional treatment and wisdom, she found that very stable vital signs were observed prior to serious infection. 

Oh yes, Dr McGregor is a Ph.D in computer science, not a physician.

Human experts are critical in a small data world where people could not access enough or the right information. "In such a world, experience plays a critical role, since it is the long accumulation of latent knowledge - knowledge that one can't transmit easily or learn from a book, or perhaps even be consciously aware of - that enables one to make smarter decisions." (p.142)

What makes one expert "better" than another expert is usually the depth of their knowledge and experiences, and how often their decisions turn out "right".  Specialist doctors are considered experts in their fields; doctors with accurate diagnoses and good treatment.  It is but a small leap to suggest that more data = more knowledge = better expert. 

Big data and the new expert

Which leads to why big data sets the stage for expert systems... Where there is a lot of relevant data, both in the number of data points and the breadth of data types, that can be analyzed, big data enables a new type of expert - the non-human expert.  To take another view, a big data expert system essentially duplicates the learning and pattern recognition of a human expert, under circumstances that would totally swamp a human being.

Lest you think that big data cannot apply to creative endeavors, bear in mind that big data expert systems will work reasonably well when there is enough data points for the algorithms to glean patterns.  Consider the film industry, we now have over 30 million data points about commercial films going back decades.  The-Numbers.com crunches this data to provide predictions on the outcome of film projects.  Producers use this information in pitch meetings with investors. 

When a sufficiently large and accurate dataset of games is available, someone will apply big data methods to predict the outcome of game projects.

What does this mean for the game industry?

One management challenge here is evolve the game industry culture to embrace big data experts.  Unlike other scientifically oriented industries such as healthcare which are more open to data experts, the game industry historically has had an independent and creative streak.  It is not surprising that game industry veterans and experts decry the Zynga data driven approaches and recent corporate struggles, while secretly envying Zynga's financial success.  231M in revenues per quarter is nothing to sneeze at.

One approach is to divide and conquer: deploy big data expert systems where they shine and enhance human experts in other areas.

Management will have to decide where to use big data analysis or the human expert.  Since my company Sonamine has been leveraging big data in games, we have some experience that we can share on this topic:

- where there are more data points that are available than a human can comprehend, lean on the expert system.  These include predicting the behavior of millions of players, level progression, sales performance of virtual items and marketing campaigns, fraud, gold-mining.

- where you cannot access enough high quality data, lean on the human expert. A prime example is deciding where to spend advertising dollars. The highly fragmented user acquisition system with its many ecosystem players with siloed and non-correlated data makes it a challenge for any expert system.  Channels without tracking data such as TV ads make predictions difficult.  Simple heuristics and recent experience is what counts in a these situations. 

- where there is no data at all.  A good example is when a game designer wants to introduce a new game mechanic that does not exist.  Human judgement and small scale user testing will almost always win against a big data expert system.

Caveats and the future of experts in games

It would seem that the right thing to do would be to collect as much data as possible, since that would enable big data expert systems.  However there are downsides to that approach.  The cost and complexity to set up big data collection, storage and analysis capabilities sometimes outweigh the benefits.  I personally spent 10 years in the business intelligence industry where many companies wondered how to justify the ROI of expensive data warehouses and reporting systems.  In many cases, human experts will cost a fraction of that incurred with building a big data capability. 

I can already hear the outcry: big data will never create anything new; if we have looked at big data we would have gotten faster horses and not a car, and so on.  I agree, in fact there is another type of expert in medical science: these are the doctors that pioneer practices such as washing their hands before surgery. 

We have these experts in games too:  they are the creative game designers that invent a completely new mechanic.  They are the visionaries such as Jenova Chen, creating a game that tugs at the emotions of all players.  They are the marketers that decide to leverage live game play videos uploaded  to YouTube for viral acquisition.  They are the technical gurus that figure out how to enable players to blow up anything in Everquest Next

In my view, experts in the game industry must migrate towards being good at creating the new.  The new can be characterized by not much data being collected, unusual collaborations between different fields as well as just totally whacky stuff, such as the hyperloop. 

Secondly, algorithms can decipher the patterns; but only humans can make decisions.  And only humans can make decisions in a larger context of organization goals, objectives and targets.  As someone once told me, there must always be a fall person.  Blaming an algorithm for messing up is not going to fly. 

Conclusion

In summary, big data creates a new type of expert, one that crunches extremely large amounts of data to see new patterns.  The games industry can leverage these new expert systems to improve design, marketing and monetization. 

In part 3, the last of this series, we’ll discuss the concept of data value and apply it to games.

Reference

Mayer-Schonberger, V and Cukier K.  Big Data: A revolution that will transform how we live, work, and think.  Houghton Mifflin Harcourt, Boston, New York 2013 (Amazon link)


Related Jobs

Digital Extremes
Digital Extremes — London, Ontario, Canada
[10.25.14]

Character Artist
Digital Extremes
Digital Extremes — London, Ontario, Canada
[10.25.14]

Sound Designer
Disruptor Beam, Inc.
Disruptor Beam, Inc. — Framingham, Massachusetts, United States
[10.25.14]

Lead 3D Artist
Red 5 Studios
Red 5 Studios — Orange County, California, United States
[10.24.14]

Graphics Programmer






Comments


Isaac Knowles
profile image
I see what you're saying, but I disagree somewhat with this assessment. To determine the necessity of an expert's advice, the distinction should not be between whether I have access to a lot of data or a little data. Instead, it should be between whether I desire a prediction or an explanation for a phenomenon. Computers, fed enough data and primed with the right algorithm, are marvelous at predicting outcomes. They are, however, terrible at actually explaining why (for example) children will get infected 24 hours from now, much less why players are churning. By contrast, subject matter experts - clinicians, economists, psychologists, etc. - are equipped with the training and the statistical knowledge necessary to actually come up with and properly test valuable hypotheses about why kids will get infected in 24 hours, (or again, why players are churning). Moreover, subject matter experts are quite capable of dealing with very large amounts of data, despite reports I've heard to the contrary. For example, FTC economists run very sophisticated econometric analyses on many terabytes of data. Physicists routinely analyze petabytes of the stuff.

The way I've come to see big data in video games is this: If you want to tell someone whether Player 235772 is about to churn, or is ready to convert, then a computer with an algorithm is useful because it is good at making that prediction (assuming you have enough data). But if you want to understand what it is about your game that is making Player 235772 want to churn, you've got to ask someone who has an intuition about human behavior. In that case, computers are at best useful helpers, in that they might point to some possible hypotheses. At worst, those hypotheses are misleading or behaviorally meaningless.

In short: Reports of the demise of the subject-matter expert in a world of big data are greatly exaggerated.

David Serrano
profile image
Isaac,

Off topic but I read your profile, very interesting stuff. Added your blog to my daily read list!

Nick Lim
profile image
I hope it's clear that my perspective isn't that the subject matter expert is doomed; the use of algorithm as tools is certainly one way to go.

The churn example actually illustrates a good way to leverage the right expert at the right time. You are actually listing two different tasks : (a) guessing whether a player will churn and (b) why the player will churn. If you assign the former task to a expert database marketer and/or consumer psychologist, she will likely perform poorer than an algorithm.

For the second task, the algorithm will be worse, because it does not have sufficient accurate survey data from users; the human will do better from an intuition standpoint. The real question is what if there were sufficient and accurate survey of churned users, would an algorithm work better than a human?

Thanks Isaac for your comments!

Isaac Knowles
profile image
Hi, Nick. Thanks for your reply.

With respect to your question, I think the human will always do better, and again, I don't think that the issue is the absence of sufficient data. As you point out, humans have intuition about the behavior under study. They also have respect for the statistical properties and assumptions of the algorithm in use, which an algorithm simply cannot have.

For the sake of argument, suppose I have a huge budget and a massive alpha-testing group. Suppose that I did have data on every player, churned or not. Not just a full array of in-game variables (say a millisecond-by-millisecond telemetry of the person's activitites), but also pre- and post-game surveys that assessed employment, income, vacation plans, psychological state (!), religious beliefs, demographic info, upbringing, and a complete survey of every person connected to the player, regardless of whether or not they were players. Suppose I also included data about demographic, weather, and geological patterns near that person, as well as the service records for local utilities, and the political state of municipality, city, region, and country in which that person lives. Suppose I also know what other games the person played before, during, and after playing my game, and how long they played them, as well as what game advertisements they saw, as well as a full catalog of the items in those other games.

So suppose I have all this data - a fairly complete set, I think we can agree - and then I set loose a particular algorithm (pick your favorite - decision tree with info mined from the social network, or whatever) on the data, with the goal of predicting churn. Now, I think such an algorithm would perform rather admirably here, for there is no part of a person's life that I have not tried to quantify. The algorithm has predicted well, per my needs.

But now suppose I want to know why someone left my game. To me this means I want the algorithm to tell me whether the person left because of a particular aspect of the game, or because of a series of aspects, or because they got griefed, or whether the issue was with the person's private life. It also means that I need accurate and precisely measured estimates of the relative effect of each of those experiences on the person's probability of churning. Moreover, the explanation has to make sense, and it has to be behaviorally meaningful. Without that information, it's not worth changing the game. In other words, I need the algorithm to relate a human decision to human experience and then provide me with one or more useful recommendations.

Now, I love computers and data and regressions as much as the next guy - more than most guys, I'd bet. But they can't provide robust explanations to my customers about what they're doing wrong, or how to improve the game. Explanation requires a human intercessor. Deep explanation requires a subject matter expert, regardless of how much data you have.

Ramin Shokrizade
profile image
I'm almost totally in agreement with Isaac. We are both trained economists and statisticians but given that bias we still see domain expertise as primary, and perhaps prerequisite knowledge in predicting and explaining consumer behavior. I realize we had the discussion about why "why" was not important in your last paper, and there I disagreed fairly vigorously (I think the "why" is very important in this industry).

For instance, who builds the "predictive" algorithm that you plug your data into? Even this is not predictive, because you have to gather data after you have already made the product before you can get that data. By then it can be a bit late if you have already spent $200M and 5 years making a game (cough*SW:TOR*cough). Yea great your data now tells you that you are screwed. This saves you some money on marketing.

I get contacted by a lot of AAA companies that are building data analysis teams, and when I try to explain to them that I am a domain expert they often lose interest. Thus Nick your opinions are shared in many big companies. I think the ideal situation is to attach a domain expert like Isaac or myself to a BI team so that the team can be more accurate and more rapidly send useful interpretations to design and marketing departments. Ultimately such teams can build much smarter predictive algorithms, the likes of which do not yet exist.

Nick Lim
profile image
It will probably be useful to specify the question before we can decide whether to choose an algorithm or a human expert. If the question is "what type of game and mechanics should we build to have the best chance of success?", then given the lack of sufficient data and repeatability, a human expert will be my recommendation currently. This is in-line with your thoughts, Ramin, on consulting on the specifics of a game before it is built.

The tool being used must match the problem at hand. Most game companies I encounter who are builidng data analysis teams are hoping to use them to describe what is already happening, and make improvements to an existing game. They are certainly not intending to make wholesale changes to game design from data-driven insights.


none
 
Comment: