Gamasutra: The Art & Business of Making Gamesspacer
Predicting Churn: When Do Veterans Quit?
arrowPress Releases
July 30, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Predicting Churn: When Do Veterans Quit?

August 30, 2012 Article Start Page 1 of 3 Next
 

In the follow-up to his original article on predicting player churn, Dmitry Nozhnin, head of analytics and monetization at Russian MMO publisher Innova, shares his methodology for predicting when veteran players will quit the game -- identifying when players will drop two to three weeks before they do with 95 accuracy, all carried out in the live environment of the Russian version of NCSoft's Aion.

In my previous article, I showed the process we developed for predicting churn of our freshest users, who just registered for the game, based on data collected during the first couple of days of their adventures. However, on the other end of spectrum are seasoned gamers who have spent months and months in the game, but for various reasons decided to abandon it. Predicting their desire to leave the game is possible, and in this article, we're sharing our data mining methodology.

Tech Side

Nothing changed from the first data-mining project; we were still on two Dual Xeon E5630 blades with 32GB RAM, 10TB cold and 3TB hot storage RAID10 SAS units. Both blades were running MS SQL 2008R2 -- one as a data warehouse, and the other for MS Analysis Services. Only the standard Microsoft BI software stack was used.

Our dataset had up to six months of recorded gameplay for about 38,000 veteran players.

Defining the Moment of Churn

For new players, defining churn was dead simple -- they just leave the game after a couple of minutes or hours. That's it. The last day of play was clearly defined, and data mining models on such churn factors were already well established. However, for veterans, it took us several iterations to define churn correctly. Our first assumption was this: the player is enjoying the game for some time, but then he decides to quit and leaves. Marking his play days with green, we expected something like this:

Our guess was that defining the churn point would be straightforward -- the last game day. The reality, however, was more complex; the majority of players behave like this:

Is August 25th, when we've seen the player for the last time, the churn point? Or in fact August 16th, the day we hadn't seen the player for seven consecutive days? Or July 31st, the first time she hadn't launched the game for more than seven days? We tried several hypotheses, and the simple ones didn't work out. Defining the churn in a simple way -- predicting that a particular play day will be the last one -- resulted in unimpressive 65 percent precision.

Manual data investigation revealed that majority of churners have a "long tail" of play days -- those occasional activity days during several weeks, or even months, as shown on the second calendar example. They effectively stopped actively playing the game, but still log in from time to time. In fact, they had already quit; occasional logins are for auction sales, random chats, or probably indicate that the account has been passed on to guildmates.

The next step was to cut off this tail using some empirical thresholds in order to trace back to the day when the player's activity decline started. The most effective query was something like "the last day of play when total game days for the previous 30 calendar days were fewer than 9". Still, the precision was under 80 percent, and empirical rules didn't work for loyal but very casual players.

Redefining the Moment of Churn

Key success factor of this project was reframing the moment of churn from "the player has left the game" to "the player's activity has dropped below the churn threshold". We already store and widely use the Frequency metric, defined as "days with game logins in last 30 calendar days". In short, it means how often player has been playing -- every day, every other day, on weekends, or just a few days a month. We segment players according to their play frequency:

The next step is redefining churn as they fall into The Pit, an area of extreme inactivity with very high probability to churn. This idea really makes sense from business point of view -- instead of detecting churners the day the leave the game forever, we're now focusing on early detection and prediction of disinterested players, and have several weeks to incentivize them to keep playing.

The new approach was to predict players who will fall down into The Pit in two weeks for 7-9, 10-15, and 16-20 cohorts, and in three weeks for the 21-25 cohort. So we're looking for players who are losing momentum, and whose activity will drop significantly over the next several weeks:


Article Start Page 1 of 3 Next

Related Jobs

Turtle Rock Studios, Inc.
Turtle Rock Studios, Inc. — Lake Forest, California, United States
[07.30.14]

Technical Artist - Turtle Rock Studios
Nordeus
Nordeus — Belgrade, Serbia
[07.30.14]

Senior Game Designer
Technicolor
Technicolor — Austin, Texas, United States
[07.30.14]

Core Systems Engineer
CCP
CCP — Newcastle, England, United Kingdom
[07.30.14]

Senior VFX artist






Comments


Curtiss Murphy
profile image
I played Aion. Well, actually, my entire family played. The graphics were beautiful, combat was interesting, and the wings. ... Ahh... the wings! My kids loved the wings. They'd setup little races, to see who could glide the farthest ... Such fun!

We were having a blast: leveling up, running quests, and farming for items. And, then, one day, we weren't. We realized it would take 900 combines to craft a single nice item. Or that we each had to raise 5 characters to level 30 to get good gear. PvP zones was just a school-yard bully fest. The illusion was broken. It went from 'fun' to ... a LOTTA work. After that, I couldn't get into flow, no matter how hard I tried. A few weeks later, we quit: 4 churns to add to your database.

Here's the science: game play is about flow, which requires: 1) clear goals, 2) immediate feedback, 3) no distractions, and 4) a challenge balanced uniquely to their skill. All 4 are required!

On my 'prediction day', I realized that success would take too long. It wasn't hard, it just felt like an impossibly long commitment. At the same time, I realized I would never be good at PvP without that investment, so I lost sight of my goals. The feedback was showing me goals would take longer and longer to achieve. Crafting was a big distraction and there were too many options (another distraction). That's all 4 requirements: flow became impossible.

As one of the comments discussed in your previous article, most of your players will quit. But, once you have a player that is sticking with it, their behavior has to be in the data! With your wonderful predictions, you know WHEN player churn begins, so figure out WHAT players were doing in the several days prior. This will show WHY they lost flow and give insight to how to fix the game. Look for: recent changes in crafting activity; character making; a series of failures/deaths; leveling/xp/gold plateaus.

For what it's worth - incentives won't help. Once I've lost interest, it's just more spam. But, I bet the answer is in your data - keep looking.

Thanks for sharing this wonderful article.

Hendrik Ruhe
profile image
I really like your comment but I also think it is quite too simple. Of course your example does apply for many players but still there is a high variety of different players which have their needs and wishes and they quit for different reasons.
In the end, the publisher/developer wants the player to pay money. So when he reaches a certain point, the commitment should be already so high that you willingly play in order to stay with the game even though it takes this long to progress.
There is absolutely a lot of space for improvement but I think it is not this easy.

toni xu
profile image
I like your comment: precise and key to the point. I'm the developer of vCruise. Wonder if you would like to try it out and share what you think about it. We keep close relationships with our players and make changes based on their feedback. Here is the link: https://apps.facebook.com/vcruise/?src=6&id=0

Brian Lockhart
profile image
Great article! What would also be interesting is overlaying / correlating the "days played" data against other relevant events on the calendar, in order to see whether or not player behavior changes in the face of other things that compete for their gaming time. i.e. what's the difference between a 60 day window that happens over a summer holiday period vs. a "back to school" cycle, or over Christmas holidays? You could also map out release dates for other games, major sporting events (Superbowl, World Series, World Cup, etc.) that might be distraction points for your users, and use those dates to plan out updates for your games in advance. i.e. proactively improve the user's experience right before a fresh temptation to leave arrives in their inbox.

Great stuff. Using data and predictive analytics is become more and more important as games move more and more towards never-ending service offerings.

Paul Laroquod
profile image
"We tried several hypotheses, and the simple ones didn't work out."

Would could the definition of 'didn't work out' be, I wonder, besides 'didn't give us the results we desired'? Statistics is a dark art — I have seen it practised in the very bowels of the largest banking institutions, and it was no different there. The definition of a statistical model 'working' is, in almost all unscientific contexts, 'making the data look better'.

Hendrik Ruhe
profile image
I just participated in the "Data Science Day" in Berlin. Great event together with guys from German social games developer "Wooga" and some other great companies like Innogames and BlueByte. There were some lectures and enormously interesting discussions about the collection of data in games. I can only advice others to check that out

Brian Lockhart
profile image
An excellent tool for playing with data is Tableau. Super slick visualization and "slice 'n dice" tool to let you go spelunking in your data and unearth insights you may not have considered before. Highly recommend checking it out, they have a free trial: http://www.tableausoftware.com/

Dmitry Nozhnin
profile image
Hi Brian,

We already use Tableau as a front-end for OLAP cubes and data marts. Indeed it's a very nice tool, however not particularly suited for data mining.

Jacob Alvarez
profile image
Have you ever considered just talking to the veteran gamers? Is it possible they might drop hints as to how long they will continue to play?

Michael Guy
profile image
it's a possibility, that not even veteran players know when they will stop playing, or pick the game up again. or their reasoning for doing so.

it might be as simple as an emotional realisation that stops them from playing, burnout with repetition or a non-challenging end-game, or they have to stop due to work or school, friends or family commitments taking precedence. it may be financial or personal, or social or just random.

it's interesting to see that they can measure the slump like this over a period of weeks, but it doesn't precisely go into redress measures, which i'd imagine would have to be quite awesome. my experience with Aion in the beta, turned me off the retail game, and i bought a LE copy, still unopened, i can't imagine what's changed from the sisyphean grind for no purpose, even 2 expansions, and F2P, later.

Aaron Casillas
profile image
Haven't played this mmo, but I can tell you every single mmo I have ever played EQ, WoW, COH, Star Trek was due to one reason why I left....an update that aggravated the amount of time that I had already invested or made it easy for other people to get what I worked hard for...

TC Weidner
profile image
there shouldnt be so much churn, it shows the flaw and failure of current MMOs IMHO. They design the games all wrong. The game should be designed end to front. Endgame is all important, leveling and all that other stuff is nonsense that needs to be tossed aside.

Create a dynamic world, add conflict, interest, and immersion and people will stay. I mean I find it amazing how poorly you have to design something in order for people to be SOO bored they are willing to abandon 100's if not 1000's of invested hours.

Danny Bernal
profile image
I do think it's simple, and I'll tell you what has done it for me. Put yourself in the players shoes. Suppose you have this fantastic world with lore and legend and magic. Exploration, discovery, overcoming seemingly epic (note: not necessarily difficult) challenges is the heart of what makes players excited about your game.

suddenly, the next patch changes the skill system to be more linear because some players found it too complex. or, XYZ leveling method has been nurfed and it will now take you 4x as long to catch up to your friends. perhaps you're forced into crafting because you need that next armor or the game is annoyingly difficult. everyone you know is now doing something different to get by and your game harmony has been disrupted.

Now that the game world has changed in some way, players are going to reevaluate how they play and if they want to keep playing. Also, if the flow of your game becomes "work" players lose interest.

...ok, my point is to pay attention to what players were doing before they quit. You'll know if your game harmony is fubar.

Jonathan Crow
profile image
Great articles! Both this one and the previous one. We've looked at churn prediction in the past and defining churn can often be problematic. Interestingly enough frequency of play comes up as the primary driver for churn systematically. And we had thought of trying to instead develop models for predicting change in frequency of play. Your article seems to confirm that we are on the right track. Thanks for sharing such insightful findings.

Dmitry Nozhnin
profile image
Ty Jonathan, and for sure defining churn right was the most tricky part of the research.

Matt Mihaly
profile image
Fantastic article, thanks. This is a new, and more sophisticated way of looking at churn. Much appreciated!

[User Banned]
profile image
This user violated Gamasutra’s Comment Guidelines and has been banned.

Preet Kukreti
profile image
I love how much effort is put into determining churn point to inform the right time to offer last minute incentives.

If instead this effort was focused on making the game more rewarding and enjoyable to play, then you would probably get a better net effect on player population than trying to facilitate relapses by standing at the exit gate and bribing players to stay with in-game resources; chances are they are simply sick of it and only a few will bite.

Jeremy Reaban
profile image
In this case, the entity collecting the data is only the publisher, not the developer of the game, so there really isn't much they can do except give offers.


none
 
Comment: