Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
October 31, 2014
arrowPress Releases
October 31, 2014
PR Newswire
View All
View All     Submit Event

If you enjoy reading this site, you might also want to check out these UBM Tech sites:

Why OUYA's Curation Metrics May Not Be Enough
by John Warren on 02/10/13 12:47:00 am   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.


I had the privilege of being part of an OUYA event in Austin this past Friday with OUYA founder Julie Uhrman. She graciously took the time to answer questions from Austin developers about everything from the reported yearly hardware updates to parental control guidelines. Most notably, the issue of storefront curation came up.

Her response was in line with this Michael McWhertor article on Polygon from Friday. I believe Julie Uhrman cares about curation, so let’s get that out of the way. My issue with their plan as a whole is the focus on what is referred to as “real metrics of engagement..” She posits that sales charts and download numbers don’t properly curate -- something on which she and I agree.

Her granular plan right now, as I understand it, is to take metrics like how often a game is played among its download base and whether a certain game is played first when the player turns on the OUYA and use them to create “engagement” charts that will serve to curate games on this incredibly open platform. Those metrics, to me, are a couple of versions of a curation story most players I know could tell.

Games are becoming so diverse -- often indescribable -- that the idea of taking quantitative measures of engagement seems like only part of an equation. Let’s lay out a very specific scenario that shows the importance of tweaking OUYA’s current curation strategy.

A few weeks ago, I bought all five episodes of Kentucky Route Zero at once for $25 instead of $7 for each individual episode. That’s not really an important detail aside from telling you what a good deal I think that is and you should jump on that train. I played the first episode in one afternoon. Without spoiling anything, I could easily play it again and make different choices to see how they play out, but I’m not going to. Do you know why? It’s not because I didn’t like the game. On the contrary, I loved it. I’m not going to replay it because the choices I made felt personal and going back to undo them seems like a disservice to how I imagine the remaining four episodes will play out for my personal protagonist.

That’s a special feeling and a special game. I imagine many who play it will feel as I do regarding staying true to choices.

I won’t speak for Cardboard Computer (developers of KRZ), but as the head of another independent studio making games that don’t necessarily fit the mold of “high engagement metrics” as Julie Uhrman describes them, the current plan scares me. If KRZ or a similar title comes to OUYA and the only curation on display is informed by the engagement metrics, lots and lots of people are going to miss out on something important.

That drives me nuts.

I know Uhrman says they’re kicking around additions to the current curation plan, which may be totally true and will fix everything and this will be a non-issue. My hunch, though, is that they’re attracted to Valve’s current model of not paying for any kind of curation whatsoever and letting the public decide the fate of what makes it, what doesn’t make it, what gets promoted, what gets buried, etc. I see the attractiveness there for a company trying to stay lean. I wish I had their restraint in terms of lean operation, but that’s another blog entry.

Might I suggest a quasi-free alternative? Another Austin developer and I were discussing things on Friday and he mentioned that having 3rd party critics (note: a critic and a reviewer are not necessarily the same) create lists either weekly or monthly and having OUYA promote them along with the engagement metric lists. An easy candidate would be Austin’s own Brandon Boyer, founder of the brilliant Venus Patrol. He already creates lists of awesome, often-easy-to-miss gems that even people like me (someone super plugged-in to industry goings on) see slip through the cracks. Recruit him (or people like him) and cross-promote their lists. Lots of very smart people are already curating online, so just make it an official OUYA curation squad and drive traffic to their causes.

That’s how they can stay lean and tweak the metrics system in one fell swoop. I just wish Steam Greenlight would add that functionality. When OUYA adds more curation than engagement metrics, they can consider me a total believer. Until then, I remain a healthy skeptic.

Related Jobs

Activision Publishing
Activision Publishing — Santa Monica, California, United States

Tools Programmer-Central Team
Vicarious Visions / Activision
Vicarious Visions / Activision — Albany, New York, United States

VFX Artist-Vicarious Visions
Magic Leap, Inc.
Magic Leap, Inc. — Wellington, New Zealand

Level Designer
Amazon — Seattle, Washington, United States

Sr. Software Development Engineer - Game Publishing


E Zachary Knight
profile image
Reading through the Polygon interview, I don't get the impression that engagement based metrics are going to be it. It looks like it is just another tool to measure a game's success. I doubt they would not have "top downloaded" and "top conversion" game lists. Adding a most played list etc would be a good edition as well. Depending on how each of those metrics are used to determine top overall games, it could be a great edition to the overall curation problem.

James Coote
profile image
At the moment, the store doesn't have "top downloaded" and "top selling" (or similar) lists, and all the signals we're getting out of OUYA suggest they won't be adding them. The polygon interview actually says "We don't believe that number of downloads or total dollars spent is a good indicator of a good game"

John Warren
profile image
Yes, I didn't mean to imply that they're stopping at engagement metrics for sure. The article as well as the actual presentation I attended made it clear they're open to new ideas. I guess what struck me is how non-committal they seemed to be (to my skeptical ears) to specific ideas about harder qualitative curation.

My hunch is that Julie Uhrman and the rest of the crew have done and will continue to do a good job listing to players and devs and coming up with solutions that benefit everyone.

James Coote
profile image
This has already been noted on the OUYA developer forums, and OUYA have said they are looking out for any suggestions on how to measure engagement aside from just the obvious number of times played and number of hours played

All the stuff like recommendations, reviews, user (or critic) created lists and the tools to help users actually find stuff in the store is all great, and I think it'll probably come. But not right away, because all that stuff takes time to code, whilst metrics are relatively quick and easy to implement and tweak

I think also it is worth saying that no matter how good a store is, developers can't rely on it for all their marketing. It can't be just a case of throwing it on the store and expecting to kick back and watch the downloads rack up. The store that benefit one gameplay style or game genre might not favour another, so game devs should assume it won't work for them and still pursue those other channels

John Warren
profile image
I definitely agree that devs shouldn't rely on the storefront for their marketing. Totally agree. I think from the platform developer's perspective, though, wouldn't you want to create a space where devs feel like if their work is qualitatively accepted and adored it would yield some tangible result?

Devs will (and should) explore other channels, but isn't it the platform holder's ultimate goal to get developers to jump head first and not have to worry about other channels? I'm not in hardware manufacturing or platform building, so I have no idea, I guess. Just a feeling I have about "the ideal" success of a platform.

Kujel Selsuru
profile image
I was very interested to see how they handle discovery and it looks like they're not trying the same crap as everyone else at least. This system also seems geared toward favoring gameplay heavy games over narritive heavy ones which I'm all for as I play games for the gameplay not the story.

Joseph Elliott
profile image
You may like games for one reason, and others may them quite another. Why not have options for both?

This system is interesting, but limiting, which seems to be the kind of thing they're trying to avoid.

Peter Odom
profile image
I agree with Joseph - it does seem a limited system. I enjoy many short games with novel concepts that have a heavy emphasis on art, writing, or presentation - and as John says, the engagement metrics under discussion could underweight my appreciation for that kind of experience.

But more to the point, I wonder if these engagement metrics (tracking playtime, length of time the game remains installed, and so on) would even do a good job of representing how the audience feels about games with more of a focus on "gameplay" (however we distinguish that from "story"). I can't count the number of times I've seen people in forums and discussion threads make comments like "I played Skyrim for 20/50/100/250 hours [or WoW for 6 years, or whatever] before deciding it was crap."

A game's ability to draw players in via the stickiness of its systems, in other words, doesn't always seem to correlate very reliably with those selfsame players' evaluation of how much they enjoyed the game. It makes me suspect that human curation really is incomparable with metrics-tracking, but that's a much larger subject.

Vin St John
profile image
How is Valve's storefront un-curated?
1) It requires approval merely to be included in it at all. (even in the case of Greenlight).
2) Games are featured in many curated positions, including the Store home page slideshow, Daily Deals, and "Specials."

The more automated lists that Steam produces also have a lot of variety:
1) Games your friends have spent the most time playing recently.
2) Games your friends have recently purchased.

The point being: In the end, EVERY digital storefront starts off simple and adds more ways of getting quality games discovered. OUYA's solution sounds like it improves on the standard 'charts' offered by Apple and Google Play, and will probably work well for most of the games out there. Even in the situation you described, a 1-hour game that is played to completion by most of its players will appear much higher on the list than a free-to-play game that most players drop in the first five minutes.

John Warren
profile image
The only way to get on Steam for new developers is through Greenlight. Greenlight is a Reddit-esque vote machine popularity contest.

There's nothing wrong with that for many players and that's cool with me. My issue is that Valve used to have a person (and it was apparently a single person) sorting through emails and unofficial applications and buzz looking for games that stood out and fit the Steam mold. When Greenlight was created (much cheaper than curation), they threw out that up-front curation entirely except for existing Steam developers and publishers. What they do with their storefront is a type of curation, yes, that's true. However, Greenlight feels like a barrier of entry for many new or young developers who see what makes it in (i.e. The War Z) and can't help but feel a little helpless about their chances of even making it on the storefront.

It's their platform and they're not hurting. I understand that. They've built an extremely successful business that people like me can only dream of at this point. Your assessment that OUYA improves on the "sales" model of curation is apt -- I do think it's better. What's OUYA about, though? Openness. Dialogue between players and developers. I think adding qualitative curation -- even free outsourced curation that takes place every day on the internet -- help add value to OUYA as a platform for developers and players alike.

Bob Johnson
profile image
I always have a difficult time with curation discussions.

Because I always get the impression some people think it is a magic pill. You just need good curation and all good games will find their way into the hands of the public and be successful and everybody will live happily ever after.

No matter the sheer numbers of games or how similar they are to each other.

Maybe I am just reading that into the opinions of some folks. But I always get that vibe.

John Warren
profile image
I definitely understand. I think with people like me it's a wishful thought, right? I do wish there was a magic curator and a magic list full of magical games and (selfishly) I'll find my way onto that list and prosper forever. That bias is there, for sure.

That's crazy, though. I know that. Just like solutions to any complicated problem there will always be gaps. Even great titles will fall through the cracks and land outside of Engagement Box and Curation Box and that sucks. I guess my hope is that OUYA (or any open platform) will at least build that second box.

Andy Lundell
profile image
No matter how they set up their store, you can construct a hypothetical scenario where it fails.

If that's where you stop your analysis you're going to conclude that all curation systems are equally unsuitable.

John Warren
profile image
Well yeah, I mean I could sit around and poke holes in every single thing they do, but I haven't done that. I think they have one hole and it's this hole. I think they have to use those engagement metrics like they've said and I also think they should set up curators to help push games that don't fit that mold. If your game doesn't fit into Engagement Box or Curation Box, then it stands to reason that it's not a great game.

That's not ironclad, but quantitative plus qualitative equals a pretty well-rounded view of what's worth playing, don't you think?

Jeremy Reaban
profile image
On the flip side, I see this as a good thing. I think there is a definite media biases towards artsy, hipster games that very few people actually play, but just say how wonderful and important it is. The game you mention sound exactly like one I want to avoid.

Joel Bitar
profile image
In what games-media do you find that kind of bias?
(Obviously ask for examples of publications aimed at gaming in general, not indie/dev specialized things)

John Warren
profile image
I think a fraction of the media is biased toward art games. I'd argue that the largest and most influential gaming sites on the planet (IGN, Kotaku, Destructoid, etc.) have an obligation to their advertisers to feature mainstream hits and most of what I see from those sites on a percentage of articles basis backs that up. Games that innovate but are hard to describe -- Journey comes to mind -- can sell very well. They generally need a qualitative curator to push that agenda, though. That's all I'm really suggesting. Let's get qualitative curators to catch great games like Journey that aren't already backed by a huge company like Sony, you know?

Steven Christian
profile image
Maybe allow users to recommend games, and then have a list such as "users that play the same games as you recommend these games:" or "based on this game you're playing right now, users have recommended:"
Not dissimilar to Pandora Radio where you pick some of your favourite artists/tracks and it recommends similar artists/tracks and adds them to your playlist, and you can either leave them or remove them.

Obviously it has to be better than the Steam Recommendation system which keeps recommending things that you've opted not to be recommended in the past.

Randy Angle
profile image
Discovery, curation and business model are all key factors in our new digital market place. Matching players to games they will enjoy is a difficult task to solve. I'm not saying I have a magic bullet, but I realized some very simple facts - which can be used to make discovery and curation easier:

1) As Steven C mentions above - analyze game playing behavior and recommend games based on similarities - most of the web and mobile analytics companies have plenty of data to support promoting your game to people they know will like your game... app stores can do the same

2) Recognize that the business model effects the curation method. In fact the business model should be highly dependent on the kind of game you make. Not every game should be F2P, choosing wisely is key to a games success - F2P games should be curated based on conversion, and engagement - Pay games should be curated based on player satisfaction ratings - Episodic or Subscription games should be curated based on ratings and repeat business (number of episodes purchased or months on subscription) - there are kinds of games that would do best with ad-based revenue, curating them should probably be based on audience size.

You only have to look at the Amazon model, verses YouTube model, verses F2P App Store model to see that people consume content in a variety of ways. Even just comparing music on iTunes to Spotify shows that the same content can be consumed in multiple, profitable, ways.

The eternal arguments of indie games as art and retail games vs F2P games shows that there is no one size fits all solution to making, discovering or enjoying a game. So your curation system has to account for the business model when it ranks competing apps.

The app stores need to have video trailers and promotion graphics (icons and screenshots with ad copy) - we've seen from the recent DICE talk by Jesse Schell that try-before-you-buy is only hurting sales so skip trials altogether... unless you find a kind of game that breaks that rule too. This is an iterative process.

Mark McGee
profile image
This article reminded me of this older article: