Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
October 25, 2014
arrowPress Releases
October 25, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
An Algorithm for Discovering "Hidden Gems"
by Lars Doucet on 04/17/14 01:51:00 pm   Expert Blogs   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

Cross-posted on FortressOfDoors.com

So you've probably seen that Ars Technica article that has magically reverse-engineered every steam game's  gross (sales+free-key-redemption) numbers.

Everyone's talking about this graph:
 

(They have since updated that with a logarithmic y-axis, see the Ars article for the new one).


The obvious first statement is "WOWS! Video games are a hit-driven business!!!" For what it's worth, our game Defender's Quest is probably located somewhere between 800-1200 on that graph, and we're doing just fine.

But what about the game at 2600? And what's going to happen when Steam totally opens the floodgates?

Every developer I know has a list of obscure indie games they know are great, but just can't get the exposure they deserve. Jay Barson talks about this phenomenon in his recent article The Best-Kept Secrets. (Incidentally, if you like CRPG's and haven't bought Frayed Knights yet, do yourself a favor and grab it)

Here's Jay (emphases mine):

A friend told me about an experiment in inequality and the “cultural” marketplace. Test audiences were exposed to new music, and invited to download and keep some songs. The songs had indicators of how much they’d already been downloaded. Some of the songs had an artificially inflated number. What they discovered is that the artificially inflated number helped the popularity of some songs within the controlled audience, but could not others.

In other words – some songs had hit potential and others did not. If a song had both hit potential and an apparent wave of popularity behind it, it could take off. A song that sucked rarely did well. Hit potential could not guarantee success, but it’s lack could guarantee the lack of success – or at least a large success. In other words, the distorted perception of popularity had some effect, but was not the sole contributor to the success or failure of the song.

Right now stores like Steam and the App store are dominated by "top selling/grossing/downloaded" charts, which all suffer from the same problem -- the charts are self-reinforcing and prone to freezing, thus encouraging shady behavior. You can mitigate this effect in various ways (such as sampling over a shorter time, as Steam seems to do), but the problem remains -- the games on top stay on top.

My twitter friends are constantly trying to personally give visibility for such "hidden gems" and I think that's admirable, and we should keep doing stuff like that. At the same time, I think we can supplement those efforts with some mathematical tools.

Here's the idea -- if the above experiment's results are applicable to video games, there might just be a way to build a "hidden gem" detector, given access to the right data. What we're looking for is some quantifiable way to find a game that people really seem to like, but that hasn't yet had a chance to be popular. The kind of game my twitter friends are talking about when they say, "RT! Man (Game X) is wonderful, such a shame no-one has heard of it."

 

 


Building a "Hidden Gem" detector

What we're looking for is games with high engagement stats but low overall popularity. The data Ars Technica has been working with is truly awesome, but for these purposes it's a bit thin -- all we've got is gross units owned by players (lumping together actual sales with steam key redemptions), and some play time stats which have some known reliability issues.

But Steam has access to better data than that. Steam knows exactly how many copies of each game have been sold as well as the number of concurrent players, all down to the hour, region, and platform. It also has overall playtime stats per player, though that might be less reliable. There's also conversion rates for games that have demos, achievement data, and more. And of course, there's also the recently-added user reviews.

In short, Steam itself has plenty of data, both private and public, to make a pretty good guess at overall player engagement for a particular title. For "traditional" games, a higher-than-average playtime is a good heuristic for engagement, and for short-form games like Dear Esther, some metric for "completed the experience," such as a specially marked achievement, could work. Or we could just follow Kongregate's example and simply go by user rating, as I suggested in a previous article. (I personally prefer something closer to the Tomatometer over anything like MetaCritic scores).

The key is you have to be careful not only with what you measure, but how you measure it. Give these two articles a quick read before proceeding. It'll just take a minute:

How not to sort by average rating
reddit's new comment sorting system 

Back already? Great.

As those articles detailed, the ranking system should measure something other than raw popularity. Otherwise, the first item to get close to the top gets seen more, and thus collects more (sales/upvotes/downloads) and the cycle continues. The ranking system should also automatically take into account uncertainty based on sample size.

If you're ranking based on an engagement stat like play-time, or by user rating, then getting to the top does not guarantee that a game will stay there. In fact, the influx of new eyeballs will more than likely kick the game down a few notches as it gets exposed to a wider audience. This way, only games with high engagement --regardless of popularity-- will be able to stay at the top of the charts for a long time, which is how it should be. The chart is self-correcting rather than self-reinforcing.


Quick example: imagine a single-player RPG with median playtime of 5-10 hours -- well above the average for Steam games, but with less than 2,000 sales. This should be easy to detect. Or one of those crazy avant-garde "experience" games, the kind you only play once but it's amazing. Say that game has 2 hours median playtime, but 90% of it's 3,000 or so players have gone all the way through. This should stand out, too.

We don't have to get super-detail-oriented with this. In a world where 37% of player's Steam games haven't even been loaded a single time, it should be easy to make some simple broad-strokes guesses and pick out games that make you say: "wowzers! A small niche of folks really love this thing."

Let's make a "hidden gems" list for those.


Limitations

The "hidden gem" detector isn't a silver bullet. It doesn't make the game of platform power go away, nor does it solve the structural, sociocultural and/or economic issues that favor the big boys. It also does nothing for games that aren't already on Steam (though Valve has given every indication they're going to let just about everybody in eventually). Here's a few other issues.

Beloved by Niche, Hated by Mainstream
If the "Hidden Gems" list is featured right on the steam front-page, then a lot of cool, weird stuff that's beloved by small fanbases will shoot to the top of the list. Immediately afterwards, there's a fair chance the mass market will cast its withering gaze upon it like the Eye of Sauron and mercilessly smack it down faster than you can say "Walking Simulator." So, perhaps hidden gem lists should be segregated into their natural niches, using steam tags, or an amazon/netflix-style recommendation engine?

On the other hand, you never know what seemingly crazy game idea will actually catch on if it's just given a chance with the mass market. This is exactly the kind of tool that could bring us the next Goat Simulator.

Different Strokes for Different Folks
As I mentioned above, you can't use a one-size-fits-all engagement heuristic to catch all the gems. "Did you finish it?" is much more relevant to judging overall engagement for Dear Esther or The Stanley Parable than it is for World of Warcraft or Dwarf Fortress. Since the whole point of this is to identify interesting things that would normally be passed over, special care will have to be taken with the various heuristics used. And what about things that are harder to measure like Proteus? I'm confident there are ways to detect people's interest in these things, but we'll have to be smart about it.

We Won't Find Everything
This method can't possibly find all the gems, nor should we expect it to. It's just another tool, and that's important to remember. It won't fix everything, but it could help make things a lot "less worse."


Gaming the Charts
Anytime you have a system where charts drive sales, it encourages people to game the charts. If it measures downloads, you drop your price to zero, cross-promote, and use your own money to inflate your stats. So I could easily imagine similar scams if we're measuring user playtime -- with proverbial goldfarmers idling steam games to inflate their client's chart positions. Safeguards could be put in for this -- it should be pretty easy to look at other metrics to detect if someone is "idlefarming" for chart position. Of course, cheating of any form is always an arms race, with the only final solution being to make it not worth the cheater's time and money to cheat in the first place. Which brings me to my next point.

Moving beyond Charts

By itself, I don't think another chart on the steam front page is going to make a huge difference -- being hand-picked for a feature still makes a world of difference and most Steam developers put their marketing efforts into securing that rather than trying to micro-manage the charts.

Valve has said their vision for Steam's future is that anyone can make a steam front page and curate it themselves. The algorithm I've described above could be used not just by automated charts, but by actual human curators. You wouldn't have to expose any sensitive data (like sales or playtime), just give curators the ability to look for these kinds of patterns with advanced search tools.

Right now, if I want to promote an obscure indie game, it has to already be on my radar. How cool would it be if I could just whip out my Hidden Gem Detector and go treasure hunting instead?


Related Jobs

Digital Extremes
Digital Extremes — London, Ontario, Canada
[10.25.14]

Sound Designer
Disruptor Beam, Inc.
Disruptor Beam, Inc. — Framingham, Massachusetts, United States
[10.25.14]

Lead 3D Artist
Red 5 Studios
Red 5 Studios — Orange County, California, United States
[10.24.14]

Graphics Programmer
Red 5 Studios
Red 5 Studios — Orange County, California, United States
[10.24.14]

Gameplay Programmer






Comments


Ian Richard
profile image
I would love to have a hidden gem detector. Though I'd be happy to just get a more personalized recommendation. I'm a very niche player and the usual "Here are the highest rated games on metacritic" doesn't even come close to satisfying me.

I love your idea of actually taking into account a small percentage of players that play longer or higher completion percentages. I assume that combining those with Steam's new tag system could create a pretty decent effect.

Shannon Rowe
profile image
A lot of "hidden gem discovery" comes down to subjective personal preference. Perhaps one solution then might be to build a database of community personality traits combined with games rating. So for example I might disclose to the database my enjoyment of adventures, interest in noir mystery themes, and then list the Tex Murphy games as recommendations. If enough similarly minded people ranked the same, those games would bubble up the list, and then the next player to plug in those parameters might find their hidden gem.

Ross Kravitz
profile image
This problem is actually well studied in academia and industry, and it's called collaborative filtering. However, the prime benefit of this method is that it doesn't require users to manually submit preferences, which is both arduous and not necessarily effective, because it may be hard to quantify what exactly you like in a game.

In collaborative filtering, all users have to do is submit rankings of games (music, movies, products, etc). The core algorithm then goes and finds players who have ranked games in a similar way, and finds games that those similar players have ranked highly. This algorithm, I believe, was pioneered by Amazon in an industrial setting, and it powers much of their item recs. Of course, the algorithms that a company like Amazon uses in practice are much more complex, and likely incorporate several other data sources and algorithms, in much the same way that Pagerank is just one piece of the puzzle for Google search.

Bart Stewart
profile image
Richard Bartle suggested a similar system a couple of years ago: http://www.youhaventlived.com/qblog/2012/QBlog080212B.html .

A system like this should work well for a focused product set like games.

Daneel Filimonov
profile image
Here's a possibility; You can discover "gem-ness" of a game by the following (for example):
- How many people have bought/played the game
- How the popularity of the game is compared to its tags/genre
- How "rare" (unique or common?) the tags/genre are/is
- When the game was released in comparison to others of its tags/genre and/or popularity
- How much advertisement (flash sales, summer sales, etc.) the game has received in comparison to other games of the same tags/genre (or just of games in general)
The list can go one, but that's quite a starting point. Anything to find something unusual could be of benefit.

Kevin Fishburne
profile image
Maybe a user rating system could be implemented based upon human emotion, using something like this as the basis for its UI:

http://fullfeminine.com/wp-content/uploads/2013/06/plutchiks-whee
l-of-emotions.jpg

Just like all films and novels aren't intended simply to amuse (Pan's Labyrinth, Schindler's List, Nineteen Eighty-Four, etc.), neither should games. It's more about the intensity of the emotional experience than anything else, so why not rate them that way. The only complexity to this system I can think of is users consciously having to make a distinction between feeling anger due to the game sucking or due to the emotional weight of its narrative or difficulty.

I do still think quality ratings such as graphics, sound effects and music, as well as play-style categories like "strategy," "twitch", "puzzle", etc. should be implemented and available to users as filtering tools, however.

Kyle Redd
profile image
Another possible method of detection could be a "passion index." These would be games that have a high level of community content per owner - things like screenshots, workshop contributions, even discussion threads or reviews. This would help highlight games that work to be truly unique or memorable in some way (even divisive), giving them a leg up over those that emphasize a feature checklist.


Edit: Ninja'd by Kevin. He has roughly the same idea.

Kevin Fishburne
profile image
Your idea's a bit different, but a damn good one. Could be called "buzz" as well, as in something everyone's talking/posting about. It's one thing to just play a game, another to actually talk about it on the net.

My idea was more or less to show the overall and specific levels of emotional connectivity a game has with players. A game that has little emotional impact on any level is probably not worth playing at all, while games with strong impacts (on any level) will be enjoyed by somebody. Which "somebody" that is depends on what kinds of emotions you like to feel when playing a game. One filled with sadness, anger and happiness is probably narrative driven like a Final Fantasy. One filled with anger and fear is probably more like Doom. Some people want to play a game to relax, others to challenge their logical problem solving skills, some to enjoy a good story with all its ups and downs, and other just want to feel the adrenaline and kill or be killed. The "emotion wheel" would not just describe how a game makes people feel, but how intensely/successfully.

egos doud
profile image
I think this kind of concept can influence the next generation of marketplace.

A "like" option ingame for steam user : it's maybe the easy way for support developer

R. Hunter Gough
profile image
I built one of these. Only a dozen or so people used it regularly, so when the server died I didn't bother to resurrect it. But maybe it was just ahead of its time. :)

http://studiohunty.com/reccr/

Felipe Budinich
profile image
Care to share source on github? I think it's a very interesting idea worth resurrecting.

R. Hunter Gough
profile image
perhaps! I keep also halfway thinking I should resurrect it myself. :)

Kujel s
profile image
I just want to say kudos to you Lars for trying to find and coming up with some solutions for helping games find gamers. As you say this isn't some kind of silver bullet but this idea is one of the best suggestions I've seen for dealing with the discovery issue.

James Yee
profile image
I'm still hoping for "Video game DJ" playlists. Sure you can kinda do it now and do it on your own sites, but recommended lists right there on the front page from people you "subscribe" to or what not would be great. With the caveat that the lists have to be made of games the DJ has actually played.

Then I can follow say TotalBiscuit or Angry Joe on Steam directly and have their curated lists to reference right during that shopping itch moment. :)

Daniel Gutierrez
profile image
Pretty much every recommendation system moves from trying to generalize their population's interests to individual's preferences. Most recommendation systems follow a pretty simple idea (Pandora, Netflix, Amazon, etc) that looks like this:

Person A rated movie X 5/5.
Person B rated movie X 5/5 also.
Person A rated movie Y 1/5.
I guess person B will not like movie Y.

By personalizing like this you completely nullify many of the issues you described (rigging the system, quick spikes up and down, biasing new comers or old comers). And the reality is these are incredibly easy to make (this was literally a 1-week homework assignment in college).

From what I can tell on Steam there's only one reason this isn't used... and it's that global ranking (absolute dl's and similar) heavily favor Valve games. Not trying to tinfoil hat here... but I greatly agree with your desire to get a hidden gem finder, but I can guarantee you it's not a matter of technical implementation they haven't bothered.

I'm hoping that ars does a data dump of everything they collected, then it'll be pretty easy to make this type of recommendation system even w/ the limited data that they have.

Dane MacMahon
profile image
If you fall into a specific group like I do (old school PC RPG fans) then it's rather easy to curate by yourself. Any title I might want to play is mentioned and heavily discussed on the sites I most visit (RPG Watch, RPG Codex, Games Banshee, GOG). Getting mentioned on those enthusiast sites is the best way to get your game out there and noticed.

I would imagine it's the same for adventure, shooter and other genres as well.

Phil Maxey
profile image
To me the future whether it be Steam or the App store is tailor made curated playlists. There are many people who actually want to recommend games, they don't mind the job of giving their opinion on their kind of games. So if these people were more visible you would then just need to follow them or incorporate their "recommended" list into your own list to find out about the kinds of games you like as well.

This approach solves a lot of issues, it means that platforms themselves don't need to get involved with review or featuring any content as the community will do it for them. It doesn't require any algorithms (apart from very basic ones) and It helps build a community around certain kinds of games.

All this is possible right now, it just requires the platform holders to create a much more sophisticated system by which people can be involved with their platforms, and there's the rub. Because even though it undoubtedly is the way forward will the platforms spend the time and money making it happen? Valve keeps dropping hints that they are maybe thinking of something like this, but it's yet to appear.

andreas grontved
profile image
You are so right about this. And it applies to every store out there.
Why is it so hard for these stores to understand that there are alot of games that I have absolutely no interest in but keeps shoving them in my face --- while completely ignoring the existence of others?

E Zachary Knight
profile image
This probably isn't what you are looking for, but it is pretty neat. It uses your current Steam library, asks you to "throw out" one of two games in succession and uses that data to match you up with games you should probably be playing.

http://www.crunchmagic.com/

Waylon Flinn
profile image
The process we use is actually pretty close to one of the methods described. We use fancy collaborative filtering algorithms that are modified versions of the ones from the Netflix Prize. We combine data on whether someone plays a game more than average and data from our own custom game. The game generates stuff that's like a really high quality rating (rating quality is a big problem in the method Netflix uses).
We'll also be launching a new version of the site with a lot more features in the next week or two. It'll give you the ability to search any game in our library and try to predict whether you'll play it more than average and whether or not you'll like it. Check back in a week or two. We'll also announce it on our Facebook page.

Justin Kovac
profile image
I like stores like Amazon which do:
Give recommendations based on what I have been browsing, On what I bought
Mentions what other people bought who bought the game I purchased

I have found some games (rarely, less niche titles) and a lot products I did not know exist via this method.

"Bought Avadon and played 5 hours of it? Here is some other games you may like."

Michael Parker
profile image
"To me the future whether it be Steam or the App store is tailor made curated playlists. There are many people who actually want to recommend games, they don't mind the job of giving their opinion on their kind of games."

Gabe newell said exactly this at the DICE keynote back in feb 2013, so don't worry, it's coming :)

http://www.youtube.com/watch?v=PeYxKIDGh8I&feature=share&t=29m37s


none
 
Comment: