Gamasutra: The Art & Business of Making Gamesspacer
arrowPress Releases
July 22, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Lacrosse Fans Manipulating the XBLIG Rating System
by Robert Boyd on 03/23/11 11:26:00 am   Expert Blogs   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 
About 5 days ago, I noticed something odd. After hanging out around the #6 top rated slot on the Xbox Live Indie Games US dashboard for a month or two, our game's rating started to drop very quickly. In less than a week, our game went from being #6 top rated to our current spot of #11.

 


I did a little investigating and it turns out that 5 days ago, the "College Lacrosse | The Video Game" Facebook page (which has about 175,000 fans) requested help from its fans to raise the ratings of their games. Now, there's nothing wrong with asking your fans to rate your game positively, but it looks like some of their fans were overzealous and started 1-starring other high rated games in attempt to make it easier for the Lacrosse games to rise through the ranks.

 

Now for something like Z0MB1Es (the #1 rated game) which has over 50,000 ratings, a sudden influx of 1-star ratings isn't going to do much, but since our game has been out for less than 3 months and has far fewer ratings (about 1500) than the other top rated games , it affected us big time and we dropped 5 ranks. In contrast, the Lacrosse games which were in the middle of top 20 best rated (I don't remember the exact positions) are now ranked #4 & #5.

 

Note, that ranking isn't just a matter of prestige, it's also a matter of money. The higher rated your game is, the more likely it is that you'll continue to get long-term sales. Our first game, Breath of Death VII, has sold nearly 50,000 copies in the year or so it's been out and the vast majority of those sales (over 75%) occurred after the first 30 days. Before the ratings attack, we had one game that was easily visible in the first screen of top rated #4) and one that was partially visible (#6). Now, we have one that is partially visible and another that you have to scroll over a screen or two to see.

 

If the Lacrosse developers had explicitly told their fans to attack other games, that'd be one thing, but as far as I can tell, they didn't. They just told their fans to rate their game highly and gave instructions on how non-XBox 360 owners could create a free account and use it to rate the game. Of course that raises another question - should people who don't even own the system be able to rate games on it? With the current system, anyone can rate anything - even if they have never even downloaded the trial. Should this be changed so that only people who have played the game or tried the demo can rate the game?

 

I have no idea how this sort of thing could be prevented, but I'm really upset. We were really hoping that the upcoming enhanced version patch for Cthulhu Saves the World would give our game the little push it needed to finally stabilize in the top 5 rated. Now, it looks like we'll be lucky just to get our old rank back.
 
 
What is to be done about such situations? Starting a rating war with fans of other games 1-starring the Lacrosse games is no good - that'll just make the mess even bigger. What is to be done?

Related Jobs

CCP
CCP — Newcastle, England, United Kingdom
[07.22.14]

Senior 3D Environment/Generalist Artist
CCP
CCP — Newcastle, England, United Kingdom
[07.22.14]

Development Director
FitGoFun
FitGoFun — Mountain View, California, United States
[07.22.14]

Unity 3D Programmer
Vicarious Visions / Activision
Vicarious Visions / Activision — Albany, New York, United States
[07.22.14]

Concept Artist (Temporary)-Vicarious Visions






Comments


Jim Perry
profile image
"... should people who don't even own the system be able to rate games on it?"



Of course not! It's an idiotic design. Several suggestions have been made to change this and the one that makes the most sense to me is to have people that have bought the game (actually bought it, not used a code) weighted normally and people that have downloaded the trial weighted slightly less so. People that have not at least downloaded and played the trial version should not be able to rate at all, period! :

Reynaldo Lor
profile image
Jim,

Your stance is quite understandable; if you bought the game, your opinion should matter more. I agree with you there. I'm not sure how you would weigh people differently, perhaps you could assign different colored stars for people to rate with. This could be done when you sign in, the system would know if you own the full game or demo and assign the correct colored star. Say gold for purchased based reviews, silver for demo owner reviews, and a different color for reviews from non-owners.



Why non-owners? Because this is America damn it and anyone should be able to voice their opinion. Suppose I went to my buddy's house and played on his system. I really like the game, but I won't buy it for myself. Suppose also that I want to share my opinion and tell everyone on XBLA that the game was good, but I don't own it. Why shouldn't I be allowed to rate it? Maybe the color of my non-owner rating star is red or pink, maybe I don't like red or pink and it shames me into downloading the demo in order to rate with a silver star.



I'm being silly of course, but the point is that we can always find solutions to our problems. We just have to work together and collaborate, excluding people may not be ideal.



rey ;)

E Zachary Knight
profile image
I still don't know if that is a valid justification. That still leaves the system open to abuse. All you would have to do is lie and say you played the game somewhere at some time. Anyone can do that. I think with this system, you really should have to provide some evidence of actually playing the game through the records MS keeps on which accounts download demos and which buy games.



I guess if they did implement some kind of separated review system, non owner reviews should have zero bearing on top rated lists. At least that is what I think.

Ian Fisch
profile image
@Reynaldo



So in your opinion is the Nielson TV ratings system unAmerican because every TV exec can't just buy up a bunch of neilson boxes and plug them into their own TV?



Also, how are you "not sure how you would weigh people differently"? It's called a weighted average.



My god your comment was idiotic.

Reynaldo Lor
profile image
Just stumbled on this thread again, Ian you are one angry dude. I read through some of your posts and comments on this site and noticed that you tend to write on the negative side of viewpoints. I'm just wondering if game development has jaded your ability to enjoy video games? If so, cheer up; you're making games remember.

Kobun Servbot
profile image
There's an easy fix for this: only have the rating panel appear within a player's game library. This would both correct the issue of rating games being more of a pain now as well as forcing people to at the very least download a game to be able to submit a rating on it. It would be a simple solution I'd imagine Microsoft would have no issues with implementing while addressing the problem.

E Zachary Knight
profile image
I agree that the best solution is to implement a change that would only allow a user who has downloaded an played the demo and/or purchased the game be allowed to rate games. That seems to be the only real solution as there is no other way to get these games.



Is there no fraud reporting option with XBLIG? If there isn't there should be. I think any developer who has lost positions in the top games list due to 1 star reviews since the Lacrosse promotion should be reporting it to Microsoft.

Evan Combs
profile image
First off no one who never played the game should have ever been able to vote anyways. Secondly one possible solution is if there is a huge influx of similar ratings just temporary freeze the ratings.

james sadler
profile image
It should be pretty easy for the Xbox managers to go in and look at who has given these 1 star ratings, and find the pattern that they in turn gave everything in the top X 1 star, and yet gave the Lacrosse game a 5 star, and just null their rankings. This is obviously abuse of a system, which does have monetary effects. One can not put a real price tag on this, but just knocking someone out of their spot takes away their potential to earn more income. If this is done properly, through actual valid reviews, then its an "oh well" situation, but to do it like this is just wrong.

Matthew Doucette
profile image
This problem is solved with more traffice to XBLIG, and blocking off non-Xbox users. The right to vote on games you have not downloaded and played should stick I think. And Microsoft should weed out votes on one game with 5 stars and the rest with 1 stars. hotornot.com did it. MS can do it too. This is not a hard problem to solve.

Ian Stocker
profile image
I just wanted to chime in and corroborate Robert's experience. Soulcaster II, with only 350 ratings, dropped about 20 places in the last couple weeks (unheard of for any game). Soulcaster, with 1300 ratings, only lost half a star due to this assault and is still holding strong.



This episode should prove how the system can be gamed, so I hope Microsoft takes some steps to safeguard against this sort of thing. I would go so far as to say that you need to have downloaded the demo in order to rate a game.



A Top Grossing list would be a welcome addition as well, but that's another topic...

Ian Fisch
profile image
Every XBLIG developer I know games the system hardcore. Creating multiple accounts and downrating competitors' games is very common. It's a really retarded system.

George Clingerman
profile image
They must not share that with me because I've personally never had ANYONE tell me they did that. You saying it here is the first time I've ever heard of it happening. I'm fairly involved with the community so I'd be kind of surprised if it's as rampant as you're suggesting...

Delorian Tokes
profile image
Hrrmmm. Not cool. Not uncommon but surely, not a great thing. Maybe a good way to remedy and keep things fair is to only allow ratings from a logged-in user, who's account has actually purchased the game. XBL shouldnt let me rate something i havent even played.

Ryan Moore
profile image
175k fans for a lacrosse game? I didn't know that many people even knew what lacrosse was. After clicking on the link to their facebook page, I couldn't find the post that requested ratings help. Strange.

Regardless...



Why not fight fire with fire? Use this opportunity to get a gamasutra army together and up your game's rating? And maybe a few of us might even give that lacrosse game a 1star rating. If microsoft isn't going to fix the system, then you have the right to work the same angle this lacrosse game is. For that matter, if there are any games in the top ratings that you don't believe should belong there, go rate it low. But I suppose I'm a little less shameful than Robert. He seems like a pretty humble and reasonable guy.

Ian Stocker
profile image
Kobun hit the nail on the head--rate games from that game's panel once it's downloaded.



I'd like to follow up on my comment to say I don't think the Lacrosse guys necessarily had any hand in the downvote bomb. They probably just have some dedicated fans who wanted to see them rise in the list.



If a bear breaks into your tent to steal food, you can't blame the bear, you should have put the food in the storage box. Humans are humans, it's the system that needs changing.

Jamie Mann
profile image
Personally, I think XBLIG's game rating/information is currently broken, both on the website and on the console. For instance, the game page on website has a single "media" pane, in which you can see either a single screenshot or the game's information/description. And by default, it shows a screenshot; the user has to click on the media-pane UI to find out more. Not the most inviting way to greet newcomers...



Similar applies to the console dashboard, as it's modelled on the WM7 UI; as a result, information is broken into multiple pages (as with the old dashboard, but at least that offered a "stacked tiles" view which gave you a bit more visibility), each with minimal SDTV-friendly information on it. As such, casual browsing is a slow and frustrating process.



(and from a personal viewpoint, it's also annoying that the marketplace menu defaults to "buy", not "download demo" - and it also doesn't detect if you've already downloaded the demo or not. Which is a step back from the old dashboard...)



However, the icing on the cake is the fact that while full information on a given game is available from the marketplace view, most of this information is unavailable from the "downloaded games" view. Most frustratingly, you can't rate games from the "downloaded" view; instead, you have to navigate into the depths of the marketplace system.



All told, the current WM7 UI generally had a negative impact on the marketplace, both for XBLIG and XBLA games. Roll on a refresh!



Back to the main discussion: fixing vote manipulation. Restricting voting to people who've downloaded the game is an immediate quick-win; it'd stop the problem Ian Fisch mentions, of people creating multiple accounts on the xbox.com website. However, it might not have that much impact on the "downvote bomb" issue, as they're targetting specific games; it's not that hard to download a dozen games, rate them and then delete them.



One solution to this is to restrict voting to people who've bought the game (as opposed to just downloading the demo). However, as mentioned above, this tends to lead to only positive votes being given; few people buy a game because they dislike it! Implementing this approach would therefore be counter-productive, as it'd make the ratings useless.



Instead, what I'd suggest would be to go for a hybrid system:

1) Restrict voting to people who have downloaded the game (demo or full version)

2) Allow unrestricted voting in the first X days after release (e.g. 14 days)

3) Limit the number of votes which can be cast per day after that - either with a fixed level (e.g. 10 votes per day), a sliding scale (50 votes in the first month, 25 votes in the next month, 10 votes after that) or a percentage (e.g. 10% of the number of existing votes)



This would allow the game to get a (hopefully) fair assessment during it's launch period and would then minimise the impact of vote bombing; it's one thing to click a couple of icons on a website, it's another thing altogether to have to download the game and then wait for several days/weeks until you can register a negative vote.



It's not a perfect solution, and some number-crunching would be needed to decide what level the vote-limiter should be set. But I think it'd offer a reasonable balance between being functional and protected...

Matthew Doucette
profile image
What makes this even worse is that XBLIG games (and XBLA games too) get less ratings due to the Nov 1st, 2010 dashboard update that doesn't allowe anyone to rate games after they are finished playing them, unless they "surf" back to the Games Marketplace and search for the title, and cannot rate games from their game library.



Vote on this issue here:



https://connect.microsoft.com/site226/feedback/details/619523/game-library-lacks-rating-s ystem-difficult-to-rate-games-after-youve-played-them



I woud also suggest commenting on this issue with this rating abuse story.



It's unreal Microsoft created this problem and has not fixed it. When you reduce the number of proper ratings, you increase the effectiveness of bad ones.

Chris Johnson
profile image
I was saddened to hear this! Zeboyd, MagicalTimeBean, and Xona all deserve far better than this from Microsoft. Unfortunately, my 5 star votes for you all only counts once. As far as I can tell, this is just one more thing that MS is screwing up with the XBIG service.

Chris Johnson
profile image
As an aside, I've already complained to Microsoft via the Feedback option on Xbox.com. I suggest others do so as well.

Henry Shilling
profile image
It seems that the ratings systems on everything are broken, I can go to the AppStore and write a review and rate a game I do not own. I dont have a PS3 but people are saying it's pretty borked as well. It seems these providers just add the user rating/review as a checkbox item that people want, not paying attention to if they work or not.

Matt Rix
profile image
Um, the App Store is one of the few places that gets ratings and reviews right. You have to own a game to rate or review it. The ratings on the App Store are usually quite accurate to how good a game actually is.

Sven Bergstrom
profile image
Agreed!

Jim Perry
profile image
So how many crappy games have ratings? This was discussed as a fix, but you'd probably end up with only decent games having ratings.


none
 
Comment: