Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
September 17, 2014
arrowPress Releases
September 17, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


Opinion: Would you pay to have your app reviewed?
Opinion: Would you pay to have your app reviewed?
October 2, 2012 | By Kyle Kulyk

October 2, 2012 | By Kyle Kulyk
Comments
    21 comments
More: Smartphone/Tablet, Business/Marketing



In this reprinted #altdevblogaday opinion piece, Itzy Interactive's Kyle Kulyk looks into the seedy practice of paying sites for mobile game reviews, and shares his results from trying it out.

I've been gaming about 30 years now, ever since that Commodore Vic-20 made its appearance under our Christmas tree back in 1981-82. Video games have been a passion of mine since well before I decided to try my hand at creating them as a profession, and during these past 30 years, video game reviews have evolved just as the games they reviewed.

Back then, if a game shipped broken, it shipped broken. There was no patching, no do-overs, so I relied on print reviews to help me save my money and time from busted games. Not to say I still wasn't duped into buying a broken game from time to time (I'm looking at you, Atari E.T. How many hours of my childhood were spent in those stupid pits?), but generally I trusted game reviews to help set me on the right path when it came to spending my gaming dollars. Today the internet is awash in review sites.

Now as an innocent kid sneaking a peek at the latest review scores from a magazine at a rural pharmacy, it never occurred to me that companies could be paid for these review scores. Certainly over the years and with the rise of the internet, there's been much more talk of review sites and publications taking money in exchange for their reviews.

It's hard not to look at a review score on a site that's pretty much one giant advertisement for this game or another and wonder how much money has exchanged hands prior to the review being written, and how could that not on some level influence the review in question.

The internet seems to be in general agreement. Accepting money from game makers for reviews could potentially compromise that review and is, to put it bluntly, a bad thing. Then there are app review sites.

After launching our first mobile game, Itzy3d, on Android and iPhone in January this year, I next set about the daunting task of trying to get our little game noticed. I wrote up a press release, sent this to various PR sites, and then set about contacting as many review sites as I could find in the hopes that someone would have a look at our game and, for better or worse, inform their readers of their opinion of the title while the rest of our team started work on our next title.

Over 200 emails went out with a short blurb about who we were, what our game was about, a few screenshots, and our desire to have our game reviewed.

Almost immediately I started receiving replies from sites, and the average reply would go something like this. "We have a lot of submissions and may not get a chance to review your game. Send us money, and we'll be able to do something for you."

Now at first I was dead set against this. Paying for reviews? The entire notion just struck me as wrong. I thought reviewers had a lot of testicular fortitude to even suggest I spend money on their opinions.

But then more emails came, with more offers for "expedited review services" for a fee. Some offered up advertising as well as a review to add value, but when questioned about click rates, monthly visits, and the like, the information I would require in order to make an informed decision on where to spend my advertising dollars was almost never forthcoming. Just send us money.

At some point I realized that this wasn't just a few sites looking to cash in on desperate app developers; asking for money in exchange for app reviews seemed to be the norm. After waiting for weeks for someone, anyone to review our title, reluctantly I opened my wallet and paid for a few reviews.

Obviously, I had to pay if I wanted to play their game. The reviews from users on the Apple and Android app stores had been very positive, but what we were hoping for was a detailed, well thought out review that could give us some feedback and maybe boost our visibility.

I paid for four reviews. Two were helpful, one was ok, and one just took our blurb from the press release, put it up on their site, and slapped three stars on it. Of the $100 out of my own pocket spent for reviews, the 200+ initial emails sent out, the follow-up emails, and the odd correspondence between myself and review sites, we ended up with six reviews, seven if you count the site that simply slapped a score on our press release.

I resigned myself to the new fact that app sites generally existed to fleece ignorant indies out of their money, and given my experience vowed that I would not waste my hard-earned dollars on review sites ever again.

Last month, I happened across AppyNation's: Hall of Infamy. It was a list of sites that engaged in the practice of accepting money in exchange for reviews and encouraged developers to add to the list as they came across sites doing the same.

All I could think was, "But don't most app review sites do this?" Certainly based on my experience with the hundreds I contacted, sites taking money for reviews seemed rather the rule than the exception. Pay or we'll ignore you completely.

App review sites. Like headcrabs for your wallet!

In a month's time (fingers crossed) when we're ready to launch our next title, Vex Blocks, I'll still put out a press release, I'll still attempt to raise awareness of our title any way I can, but you can be sure I'm not about to hold my nose and shell out for any more "expedited" reviews ever again.

It didn't improve my game's visibility, it didn't push more downloads – it was simply a waste of money. It shouldn't be the norm for review sites to ask for money for their reviews.

As the indie development community grows, these sites will continue their parasitic ways by preying off developers desperate enough to throw what limited funds they have away in the hopes of giving their game a better chance of success.

If a site is reputable and getting decent traffic in the first place, they don't need your money from reviews. I was fooled by thinking this was the norm simply by the sheer amount of sites doing this. Don't prove another willing host as I did.

[This piece was reprinted from #AltDevBlogADay, a shared blog initiative started by @mike_acton devoted to giving game developers of all disciplines a place to motivate each other to write regularly about their personal game development passions.]


Related Jobs

Blizzard Entertainment
Blizzard Entertainment — Irvine, California, United States
[09.17.14]

SENIOR SOFTWARE ENGINEER, GRAPHICS
Blizzard Entertainment
Blizzard Entertainment — Irvine, California, United States
[09.17.14]

ASSOCIATE SOFTWARE ENGINEER, TOOLS
Trion Worlds
Trion Worlds — Redwood City, California, United States
[09.17.14]

Senior Graphics Engineer
Playdek Inc.
Playdek Inc. — Carlsbad, California, United States
[09.17.14]

Artist










Comments


Martin Edmaier
profile image
I agree on your article almost. But i think depending on your question i would say paying for an appreview depends on the price or on the package you get. I would also paid or paid more if i can support a other company that just started.

Steve Fulton
profile image
I'm so glad you wrote this. We had nearly the exact same experience. It's so unlike the world of viral Flash games, where everything was about getting the developer access to get their games out while as the portals and aggregators that paid for the privilege. I get the feeling that the situation will not be sustainable in the long run.

Maria Jayne
profile image
"It shouldn't be the norm for review sites to ask for money for their reviews."

I think it's despicable, but then....why shouldn't they ask for money when you and others give it to them?

It's the equivalent to spam emails about Mr Nairobi inheriting 10 million dollars and just needs you to give them $100 to secure a percentage of that. You can send out a thousand spam emails but you only need one sucker to make a profit.

Perhaps, rather than a name and shame list, which is basically a place to check sites to avoid, there is a list of decent sites, where developers could go to contact reviewers. But then you need someone to maintain that list who is themselves, not open to corruption.

Marc Schaerer
profile image
I personally hate the 'pay to get reviewed' as a norm, most pages that offer it have 0 penetration and should be glad for any free key they get to professionally review to change that.
On the other side, time is limited and so is 'visible space' so there needs to be some form of selection and doing that with tens of thousands of submissions a year just takes staff that has to be paid. As such a nominal fee is a good way to filter out 'the crap right away' or at least hope to do so by making it costly to send in trash.

But otherwise paying for review stinks as much as GameSpot and IGN do which survive basing on the advertisement in their mags and pages which in consequences forces them to create fake articles and reviews as we've seen in 2010 and 2011 with head staff from the review teams were fired after stating the obvious on really crappy games that wasted millions of USD in development etc, just cause said publisher didn't like that fact and likely threatened to cut the ad budget

Lex Allen
profile image
Getting your game for free should be payment enough.

Benjamin Quintero
profile image
Isn't that the same as saying, "giving everyone your game for free should be payment enough for you, knowing that I might play it someday..."

A man has to eat and free anything doesn't put food on the table.

I'm sure there are rotten people out there (most are) but there are only a handful of methods for journalists to actually make a salary; pay-for reviews (common in the book writing business) or advertisement (a little more common on game review sites).

Thom Q
profile image
Benjamin: App-reviewers are journalists?? Since when are reviewers of anything journalists?

Toby Grierson
profile image
Journalists need to eat and free anything doesn't put food on the table.

Reviewers are not journalists.

Thom, if you point isn't then that "reviewers don't need to eat and can survive on free things", than you wasted a post and might as well have chided him on how he used an ellipses.

Ian Fisch
profile image
I think that if you can't get a legit site to review your game, there's a problem with your concept.

If websites don't think it will attract eyeballs to their sites, how can you expect it to attract eyeballs on the appstore?

What's the hook for Itzy3d? Certainly not the art/graphics. Controlling a spider could be cool, but it's not as if spiders hold a special place in pop-culture right now the way that superheroes, vampires, zombies, and hobbits do.

Ramin Shokrizade
profile image
When I was the senior writer for www.unknownplayer.com from 2001 to 2005 (it was destroyed by hackers in 2005) I did reviews as a volunteer, did not take money, and we did not take advertising dollars. My reviews were clean and objective. For this reason, those companies that were used to paying for good reviews (I won't name any names here) tended to NOT want to be reviewed by me.

I did consider myself a journalist, still do, and I did more than just reviews (and obviously still do).

That said, the lower barriers to entry in the gaming industry now make for just such a huge amount of noise that there is no way to review everything anymore. A new business model for reviewers is needed. If that means you take 100 USD from every company wanting your review, that does not seem unreasonable to me. The real trick is to be able to take that money and still give an objective review. Obviously you need that money paid in advance, and you need a good enough reputation that readers will trust your objectivity. Likewise, if your reputation is good hopefully people will not send you crap to review since they will know you will just tell everyone you reviewed some crap.

I can totally see someone charging $20/HR to do a review if they are good, and an additional $100+ to send a private technical report to the game makers giving your suggestions on the game. If your site gets a really large number of visitors like UP used to in the day, then those numbers could be a lot larger. This is still a good deal relative to traditional advertising costs for those making good products.

A nice disclaimer on your site could make it clear that you do not allow any product to be advertised on your site until after it has been reviewed. Thus the real junk is unlikely to be advertised, and you might set a minimum review rating for any potential advertisers if you want to be ethical about what you allow to be advertised on your site.

As I am looking for good tattoo artists at the moment, their business model reminds of of some additional possibilities. You could have the senior reviewers/writers charge a much higher rate per hour, and for those minor products that want a review but are price sensitive, you can have your junior reviewers do those reviews at a discount. Everyone wins in this situation as your less experienced writers get the practice they need on less sensitive reviews.

Curtiss Murphy
profile image
I got those emails, but the return on investment seemed iffy. So, I ignored them and took what I could get.

I think, I've come to realize that marketing is not as important as I was led to believe. Yes, it matters, but first!!! I need to improve my own skills. I need to really understand what it means to build a GOOD product (relative to my market). So, that's my plan - learn to build GREAT products! Then, learn to market too.

Alex Kolakowski
profile image
Sounds like a conflict of interest ripe for abuse to me.

Kyle Redd
profile image
People seemed to be widely critical of this practice by app review sites. They should be. Honestly though, not only am I not surprised that sites would engage in this sort of behavior, I don't think it's that much worse than what the major gaming sites are doing.

Critics at sites like IGN and GameSpot would never accept cash, of course, yet they have no qualms whatsoever about taking tens of thousands of dollars a year in free games, hardware, and gifts from the companies they cover. Not to mention the even greater amounts of money in ad revenue from the same sources.

Kyle Redd
profile image
Well, how much money do they need to operate? There are an abundance of independent, reputable, long-running gaming sites out there (like outofeight.info and gamecritics.com) that get little or no ad money whatsoever and still manage to provide plenty of quality coverage.

Ultimately, I think getting all of your ad money from the companies you cover can still work, to a degree (though it can also result in a complete disaster, as happened with GameSpot and Jeff Gerstmann). But in order to have *some* level of credibility, these sites would need to at least start providing proper disclosure in every review detailing who paid for the games, and more importantly they need to return the games once the review is completed. There would also have to be a strict policy against receiving gifts of any sort.

If the people working at these websites want to be seen as journalists, that would be the bare minimum of what they'd need to accomplish to do so, in my opinion. In reality though they're more like promoters, or maybe "enthusiasts" if we're being generous. There's no real pretense of impartiality, and most of their readership is probably fine with that.

Kyle Redd
profile image
Oh god, really? I said that game critics have to starve to be legitimate? That's lovely. Nice discussing this topic with you, Joe.

Jeremy Reaban
profile image
No, you shouldn't ever pay for reviews.

But at the same time, speaking as a reviewer for a web site (PSP Minis, not mobile, but sort of close. Gamerankings counts us at least), it's a pretty thankless job with little to no reward. (Our site has advertising, but I don't handle it and the money goes towards hosting and buying the games we review)

I spend 4-5 hours playing a game I probably wouldn't want to play of my own free will (recently having to play and review a 4th Farm Frenzy game almost broke my sanity), then another 2-3 hours writing and re-writing a review, then another hour tracking down screenshots and such, and then finally posting it.

I've done about a 100 of them in the last two years. The only reason I still do them is because it's developing my ability to write on a tight schedule (I write one, sometimes two, a week)

So I can certainly understand other people wanting to be paid. When you like the game you have to review, it's not really work, either to play it or to write the review. But when you don't, it's most certainly work, especially trying to stay objective.

Greg Bauman
profile image
I would be suspect of any site who charged a small amount of money to review a game at all. It takes time & effort to play, write, edit, layout, publish and maintain content for a website. Certainly $100 is WAY too small a sum for many games - even indie ones that tend to be shorter time commitments overall. Are you paying $10 an hour for 10 hours? $20 for 5? You get what you pay for, right? But who is to say that this cost must be conveyed in cold hard cash?

It seems plain to me that designing a business around the relative success of a game (indie or not) is highly dependent upon the PR/journalist/reviewer RELATIONSHIPS you have every bit as much as it is on making a good game.

It is possible to make a GREAT game and have it succeed without those relationships. But that is like hitting the lottery - it's that hard, and the odds are stacked against you nearly that badly. Big publishers know this all too well - and it seems the indies are getting their dose of the same medicine on a much smaller magnitude.

No developer can expect to receive significant coverage on any publication for free AND be a total unknown to them. You must have those relationships already in place when your title ships. This is the entire purpose of building a PR department, even if the solo dev IS the part time pr guy too. The whole relationship between those "in-house" staffers and the "external" reviewers is one of mutual trust.

The tacit agreement between PR folks and the reviewers is that the developer is not going to waste the reviewers' time with a game that is a piece of junk. Too many times reviewers get burned in this equation, and so they tend to have thick skin on hearing "this game is awesome" from every developer or publisher they encounter. In those cases where trust has broken down, those reviewers spend time reviewing a game that IS frankly garbage, and THEY eat the dollars involved in building a review for a game that ultimately no one will buy. And, as a corollary to that, the amount of traffic the publication will get will also be substantially lower, too. That's adding insult to injury. It's easier to simply charge for the review to cover the time, although I would add a caveat that if this is the business model for a given website that they disclose this up front to their entire audience.

PR departments know when the game they have is a dog. They might be more disciplined and diligent about selecting specific reviewers in those cases. Heck, they might not send out review copies at all. There are plenty of games that follow that exact strategy. But you can bet your butt that if the PR guys believe the game is solid, and they value their own reputation, they will still cherry pick the best people they can consider for the review.

Knowing which editor at a publication handles which genres is a good example of how this happens. Or, being aware of which reviewers work as independent contractors and *could* get the game on XYZ site - that is the job of the PR guy. If our beloved indies are going to be the PR staff, they better be every bit as good of PR guys as they are programmers/designers/artists, etc. Otherwise, hire someone who can do it right.

It seems to me the take-away here is not to expect good results when you have not developed the relationships. Just like hiring the right artist to make your game look & feel the way you want it to - take time in cultivating the right contacts for your game to be written about. And, build these relationships over an extended period of time. Indies in particular have the ability to crank out many games in the same period of time as the so-called AAA games. And, it is likely indies won't have massive budgets to spend on advertising (aka damn the reviews, ramming speed!) So, those indies are going to be EVEN MORE dependent upon those relationships for coverage.

If reviewer John Smith over at RandomGameSnobs.com doesn't know you, nor trust you, why should he bother with the review at all if he's not earning part of his living out of it, at a minimum? You stand a much higher probability of having him review it if he knows you, or at least if you have spent time cultivating a proper relationship.

In my opinion, the very idea of "having a game finished, and then the PR starts" is a terribly risky move for a business.

Pete Morris
profile image
You couldn't be more right. Relationships are the key, and you need to start building these from the beginning, not when your game is already released.

If I'm going to spend time looking for coverage, I'm not going to start at the bottom of the barrell, emailing Z-list blogs for coverage. I'm going to look for blogs that id be proud to get coverage on. Find the top 10 blogs, figure out who the reviewers are on social media, discover what they like and don't like, start to engage with them.

Most importantly, I'm not going to just send an email saying "hi, I'm a stranger. Please can you do something for me". I'll be looking for opportunities to serve, opportunities to help the people that I want to notice me. Reciprocation is a powerful thing.

Kyle Kulyk
profile image
" If our beloved indies are going to be the PR staff, they better be every bit as good of PR guys as they are programmers/designers/artists, etc. Otherwise, hire someone who can do it right."

I'd love to be able to have our group hire a pr guy, but let's be realistic here. What indie developer can afford to shell out for a PR or sales guy working on their first few games? I agree that building relationships is important, but getting noticed as an unknown prior to your game's release? Why would anyone want to talk to you when you're working on a kid friendly spider game for mobile while instead they could be starting console flame wars or debating the merits of the next Halo? I worked on cultivating relationships prior to our game's release and my experience is no one is interested in speaking with you. We certainly had better luck as an unknown with a finished game than we did as an unknown without a game in terms of receptiveness to our communications. And that really shouldn't surprise anyone.

I also don't know where the impression came from that we're advocating simply spam emailing review sites and expecting results. Sites were researched, sites were followed, where possible I tried to find out about the reviewers, I explained our story, I explained how we thought our game might be worth looking at for their sites. I did my best to make sure the effort was there to reach out and personalize each review request but in the end, I'm still a stranger with a game that needs reviewing. No one is realistically going to start up an email correspondence with a stranger with a game to review or an unfinished product. And if you don't have an active scene in your area, how on earth are you going to develop this type of PR relationship network while spending the bulk of your time working on your product? I think you've oversimplified this issue. In reality, "build the relationships" or hire a PR guy aren't solutions to the real problems indie developers and start-ups face.

Alec V
profile image
I wrote a similar piece about my own experiences with these sites (I didn't try any of them out however). I'm glad the writer came to the same conclusion as I did, though. Hoping that other devs will also "just say no" .... http://www.juggleware.com/blog/2012/01/just-say-no-to-extortion-o
r-why-im-not-paying-for-a-good-review/

Moises Zet
profile image
This story is what the adherent problem with what the review industry has become and not really to place blame on the reviewers alone, since I'm pretty sure this all started with a big publisher hyping a game and paying reviewers to keep the hype going. But I believe this has gotten out of hand, I don't know how many times in the years past I have seen a game, been "Meh" about it only to read "It's the greatest game YET!" play it and it's absolute trash. I always wondered why, but once I got my foot in the industry and made some contact in high-places I know why. I honestly think, this is an issue which will not resolve itself as long as big companies keep making this an acceptable practice for reviewers to practice. The world of Videogames, has for the most part turned into the music industry. Do you really think what is played on the radio is the best music available? Not even close, but when that palm gets greased its hard not to stroke the hand that feeds.


none
 
Comment: