Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
July 25, 2014
arrowPress Releases
July 25, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Diablo III Economy Broken by an Integer Overflow Bug
by Max Woolf on 05/08/13 03:10:00 pm   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

Diablo III, Blizzard’s highly-awaited online-only Action RPG released almost a year ago to the day, has had its share of technical difficulties. From Error 37 to lag spikes that can cause hundreds of hours to go to waste, Blizzard has spent the past year improving the game backend to better accommodate the millions of active players.

Diablo III is also noted for its economy, with an emphasis on a region-wide auction house where players can trade one-in-a-million items for millions and billions of gold. (inflation is crazy). Additionally, Diablo III emphasized the use of a Real World Auction House, where players can sell gold or items for real world cash.

Today was the launch of Patch 1.0.8, a patch which promised improvements to character progression. After spending a few weeks on a Public Test Realm, where players volunteered to tested the patch to ensure that there were no game-breaking exploits, the patch released successfully.

Except for one patch note that was added last minute and not tested in the PTR. And it’s a patch note that broke the economy to tiny pieces.

The patch notes from the final build contained this change:

The stack size for gold sales on the auction house has been increased from 1 million to 10 million.

Normally, on the Real Money Auction House, the player can only sell money in 1 million gold increments. In 1.0.8, the player can sell it in 10 million increments. So, what happens when the player tries putting an absurd amount of money on the Auction House?

Reddit user tyropro has a nice explanation:

The gold “dupe” involved creating a RMAH auction for billions of gold while staying under the $250 limit. The example I saw in a video was 6 billion gold (600 x 10,000,000 at $0.39 per stack, for $234). When they posted this auction only ~1.7 billion appeared to be for sale, with the rest “missing” until they sent it to their stash and ended up with more than they started with. The exact numbers from a duping video:

Create RMAH auction for:            6,000,000,000 gold
Auction shows up as:                1,705,032,704 gold
This much is missing!               4,294,967,296 gold
The missing amount, divided by 2:   2,147,483,648 gold

2,147,483,648 (or 231) is the maximum value you can store in an int32 in programming. I’m no programmer, but I took one class in high school and was taught about the limits of different variable types. See:http://stackoverflow.com/questions/94591/what-is-the-maximum-value-for-a-int32

Simply put, their RMAH gold selling code wasn’t written to handle numbers over 2,147,483,648 properly, and the result was duplicate gold being added to people’s stashes.

4,294,967,296 is also 232, or the bound on an unsigned integer, which would be an interesting implementation choice on Blizzard’s part.

And so, the dupers created these 6-billion-gold auctions which only appears to sell as 1.7 billion (and therefore only had 1.7 billion deducted from the current balance), canceled them, and were fully refunded the 6 billion for a net profit of the difference (4.2 billion). Repeat ad nauseum.

A popular game streamer showed off the exploit. Other streamers followed. It was easily reproducible and everyone knew how to reproduce it.

Hilarity ensued.

diablo31

Yes, that’s 420 billion gold.

After buying up all the duped items in the Auction House, the items were then sold on the Real Money Auction House. For real money. And people bought them.

This is the definition of a worst-case scenario for Diablo III. All because of an untested patch note. What could Blizzard do? Performing a roll-back would wipe all progress obtained by players for the patch day, which would result in a lot of bad PR. But leaving the economy as-is will devalue all items in the game (and Diablo III is all about getting items).

In the end, Blizzard has not done a roll-back, but instead banned anyone who duped, and refunded anyone who spent real money. The bug was temporarily fixed by reverting the patch note which caused the entire mess.

Let this be a lesson on what happens when you include an untested change at the last minute. You could break an entire economy.


Related Jobs

Cloud Imperium Games
Cloud Imperium Games — Austin, Texas, United States
[07.25.14]

DevOps Engineer
Cloud Imperium Games
Cloud Imperium Games — Austin, Texas, United States
[07.25.14]

Animation Programmer
Cloud Imperium Games
Cloud Imperium Games — Austin, Texas, United States
[07.25.14]

Server/Backend Programmer
Cloud Imperium Games
Cloud Imperium Games — Austin, Texas, United States
[07.25.14]

Lead Online Engineer






Comments


Ramin Shokrizade
profile image
At this point every single prediction I made, in writing, about the D3 RMAH six months before it launched has come true:

http://gameful.org/group/games-for-change/forum/topics/smedley-s-
dream-part-1-2-predictions-of-the-diablo-3-rmah

Jason Lee
profile image
Spoiler Alert: Ramin is really Nate Silver in disguise

Chris Clogg
profile image
Haha that's hilarious Ramin. Nice read.

Ramin Shokrizade
profile image
Have you guys read my "Zynga Analysis", also from 2011, where I predict the Zynga Implosion at least a year before it happens? : http://gameful.org/group/games-for-change/forum/topics/zynga-anal
ysis-1

Yea, now I'm just showing off :)

Mike Weldon
profile image
This the part where you ask us to buy your book. :)

Ramin Shokrizade
profile image
I'm glad you mentioned that, I just started working on "The Engagement Equation" (working title), my first book :)

Samuel Batista
profile image
I will buy it, you seem to know what you're doing :-P

Jonathan Murphy
profile image
When money is involved, even virtual I've seen entire economies ruined from blatant human greed. Everquest 1-2, Final Fantasy 11, and Wallstreet Kid(joke). The balancing factor is you are making a game. Not a casino. Just let the players have fun. Phantasy Star Online did this really well!

Oh wait, Activision published this... Sigh.

Simon Ludgate
profile image
I'd point out that a similar dupe happened in Asheron's Call, whereby trying to buy too many expensive items rolled the cost into negative values and the store owner would hand you tons of cash for buying tons of pricey things. They dragged their heels for a few WEEKS before doing a FULL rollback.

The fact that Blizzard hesitates at a 1 day rollback is laughable... but they might not be able to roll-back the real money transactions. Their monetization design may have denied them the single most valuable tool in economic correction.

Andy Mussell
profile image
I wonder if any D3 player can trust that the game's balance is important to the developers from this point onward. And if game balance has no value, does the game has value? And then why shouldn't a current player just cash out before it turns into (even more of?) a train wreck?

Since they're refunding the transactions, I also don't see why a rollback is a question any longer. Rolling back the servers in the hope of keeping the whole thing going for a few more years, if they can, seems much more preferable than watching your player base, and your cut from RMAH transactions along with it, evaporate in the next couple months. Perhaps the issue isn't the player base, but the Activision Blizzard stock price? Although I don't see how a rollback would affect that.

Erin OConnor
profile image
I mean who could have ever foreseen such an even occurring?

Jason Bentley
profile image
/assuming this isn't satire/

Any competent programmer with experience at bug testing in interactive systems?

Ramin Shokrizade
profile image
This specific bug would never have arisen if the game economy had been stable. As I pointed out in my paper the economy in D3 would rapidly collapse, leading to currency devaluation. This is why Blizzard had to increase the size of the money stacks sold in the RMAH. While this was probably anticipated (at least I hope it was) by someone in the design staff, this subtlety was not communicated to the programming staff so the latter group did not know they needed to add this sort of redundancy on the first pass. There were probably so many numbers involved hidden all over the place that unless you planned for this event like we did with the Y2K bug, this was an inevitable result.

One of the first critical bugs I found in the EVE Online economy in 2003 was similar, but went the other direction and involved a rounding exploit. There CCP had to add a digit on the other end to fix it, by allowing "tenth" digit numbers like "1.1".

Wylie Garvin
profile image
@ Jason Bentley:
Of course it was satire

Jason Bentley
profile image
haha, I should have said "ignoring the satire"

Erin OConnor
profile image
Yes. My comment was satire in reference to Ramin's prediction.

He seems to have some real insight into the workings of virtual (and I am sure real) economies.

PS. Awesome on the book. It will be an interesting read for sure.

Ron Dippold
profile image
Ha, very nice detective work!

Blizzard's cavalier lack of testing and version control in general actually makes it amazing that they went this long without this kind of problem. On WoW test servers bugs would appear and disappear seemingly at random, apparently where whatever dev had pushed his build would lose piles of other fixes, including bugs fixed in mainline, and old bugs fixed on test servers would show up again in production patches. Or you'd have major bugs repeatedly reported on test servers that never made it into mainline and woud be frantically patched later.

Okay, realistically they probably had more testing for D3 Auction House than WoW (since they would care), but it's never seemed to be a strong culture thing there. Non-AH patches for Diablo show the same pattern, though I was never on the test servers for that.

Benjamin Quintero
profile image
I cant even be mad at the farmers :). This is just pure gold... Couldn't resist.

David Tran
profile image
I'd like to ask a related question, and I don't think anyone in the gaming industry has really looked at this. What's the risk of the pushing and shoving over trying to force all online retailers to pay sales tax of the person purchasing going to do to Activision Blizzard?

It's not law yet, but I understand that the law currently is in the House after clearing the Senate. The idea of the law is to 'even the odds' for bricks and mortar sellers, but it also raises a very interesting question for virtual goods.

After all, we're selling rights to a virtual good, and last I checked, the entities were real people with Blizzard acting as escrow, meaning that if the law is passed, Blizzard would be subject to all 50 US states sales taxes for compliance reasons, and quite possibly all the county codes, depending on where all their sellers and/or buyers actually are.

... That question makes the mistake above sound rather trivial. I don't think you can roll back out of this one...

Ramin Shokrizade
profile image
The law is a necessary step to close a real world economic exploit being used by Amazon and others to destroy small businesses and undermine tax-funded public entities. I think the issue that is most concerning you is whether this means that virtual goods can now be taxed. I'm not sure if the online sales tax applies specifically to virtual goods. Taking that next step is one I don't think Congress is ready for because they don't understand the relationship between virtual economies and "real" economies.

When they do take that step, it will be interesting to see how it would apply to entities based outside the USA selling virtual goods here. I don't think gold farmers, for instance, are going to want to register to be taxed and thus we might need a "virtual police force" to monitor virtual goods movements. Again, this is way beyond what Congress is ready for.

David Tran
profile image
(Can't reply directly, so I hope this come out right.)

Well, even though no individual user will make the threshold (It's 1 million, and if a single user makes a million off Diablo 3 in a year, they're probably going to make the news and be the exception) Activision Blizzard would have to keep track of every user regardless, due to compliance issues.

It gets more complicated of course, if say for example they're running multiple games, since some gold farmer operations CAN (in theory) clear the million as a collective or corp, due to additional accounts and combined efforts. Or at least that's the line I imagine the IRS will run when they call in the auditors.

They'd require the tracking and Blizzard as the platform provider would have to track then provide in audit form proof to any parties knocking that they remitted taxes, and/or can charge them post due to IRS (or another tax related entity) ruling that a group of accounts cleared the limit. Being the tax collector is something that's going to make them look bad.

The question becomes very interesting, because how it's currently phrased also takes square aim at any virtual goods as a side effect to the law.

Mostly because the law recognizes there's a loophole by trying to charge everything as a service, and not charging for the physical good in an attempt to tax evade. eg fraudulently charging nothing for the physical good, and charging 100% of the cost as the service of delivery as a service. A virtual good (in D3's case at least) is basically 100% delivery, as you don't technically own the actual good, as per Blizzard's ToS.

Congress is sort of landing collateral damage, but that's how government entities are with wide laws.

The real question we should ask is if Blizzard (or anyone else for that matter who think about implementing this sort of revenue stream for their games) is prepared for the regulatory requirements, which well, involve tracking every single transaction on the expectation they'll get randomly audited by the states.

The fun part of course is the fact due to its status, this threat is going to be around a while, making this a running regulatory risk, even if this iteration is defeated in the House - the supporters are not going to just give up if it gets knocked back... even though none of them would have considered its effects in gaming.

Previously, it would have been any states they had nexus in (I forget where Activision are incorporated, I'd have to look) but now it could easily look like they have all 50 states of the US knocking on their door, requiring them to start learning the other 49 tax codes and put up with the various disruptions.

And if this passes, I can't imagine the local counties would pass up this revenue stream either, and will come in sabres rattling.

I think after a while (if the law passes) we'll probably see the requirement of a virtual monitoring force, and forced registration. Granted, this would require a massive ramping up of the amount of resources to actually monitor this, and the question becomes if this implementation is net positive after the costs of collecting all this untapped tax is put into place.

The IRS does claim that they have rights to collect outside the US as well.

... The more I think about this, the more I suspect it's a massive can of worms no one's ready for, except we might end up opening this can completely by accident, to prevent people from evading paying tax for a completely different arena.

Aaron Eastburn
profile image
@Ramin
"The law is a necessary step to close a real world economic exploit being used by Amazon and others to destroy small businesses and undermine tax-funded public entities."

Thank you, this is probably the best description I have read of the current online sales tax moratorium. I have your papers bookmarked and I am looking forward to reading them this weekend when I have some free time.

Ramin Shokrizade
profile image
While this is probably off topic, it is really very important to all of us. If possible, this might push a lot of development offshore. Not just because of the tax, but the cost of administering this sounds a bit scary. If you could keep me updated on this directly, I would appreciate it.

Andrew Sega
profile image
It's really not rocket science here, you don't have to be Nate Silver to figure out how broken the RMAH idea was.

- Gold (and items) spawn infinitely in the game world
- People pick them up, and are constantly getting richer
- There are no real gold sinks
- Eventually everyone has lots of gold, and gold become pretty worthless
- It also becomes easier to buy items than farm them, since you distribute the randomness across the entire player base (someone, somewhere is going to randomly get the item you want, and now it's easy to find it)

The fact that there was a 32-bit integer bug, while cute, was just a side effect of a fundamentally bad economy... they would have been better off allowing no auction house and just have Blizzard selling items to players directly

Rindel Ryan Ibanez
profile image
They're saying in the official Diablo 3 forums that they're going to ban, penalized and rollback those players who abused this "bug". But would the "offenders" really care? What if they already encashed the money they got from selling duped gold? What would Blizzard use to refund the people who are going to lose the things they bought? D:

I think this damage is irreversible.

Jaco van der Westhuizen
profile image
I once played an online space strategy game where you could split stacks of resources or spaceships, but it asked for a number. I decided to try entering a negative number, and it worked. :)

When I reported the bug, the servers got shut down. Coincidence? :(


none
 
Comment: