There have always been rumors that the game ratings on the Xbox 360 and Xbox.com website we’re being manipulated by game developers. On March 23, 2011 rumor became fact when Robert Boyd noticed his game Cthulhu Saves the World slipped from #6 on the top rated list to #11 in a week and showed a suspicious jump in the total number of votes. During the same time frame the game College Lacrosse 2011 rocketed into the Top 10, propelled by a request to fans to rate the game 5-stars on FaceBook.
The conversation has mostly moved around which developers were suspect. Who was up voting their game, and who was down voting other games. Cthulhu Saves the World now sits at #29 in the Top Rated list. To put this into perspective, I asked Robert about the change in daily sales since his games have changed position.
The ratings started dropping in the middle of March so I've provided
our daily averages for February & April.
70 BoDVII average - Feb
40 BoDVII average - April
83 CSTW average - Feb
53 CSTW average - April
The CSTW drop is likely mostly due to it being a new game and
naturally dropping over time. However, the BoDVII drop is much more
obvious since the game had been out for nearly a year before this
BoDVII is Breath of Death VII. Doing some armchair math, we can put a dollar figure of sales potentially lost by this change.
| ||Feb ||Apr ||Cng ||Price ||Day Loss ||Month Loss |
|BoDVII ||70 ||40 ||30 ||$1.00 ||$30.00 ||$900.00 |
|CStW ||83 ||53 ||30 ||$3.00 ||$90.00 ||$2,700.00 |
| || || || || || ||$3,600.00 |
$3,600.00 USD is not much to Microsoft, but that is a huge chunk to an independent developer.
Manipulation is Wide Spread
I used Robert’s games as an example of the impact, but it’s not limited to just a few games. Rampant down voting has been present in all the top titles. Here are the number of games over time with 4 star or better rating.
|Week / Rating ||4 ||4.25 ||4.5 ||4.75 ||Total |
|2/27/2011 ||62 ||17 ||11 ||1 ||91 |
|3/6/2011 ||64 ||16 ||11 ||1 ||92 |
|3/13/2011 ||68 ||14 ||11 ||1 ||94 |
|3/20/2011 ||64 ||13 ||10 ||1 ||88 |
|3/27/2011 ||62 ||7 ||8 ||1 ||78 |
|4/3/2011 ||54 ||6 ||6 ||1 ||67 |
|4/10/2011 ||37 ||11 ||2 ||1 ||51 |
Before the ratings manipulation we had over 90 games with 4 stars or better, now it’s just 50. It’s clear looking at the volume of votes there is a good deal of ratings warfare happening, unchecked. I have data of games by week, the ratings they’ve held, how much they’ve changed, and what the average vote would be to cause that change. I wrote a bit on this last week. I’m not posting the list of games (yet) as I don’t want to focus the conversation on developers. The focus needs to remain on Microsoft, who continue to stand by and do nothing while this takes place. I will talk about one game however.
Anatomy of an Exploit
I stressed on my numbers post, in the community forums, and directly in email to Microsoft how simple the ratings exploit is. It’s clear looking at the data that it’s well known. Some think you need legions of fans, willing to vote at xbox.com, and set aside morals to attack other games. Others believe you need to create a voting bot to automate the process. I believed it was much simpler than that, and I set out to test this theory Friday. I shocked even myself how easy it was.
The steps: Visit xbox.com and click on “Sign Up”. On the first screen in the email box, enter a fake email address. On the second screen fill out the form with fake data and complete the captcha. Once you click okay, you now have a fake account that can rate games. There is no verification of Xbox Live Accounts.
I used an form auto-complete extension for FireFox so the only thing I had to do manually was enter the captcha. I did not use any automation software, not even Fiddler to replay an http request. I could cast a vote for my game IncaBlocks every 20-30 seconds, depending on how many times I messed up the captcha. I spent a few minutes in the morning voting, a few before lunch, and a few before I went home. About an hour spent in total, and I had calculated 100 votes would move us from below #1400 to just around #100.
Like that guy in office space, I must have made a decimal point error somewhere. Saturday IncaBlocks launched to #28 in the Top Rated list. One guy, with a web browser and 60 minutes of spare time can move a game from the bottom of the pack to the front.
(The point here is to show how easy it is to manipulate the lists, but to share the rest of the IncaBlocks story we went from getting 0 to 1 trial downloads in a day to 15 trial downloads in a day, and no sales in either case. No amount of rating manipulation is going to make IncaBlocks a fun game! I’m also okay with this, it was a learning experience and that experience is payment enough. If Microsoft decides to pull IncaBlocks over this, it never made the $150 requirement to get paid anyway and I’m fine using it to force Microsoft into action. It would prove Microsoft is capable of taking action, even if they haven’t.)
Microsoft Responds, but only to WP7 Issues
Around the same time as this issue was found, a WP7 top lists issue was reported. Microsoft responded the same day as it was reported with a personal email address and phone number. “Top Men” are assigned the case and it’s fixed in a day.
The Xbox Indie problem so far has only gotten a two tweets from Microsoft, and a post by a developer that Microsoft is still trying to figure out who should be in charge of the issue. (I must blog too much as Microsoft doesn’t respond to my emails *sniff*).
Both WP7 and Xbox use the same site “AppHub” for support. It’s clear Microsoft has budget to support WP7, but none to support XBLIG. Even if they cannot decide who is in charge, they can shut down the exploit at Xbox.com by temporarily removing the ability to rate.
The system of exploiting ratings though Live accounts is large. There is a story breaking now that a design firm may have commissioned 5,000 fake accounts for manipulation.
My Personal Thoughts
In all of this I have one big thought – if Microsoft cannot manage a small group of Xbox Indie games, why would I ever get involved in the larger WP7 Marketplace? How do I know that in a year the WP7 Marketplace won’t be neglected like XBLIG? In both marketplaces, WP7 and XBLIG, Microsoft holds back features for Microsoft Partners and sends me the message they are not as interested in developers as they are in publishers. This is just my opinion, and others will disagree, but right now I’m not feeling Microsoft understands the needs of independent developers and are only interested in giving their big partners a heavy advantage.