The Problem with Reviews
Metacritic, love it or hate it, the site has become a focal point for the entire industry. Publishers sometimes reward developers based on the ratings their game achieves there while reviewers often check the Metacritic score first, fearing to thread too far from the standard rating or of drawing the ire of fans and sometimes even the developers themselves.
Take for example the Creative Assembly, the creators of the Total War series. The company’s Tim Heaton told gamesindustry.biz that his studio seeks at least 90% on Metacritic. He even admitted that they were willing to sacrifice entire game features late in the development cycle if they felt the Metacritic average would be dragged down.
“We do what we call ‘Metacritic analysis,’” said Heaton. “So we will break those features down into subsets, and we both look at it from a player’s point of view, and a reviewer’s point of view, and we’ll weigh certain features as to how we see players and reviewers look at them, and they’ll build up to a 100 percent score, and then we’ll judge where we feel we are on those individual feature sets, and see the momentum on those and the velocity on those, too.
“And so if we see one flat line and it’s not where we want it to be, we then will cut it. Well, we’ll cut it really late in the day. I think teams are really scared about doing 90 percent of the work and then cutting it. It’s kind of like, ‘Well, it’s nearly finished; I… I’ve done all the work! Please don’t cut it! I’m sure I can make it better.’ And we’re fairly brutal on that,” he continued.
Currently the studio sits at 76 on the site largely because of the Stormwise, actually created by Creative Assembly Australia, which scored a 42. Rome: Total War comes in highest at 92.
But the Creative Assembly isn’t the only studio that has admitted that it fixates on review scores. Tony Goodman, the co-founder of the now closed Ensemble Studios and founder of Robot Entertainment, told Gamasutra how the company’s real time strategy series Age of Empires’ critical success, or certainly some of it, came down to what was effectively manipulating games magazines and their editors.
“I built relationships with the most recognized game magazines… I invested a lot of time with key editors, seeding the idea that Age of Empires would be ‘revolutionary’ and would become a ‘phenomenon’. They may not have believed me at first, but my goal wasn’t to convince them. My goal was to plant wondrous possibilities in their brains and create anticipation.”
According to Goodman his efforts paid off:
“When the first early previews began appearing, they were using the terms that we seeded: ‘revolutionary’ and ‘phenomenal’. These early opinions were then picked up and echoed by other publications, creating a snowball effect. Eventually, all the publications would get on board with this message just so they didn’t look out of touch.”
Ensemble was a great studio filled with talented people that produced, for the time at least, excellent RTS’s and while it is impossible to calculate the net effect of Goodman’s efforts, either critically or commercially, it is quite reasonable to believe that he is far from the only game maker to be involved in those kinds of practices.
I’m not suggesting that anything immoral or untoward occurred however if the major opinion forming games publications can be influenced to give a game an unwarranted amount of buzz, or a higher rating, then that can only put further strain on developers elsewhere in the industry – who may not have established extensive media connections – and on consumers, who may decide to buy a game they would otherwise have avoided.
If the later happens and you like the game great but if you don’t then you’re down $60. The purpose of this post isn’t to tell you to ignore previews and reviews but it might not be a terrible idea to check with friends first or if you’re still not sure rent or wait for a pre-owned copy to hit your local GameStop.
The problems with reviews, and the heavy reliance on them within the industry, does not end there. Take Fallout: New Vegas, which has an aggregate of 84 on Metacritic. Chris Avellone, Obsidian’s co-founder, revealed that his studio missed out on a bonus from Bethesda because it failed to reach 85 on the site.
Bethesda has neither confirmed nor denied that claim but it seems unlikely that Avellone, who was responding to a question about how much money Obsidian made out of New Vegas, would have revealed that if it were untrue.
On the other side of the coin is user reviews. Diablo 3 for instance. Six critics have so far given the game a score of 88. Of the more than 2600 user reviews over 1100 have been negative. The game has an average user rating of just 3.8. Many of the reviews on Amazon are also decidedly negative.
Of course people are angry with the controversial, to put it mildly, use of always on DRM and the problems thousands of gamers had logging in – the now infamous Error 37 – even to play the game’s single player. Yes Blizzard has a right to protect it’s games from pirating but the consumer is clearly in the rights, that is they bought something that didn’t work and was in no way their fault but that is a different argument.
Mass Effect 3 also suffered the ire of gamers in their reviews (the PS3 version in particular scored a measly 3.9). Some of this can be attributed to trolling. Much of it is due to the vastly disappointing ending.
Whereas PSN’s Journey has a critic score of 92 as well as a very respectable user rating of 8.6.
And then there’s John Beiswenger. Beiswenger is suing Ubisoft over the similarities of his book Link and Assassin’s Creed. In revenge gamers have taken to bombing the book. There are now over 200 reviews on Amazon and a score of just 1 star.
The point here, I suppose, it that you shouldn’t take reviews too seriously. Nor you should be put off because gamers are bombing a title (even if there is some justification). Yes, reviews can and should guide you, it’s what they’re there for after all but reviewers, both professional and consumer, are fallible and the problem of reviews and the effect they produce on developers is something that everyone should be aware of, especially publishers.