Metacritic Matters: How Review Scores Hurt Video Games

Using the $1 million Metacritic bonus attached to Fallout: New Vegas’ critical reception as a prime example, Kotaku has analyzed the detrimental effects that tying review scores to video games can have during publisher/developer agreements, with Obsidian Entertainment head Feargus Urquhart even lending a few quotes in the editorial. It’s really a shame that this is what we have devolved to, and yet it doesn’t surprise me in the slightest:

Perhaps you’ve heard the story: publisher Bethesda was due to give developer Obsidian a bonus if their post-apocalyptic RPG averaged an 85 on Metacritic, the review aggregation site. It got an 84 on PC and Xbox 360, and an 82 on PS3.

(If only it was a stable product and didn’t ship with so many bugs, I would’ve given New Vegas a higher score,) wrote a reviewer for the website 1up, which gave New Vegas a B, or 75 on Metacritic’s scale.

(It’s disappointing to see such an otherwise brilliant and polished game suffer from years-old bugs, and unfortunately our review score for the game has to reflect that,) said The Escapist’s review, which gave the game an 80.

If New Vegas had hit an 85, Obsidian would have gotten their bonus. And according to one person familiar with the situation who asked not to be named while speaking to Kotaku, that bonus was worth $1 million. For a team of 70 or so, that averages out to around $14,000 a person. Enough for a cheap car. Maybe a few mortgage payments.

Those sure were some costly bugs.

…

While chatting with Obsidian head Feargus Urquhart for the profile I wrote last December, I asked him about what had happened with Fallout: New Vegas. For legal reasons, he couldn’t get into the specifics.

(I can’t comment on contracts directly,) he said. (But what I can say is that in general, publishers like to have Metacritic scores as an aspect of contracts. As a developer, that’s challenging for a number of reasons. The first is that we have no control over that, though we do have the responsibility to go make a brilliant game that can hopefully score an 80 or an 85 or a 90 or something like that.)

According to Metacritic’s rating scale, any game above a 75 is considered (good,) but realistically, according to multiple developers I spoke with, publishers expect scores of 85 or higher. Sometimes, Urquhart told me, the demands can get unreasonable.

(A lot of times when we’re talking to publishers and this is no specific publisher but there are conversations I’ve had in which the royalty that we could get was based upon getting a 95,) he said. (I’ve had this conversation with a publisher, and I explained to them, I said, ‘˜Okay, there are six games in the past five years who have averaged a 95, and all of those have a budget of at least three times what you’re offering me.’ They were like, ‘˜Well, we just don’t think we should do it if you don’t hit a 95.’)

…

Another problem for developers: outlier scores. What happens when tons of people like a game, but for one or two reviewers, it just doesn’t click?

(The problem is the scale,) said Obsidian’s Urquhart. (There’s an expectation that a good game is between 80 and 90. If a good game is between 80 and 90, and let’s say an average game is gonna maybe get 50 scores, if you wanna hit that 85 and someone gives you a 35, that just took ten 90s down to 85… Just math-wise, how do you deal with that? Some guy who wants to make a name for himself can absolutely screw the numbers.)