The Ratings Game of Rating Games

Posted by Rampant Coyote on April 22, 2013

Ye olde day job has me once more on the road, although this time still in the U.S. Hopefully this time my trip to Savannah, Georgia will be a little more relaxed – 16-hour workdays last time got to be a little much, and I wasn’t QUITE as prepared with the blog posts as I should have been. While I don’t have all my posts for the week “canned,” I may not be super-responsive to the comments, depending on schedule and the quality of the hotel Internet connection.

That being out of the way…

Sometimes I get confused for being a “games journalist.” I’m really not. I don’t have the training. I really don’t do game reviews here – there’d be a serious problem with conflict-of-interest and stuff. I do like recommending games I enjoy, and I love to talk about games. It’s what gamers do. We talk about what we love. And I love video games. At least, I love many video games. I have ever since the first time I walked into an arcade and started dumping quarters into Defender, Space Invaders, and Starfire. Or saw a friend’s massive printout of his experience playing the Colossal Cave Adventure. I was enamored. I wanted to talk about it. I talked with friends. I read what few magazines there were on the subject. We compared and contrasted. We argued. We swapped stories.

This isn’t games journalism. This is being a gamer and communicating.

Flash forward thirty years, and we’ve got problems like this one with Metacritic. Sadly, there’s no two ways around it – no matter what system is put in place, when there’s money or power on the line, the system will be gamed. Hard.

I spent a couple of years working for a company that did multi-level marketing (I guess the preferred term now is “Network Marketing”) – and I was astonished. Every loophole, every idiosyncrasy would be exploited. Big time. You’d have distributors paying other distributors to *NOT* work to maximize their downline (“If I can skip you, I can make $46,000 this month, otherwise I only make $38,000 this month — do nothing and I’ll pay you one quarter of the difference.”)  That was counter-productive to everybody, except it maximized the gains of the distributor(s) at the top of the chain — and it was the tip of the iceberg. From government laws & regulations, to business agreements, to standardized test scores, to getting your game approved on XBLIG, to annual employment reviews – any system will eventually get “gamed,” and the more that is at stake, the more it will be gamed, and the spirit of the law will be violated to take advantage of the inevitable limitations of the letter.

So what really sounds like a great idea – publishers being concerned not just with being able to milk their audience for millions of sales, but also being concerned about the ultimate quality of their products – gets twisted into a system of vague pressure and politics.

I guess I also don’t understand why Metacritic doesn’t do a simpler Rotten Tomatoes type of system where the number represents the number of critics who gave a favorable review.

And  that goes back to the complex numerical scores so many review sites use. What’s the difference between an 84% and an 85%, anyway? Or an 8.0 versus an 8.1?  Some sites and magazines historically provided some kind of mathematical weighting system for calculating the score, but it seems to me like an exercise in deception – to pretend that the review score is purely objective, when no such thing is possible. I don’t know that I’d want it to be possible! I don’t think games SHOULD be something that can be entirely objectively scored.

Really – while there are certainly qualitative elements that can be pretty easily compared with each other to get something approximating empirical, non-subjective truth, games are such complex, diverse beasts that they really shouldn’t have that much in common to compare.

This is, in part, why there’s been such an emphasis in graphics quality over the decades. That’s something most games have in common – pictures. How nice are these pictures? For a while that worked okay, because it was really a combination of technology, artist skill, and programmer skill. And since technology kept improving, it was always possible to push that frontier. But nowadays, it ends up being a combination of what game has the most mind-blowing special effects and the fewest graphical anomalies. Is that really how we want to grade games?

And dinging a game on account of bugs? I totally understand when it’s a case of the bugs truly detracting from the experience. But if you recognize that bugs are simply a naturally occurring phenomenon resulting from complex software (something too few gamers seem to be able to grasp), then you’ll realize that being too quick to pick up the torches and pitchforks over *GASP* a character’s hand clipping into his hat in an animation provides a perverse reward for those games that DON’T TRY TO DO VERY MUCH. If you keep your interactions really dumbed-down and simple, easily enumerated, and easily testable, you’ll start with a lot fewer bugs, and they’ll be much easier to find and fix in development.

So even these semi-“objective” criteria really include a ton of subjective tolerances. Are you more willing to accept imperfections in exchange for greater depth and complexity? Are you willing to accept that a game taking risks with new mechanics is going to get some things wrong, compared to a game that plays it safe by honing the tried-and-true to a blinding shine?

And then we get into an area where the similarities and easily-compared elements are lost, and we’re comparing a game dealing with (for example) a theme of innocence lost and eventual redemption with another that is about… killing zombies in as graphic a manner as possible. How do you compare these two? Any commonalities will be grossly simplistic and cosmetic, and any deeper comparison might be like grading a fish on its ability to climb trees. I mean, if you ignore the superficial stuff like photorealistic graphics and quality of voice-overs, would a AAA company REALLY want its game compare to Minecraft?

I could argue that much of the lack of progress of the medium the last couple of decades could be attributable to shoddy reviews (and the influence they hold over the industry), but that might be making a bold claim that the chicken came before the egg. All I know is that it seems to have gotten us into a vicious cycle.  The problem predated Metacritic. Can you really fault a site that does its best to preserve the detail that the industry claims is an honest-to-goodness part of the source data?

So what this all boils down to is my belief that not only are game review *scores* a steaming pile of excrement, but our efforts to treat them as legitimate, objective data is what has hobbled us as an industry.

That’s not to say the reviews themselves are a problem – the text of a review can be extremely valuable not only to a prospective buyer but to anybody trying to scope out the length and breadth of the industry. It’s just the scores – which I will readily admit to being hypocritical about and checking often before reading the review itself – often present an illusion that I feel can be damaging to the medium and industry in the long run.

Really, what more do we need besides thumbs-up and thumbs-down. Maybe a thumb-sideways. Maybe an extra super-thumbs up for those games that really amaze. I don’t know that we really need an “extra bad” rating, for that matter – I know I’d not use it very often. Bad is bad.

I think that for the good of the entire industry, we really just need to wake up, embrace our subjectivity, and keep our review scores simple!

 

Filed Under: General - Comments: 7 Comments to Read



top