The Awards Problem
Posted by Rampant Coyote on April 6, 2015
At the Academy Awards in 2014, two of the voters for the film that one Best Picture admitted later that they’d never actually watched the film. They thought they’d have been “disturbed” by it, but voted for it anyway because of its presumed “social relevance.” I suspect that this happens a lot, but it was the first time (to my knowledge) that this was publicly admitted.
Winning a major Oscar can really boost a film’s financial success, so there’s an awful lot of behind-the-scenes campaigning, scheduling, and so forth to snag one of these awards, especially if it’s not a typical formulaic crowd-pleasing ‘blockbuster.’ But if this sort of thing actually happens a lot, it really brings into question the quality of the awards. It doesn’t matter about the quality of the film, only the behind-the-scenes award campaigning.
This weekend, the Hugo Awards – which historically has been a pretty influential and prestigious award for science fiction and fantasy – announced their nominations to something of an upset. The bottom line is that while there was always the kind of ‘behind the scenes’ campaigning by an inner circle with their own preferences (and political and social agenda), another group has taken this sort of “legal, but gauche” campaigning out into the public. This resulted in a record-breaking number of members and voters but a major “invasion by barbarians” reaction from the long-standing old guard. Both sides have their own views of what “fair” means, but the “invasion” (called the “Sad Puppies” campaign) was a direct result of these competing visions – what was fair to one group seemed completely unfair to the other. We’ll see how things shake out in the long run.
Extrapolating from Gaming
Now, I kind of embarrass myself talking to some friends about this who REALLY know their SF & F – some on both sides of the argument. And I’ve got friends in the movie business who know the industry a lot better than me, making me entirely unqualified to argue with them over what’s going on in that community. I’m just jaded enough that it takes more than a formulaic summer blockbuster or a by-the-numbers pot-boiler to get me excited, I can’t have the same level of vocabulary or experience as these friends. What is still new and exciting to me is old hat to them.
I do know a thing or two about games, though. Not a lot (it seems the more I know, the more I realize how little I know), but at least an above-average familiarity. And I do see some of the same problems appearing in the games biz. This concerns me. But it also brings to mind questions about the purpose of the awards and of the judges for these awards.
Pressures
When an award achieves a certain level of popularity, it can affect overall sales. This makes sense. Having an “independent” and somewhat trusted validation of the quality of a work elevates it above the usual marketing hype. But once there is some real monetary gain associated with an award… well, it’s no longer just recognition. In some ways, it becomes a sponsorship, or a grant. There’s a lot more competing pressures going on about how to award the prize.
For example, there’s the pressure to give the award to the most deserving. For example, of you have Creation A and B of nearly equal quality, but Creation A is already a success with plenty of media attention, but Creation B is relatively unknown, Creation B would definitely benefit far more from receiving the award, there’s going to be some subjective weighting to Creation B’s advantage.
Then there’s of course the social pressure of these overt or more subtle campaigns. If you hear a lot of your peers suggest that Creation A is somehow landmark and important, awesome, and that all the smart / good / perceptive critics are impressed by it, and you are faced with comparing A and B together and they are both in the same ballpark quality-wise, there’s a lot of pressure to vote for A. After all, you are smart, good, and perceptive, right? You don’t want anyone to perceive you otherwise. Even if you don’t “get” why Creation A is so excellent, you may be influenced to vote that way anyway just because that’s where all the other independent thinkers are voting, and they must know something you don’t.
The flip side of this is simply the popularity factor. Somewhat counter-balancing the desire to award the most needy, there’s a tendency to go with the familiar.
Then there’s the comfort / time factor. This is especially notable with games. Bottom line – the judges and voters have limited time, and it’s been made abundantly clear that most judges don’t play all the games in the major awards. This weights the small, tiny games that get to their point (and may basically be “done”) within the fifteen minutes. But as noted above with the Academy Awards, some judges may shy away from the more challenging fare and base their vote on the campaign buzz instead.
Then there’s simply the fact that the folks who are involved enough to participate as voters / judges may simply be so jaded that they need some novelty to catch their interest. Well-crafted twists on an old, familiar theme level them cold. So whereas something like The Avengers may thrill the mass-market audience and be considered by the vast majority of the best movie of the year, the critics have all felt that they’ve seen it before (and seen it better) in previous offerings.
What is “Fair?”
While you’d hope that all these factors (and tons more that I didn’t mention) balance out into some reasonably approximation of “fairness.” I do believe that in at least the major awards I mentioned above and some major industry awards in games, the majority of judges / voters really do try to toe the line or at least make the most ethical / fair decision that they can. But it also means that the calls for reform and improvement will frequently be justified. There’s no way to make these things perfect, but there may be times where the purpose or approach of a particular set of awards ought to be called into question and re-evaluated, and maybe the process should get overhauled to better fit reality. That, or maybe the barbarians gathering at the gate will crash the party and force the issue.
And maybe there’s a point where new awards need to get set up and promoted. For example, I point to Craig Stern’s discussion about the bias against role-playing games in the IGF awards (and probably the same could be said for Indiecade). By “bias,” I mean results, not necessarily the judges’ subjectivity. But the whole way the judging process is structured works against long-form games. I don’t see any easy way to “fix” that.
And fundamentally, that’s the challenge that we face in these industries where we do want to show public recognition for people doing really cool stuff. If it was easy, there’d be little value to it.
Filed Under: Biz, Books, Movies - Comments: 6 Comments to Read
McTeddy said,
I’ll be honest, I have ZERO respect for a judge that doesn’t watch/play something they vote on. If they don’t have the time to actually do their job, they shouldn’t have the job.
That said, amount of time played is negotiable. Some games can be learned quickly and no one would have enough time to fully play an IGF full of RPGs*.
Overall though, I’ve given up on award shows. It’s little more than a popularity contest where a few collectively decide the fate of the contest. The lack of transparency doesn’t settle well for me.
That said, it doesn’t help that I’ve heard rumors about the IGF since college and seen a couple… interesting coincidences. That particular event is one I’ll avoid like the plague.
*Maybe RPG developers could do demos like the old playstation discs? Start half ways through the game in an exciting scene with fully built characters. That might be an easier sell for such a event.
Rampant Coyote said,
I’ve been a judge on some smaller competitions, back when I didn’t have a horse in the race, and it *is* hard. And that was for indie RPGs back before we had nearly as many released as we had last year. I can’t pretend that I was perfectly objective (I don’t think such a thing even exists), or that I was somehow above all these pressures. They all weighed in my mind, yet I tried to do the best job I possibly could.
And I’m personally fond of the IGF, in spite of my frustrations. Ditto for Indiecade. I think they are worthy competitions, and I am pleased that they try so hard to recognize and celebrate indie gaming. But they really are structured more to support a particular subset of indie gaming. And… yeah, there’s an emerging mentality there which favors games that hit certain cultural hot-buttons. But I still appreciate all the work that goes into it – from participants, judges, and show organizers. I feel they manage to spotlight some worthy titles – or at least interesting ones. I acknowledge that they have a very hard job, and they do at least a credible job of it, given their structural limitations.
Like I said above, I think it is because these things are hard that they are of value.
WhineAboutGames said,
@McTeddy – I wish more people (not you, but some of the big complainy voices) who’ve voiced criticisms about the IGF would get involved in working with it.
It’s imperfect. There are a lot of things that could be done better. And the structure and culture of the competition is going to tend to push towards a particular kind of result and exclude others (like long games). But darn it, they do try.
I’ve volunteered as a judge multiple times and I’ve put in many hours playing games, not all of which are to my taste, some of which I have to stop playing fairly quickly because they are so obviously outside my knowledge set that I cannot usefully critique them. I try to seek out and bring attention to things that might be overlooked by the other judges because they’re not the right kind of cool.
But in the end, the IGF is falling into a place where there are just too many games. I went through as many games as I could manage and I still didn’t manage to play everything I wanted to play that would run on my hardware, AND THAT’S JUST A SMALL SUBSET of the games that were in the running. (I couldn’t play anything that was iphone/ipad, for example)
A lot of the games I played were good and did not get nominated for anything, because there were so many other games that were good too.
I’m not really inclined to enter the IGF myself, I don’t think the sort of game I make is the sort of game that’s ever really going to take off in there unless it was already a hit. But that’s okay, not every contest has to be about telling me how great I am… right?
(As for cultural hot-buttons, it may amuse you to know that I had a big rant about a game I won’t name because its plot was such a ridiculously superficial ‘white man baaaaaaaaad’ that I was offended, and I’m a pretty hardcore justice skeleton. 🙂 Said game did NOT win any awards.)
Rampant Coyote said,
It amuses me to no end… 🙂 As I say, these days, ya gotta have some kind of gimmick to get attention. Sad, but (often) true. But you absolutely must have more than just a gimmick.
I imagine trying to run a competition catering to the “long-form” games – RPGs, adventure games, more detailed strategy games, etc. – and I get exhausted just thinking about it. I mean, look at the non-exhaustive list of indie RPGs of 2014 I compiled… JUST the RPGs. And I know I missed a few. I can’t imagine any one person playing all of them to completion in one year (well, maybe this guy Even saying, “Well, play them for at least five hours” would be a pretty impressive feat.
I think of how much time I spent playing the horror-themed games last October for my blog article series, and … yeah. It makes me tired, even though I had a lot of fun doing it.
McTeddy said,
@WhineAboutGames
Yeah, as long as people aren’t actually giving a score on something they didn’t play I’m good.
As for the IGF, I’m just personally iffy based on my own experiences in college. Coincidences happen all the time and, even so, it’s been a long time so things could have changed.
I just don’t feel personally comfortable dealing with that particular event.
(Obviously, I’m avoiding specifics because I don’t want people taking this and yelling “PROOOOOF!”)
Besides, I don’t think I’d be welcome there. I don’t exactly fit in with either the core “Indy” community nor the SJ crowd. I’m not exactly “Pure” enough for either group and I’m too stubborn to just shut up and play along.
But I’m happy enough being in the middle somewhere. This world is too darn complicated for black and white.
As for an RPG style competition… that’s awesome but I think time would be a killer. I’m still at 2 hours in wasteland 2 and it’s an award winner! God, I miss childhood schedules…
Felix said,
You’ve just described the problems that plagued the Interactive Fiction Competition in recent years. And at least the IFComp organizers are working hard to update the rules and improve participation on both sides. Other events may have a vested interest in staying the same. Especially when there’s money involved.