The Prototype Problem
Posted by Rampant Coyote on June 24, 2013
There are parts of the software development cycle that may seem counter-intuitive to people new it. Thanks to crowd-funding and the increased visibility into the development process from the indies, gamers and crowd-funding “backers” may find themselves in a difficult place of witnessing what may appear as a train-wreck as they follow development. Oftentimes, it’s just business as usual in this crazy field we call game development. And sometimes, it really is a train-wreck. It’s hard to tell the difference.
In my experience, one of the main things that causes indie game development projects to fail is the incredible gulf between prototype and product. This is actually not a problem limited to indies, or even to game development. I’ve had uncomfortably close views of this phenomenon with spectacular, expensive failures outside of gaming.
In fact, let’s talk about one incredible failure at a non-gaming company. The previous management of the software department had been sacked because, IMO, they were too willing to speak frankly to technologically illiterate executives – and often told them things they didn’t want to hear, like how long a project would really take. So new management was brought in, and they selected a “silver bullet” system (which, incidentally, the previous management had considered and rejected). The sales team and techs from this third party were able to throw together a very pretty prototype of new software using our existing data inside of two weeks. On the surface, it looked like it was halfway to completion!
If halfway there, they reasoned, the rest of the software should really only take two more weeks to complete! They generously gave us eight weeks, just to be on the safe side. We needed training on the new framework, after all. They signed papers, spent a lot of money, and patted themselves on the back for finding such a brilliant, easy solution.
It was a disaster. After a year of major effort, a “death march” and two failed rollouts (and a ton of money and customer good will exhausted in the effort), we were forced to revert back to the old system – the one that hadn’t had any updates in a year because it was due to be replaced.
Was the software team simply unmotivated, lazy, and incompetent? Not at all. Maybe demoralized, but still they put in a good effort throughout the death march. Was it the “silver bullet” framework we used? Well, yes, partly. There were some things it could do really, really well – basically the stuff that was built in two weeks to create the prototype. Everything else, particularly the tremendous amount of functionality that must operate “under the hood” that the end-user can’t see, was a nightmare to work with. It was buggy, poorly designed, ridiculously limited, hard to customize or interface with, and really slowed development to a crawl. But even this wasn’t the main problem.
But what really doomed the project from the beginning was a lack of understanding on the part of the executives and the new management of just what all had to be handled to create a complete software system for our business. They fell prey to the prototype trap – they saw what was really a semi-functional mock-up, and failed to recognize just how much effort would be required “under the hood” to imitate (let alone improve upon) our existing system. Visually, the prototype appeared to be “halfway there.” In reality, it had tackled the “easy 10%” of the job. The savings it provided in development time for a small part of the job was far outweighed by its problems on the back-end, but even if that wasn’t the case, there real problem was that it only provided an illusion of easy development. But even if the framework had been stellar and really had sped development across the board, there was still a lot of work to do to take it from prototype to product.
Game development can be even worse. In addition to the technical challenges of making a product, games are also art. Games must chase things like “fun factor” and other aesthetics far beyond what’s necessary in other kinds of software.
It’s not uncommon for indie game developers to put forth a herculean effort to get a prototype up and running – to get that first level humming along – and then get bogged down when it comes time to turn it into a full-fledged product. It is ten times easier to hard-code something for a demo or prototype than to create a true general purpose system that can handle all the things you want to throw at it. The systems (among others) I noted above actually do a lot of work to make that easier on the developers and provide a really awesome foundation to work from, but there are still no true silver bullets or magical shortcuts for doing robust, commercial game development.
It’s really hard to explain until you’ve been there, and had to deal with those nightmares, particularly when integrating and tracking all those different pieces that go into a fully-featured game. It’s easy to feel like progress has ground to a halt, especially for an inexperienced team. And it’s easy for morale to slide, which truly will slow down progress, compounding the problems.
Entire books have been written about dealing with these problems, so I won’t go into detail about solutions here. I think it’s enough just to remind people about it here. You need to anticipate it, and plan ahead to deal with it.
For gamers, especially those who get excited about upcoming Kickstarter-funded games, I think it’s important to bring this up as a warning. It’s partly a warning not to get overly excited by what appears to be a game that – from the videos – looks “halfway done” already. It’s also a warning that there’s a point in development, that point between prototype and product – where things really do slow down. Badly. Don’t be surprised or alarmed when this happens. If you are funding an experienced team, while it’s never easy for them (especially under backer scrutiny), it’s not a surprise to them. For a less experienced team, however, the gulf may present a greater threat.
So just be aware of it, and be prepared.
Filed Under: Game Development - Comments: 9 Comments to Read
LateWhiteRabbit said,
Your story demonstrates something I’ve seen again and again. People are hired to management positions at companies solely on their “business and leadership skills” when they know jack-all about the job or industry they are managing. It doesn’t matter how well someone can manage if they don’t know the fundamentals of what the people under them do everyday.
At my current job, we have metrics and goals we are expected to strive for everyday on an individual level. A manager creates those goals everyday. I run my entire department in the company. I have a couple of new employees I’m training, but essentially I handle that entire side of things in the company.
The company uses acronyms more than the military, so that is the way the goals are delivered – in acronym form with numbers penned in by the assigning manager. One day, weeks ago, I looked at the goals my supervising manager handed me and snorted, saying they were insanely ambitious. He asked me, “How so?”
I pointed to one of the goals. “This, for instance. You’ve got me down to complete 4 of these today, when we consider it a very good WEEK if we can complete five of them.”
He stared at me uncomfortably, then admitted he had no idea what that acronym stood for, and had just made up a number. ::head explodes::
That acronym is only used by my department, but he MANAGES my department, and never bothered to learn what I ACTUALLY do everyday.
Indie developers may not always be good at managing themselves, but at least they have an idea of what it is they do everyday.
Felix said,
And that’s why YouTube is full of tech demo videos that show a little dude running around on a mostly empty plain… and then you realize that was years ago and the project never had another update. Poor kids.
Brian 'Psychochild' Green said,
I think most gamers don’t realize how many game projects get killed before anyone outside the company even hears about them. Sometimes it’s just a poor business decision that kills a project, but other times I it’s what you talk about here: a promising prototype just doesn’t hold up when the rest of the game is added.
I guess I’ve seen this so often I didn’t consider how this would impact the Kickstarter side of things.
CdrJameson said,
Yeah, Gamers only see the successes.
It’s Survivorship Bias (also known as the silent graveyard). You only see the games that actually come out and succeed, you may notice a few that fail in a spectacular fashion, but you never see those that failed to ignite anyones interest, or died during production.
This gives you an incredible over-estimate of how many games succeed, even to the point of being released.
When you start following a game from the start of production, you get the real odds.
Which are low.
Very low.
There used to be a thing about games magazines called the ‘curse of the developer diary’.
A games mag would get regular updates from a developer about their forthcoming game to give the readers an insight into how development worked. But many of these games didn’t ever make it. Was making the developer diary cursing these developers? No, clearly. It’s just that you got an unusual insight into failures you wouldn’t otherwise notice.
It wouldn’t surprise me if we end up with a ‘Curse of the Kickstarter’ at some point.
Xenovore said,
@CdrJameson: Yup.
Speaking from personal experience, often the games are far beyond prototypes, i.e. 95% finished, before they get canceled; but they get canceled anyway, due to the vagaries of release timing (it missed the holiday season), current marketplace (there’s already another game like it), publisher whims (“We don’t think this is fun”), etc. So yeah, even good games may never see the light of day. . . (And it’s a serious kick to the head for developers when they’ve slaved over a game for months or years, and it ends up in the trash bin.)
Noumenon said,
I just read this exact thing this week here.
You know the real problem with frameworks? They demo too well. Someone shows you their favourite framework and demonstrates how you can build 50% of your application in half an hour! Great! That other 50% can’t be hard, can it? But it turns out that what looked like 50% is actually 5%, and filling in the other 95% gets exponentially more difficult as you approach the 100% mark. Frameworks are great for building toys, and that fools us — again and again — into assuming they’re good for building products.
Maklak said,
I’ve noticed this problem for much smaller projects too. I can get something that kinda works in a few days, then tweak, fix and expand it for a month. In the process I get frustrated because it just doesn’t feel like I’m making any progress anymore, and that’s if I don’t have to bump my head against my earlier architectural decisions. Sometimes I add a bit that expands the functionality and it feels, like I’ve made more progress in hours than in the past week.
So well, I’m easily fooled by this, but I still believe that getting the core functionality to work, then use it to test what I add over time is the right thing to do. If it compiles and I can see the results of what I just did on the screen, I can fix a lot of bugs on the fly.
CdrJameson said,
This is a problem I seem to run into many times when trying to use game engines (eg. RPGMaker, Gamemaker Studio).
There are some things you can do really easily and quickly but then you get something complicated or buggy or just not the way the engine does it and development just… stops. And there’s often no easy way forward.
That’s not to mention their often poor support for code reuse, refactoring, static analysis, debugging, source control and after some time just finding the area of the game where a thing is. You just seem to reach a certain complexity level and the whole thing collapses.
Robyrt said,
So true! My (non-games) software company recently started using Balsamiq, a prototyping tool which produces faux hand-drawn UI elements instead of slick and sophisticated prototypes, for this very reason. People – especially the business guys who had to sell the nascent products – were prone to forgetting that what looks like 50% is really 5%.