Frayed Knights: Testing Flashbacks

Posted by Rampant Coyote on July 3, 2012

DGM recently sent me some semi-amusing excerpts from emails we exchanged during the very, very lengthy testing phase of Frayed Knights: The Skull of S’makh-Daon. I don’t know if they would be that amusing to anybody not living through this ordeal, but as I was eating and breathing Frayed Knights, and so were a handful of testers, we found stupid stuff pretty hilarious at times.

I’d originally anticipated around two months for full-on alpha testing, and three to four months for beta testing. As proved true in nearly every aspect of development for Frayed Knights, I was off by a factor of about two.  So what I expected to be six months or less took nearly a year.

“The augmented ceiling drop trap on the first basement floor didn’t cause any injury. Instead, it put half the party to sleep. I GUESS you could argue that being hit on the head by falling masonry would knock them out, but you’d think it would also take off some HP in the process. :)” – DGM

Now, my own view of what testing is supposed to be is that “Alpha” is supposed to be “feature complete” – pretty much all the code ‘n stuff is in place, so the game is fully playable in sections, but not all the content is done. There may be stand-in art, and some areas may not be complete. The game might not be able to be played start-to-finish. “Beta,” on the other hand, should be generally content-complete: there may still be some purely cosmetic content issues (draft versions of music, incomplete particle effects on some spells, etc.), but otherwise it’s a complete (and theoretically completable) game. But… with bugs. Sometimes those extra-special bugs that only appear on one person’s computer once in a blue moon. Those are real fun.

“Ok, I just paid for 4 spell stones from the merchant and when I pushed “ok” to pay for them, they magically transformed into 4 hunting knives???? So I double checked his inventory and all of his stones are gone now. HE ROBBED ME, that dirty bum :P” – Brian

I’d be lying if I said I didn’t do some serious development during testing. In fact, we added two completely new dungeons during early beta as we discovered the game was jumping a little too quickly to high-level content near the end-game. Lots of new treasure, equipment, and even new spells became available during the beta as we discovered “holes” and balance issues in the game. Some feat specializations weren’t getting adequate attention in-game to keep them useful. The game offers so many variations and options to the players that it’s surprising to me that it didn’t go more out of control, and cause more balance issues than it did. But in some cases, we needed to not only ‘nerf’ a particular combo so it wasn’t quite so powerful, but also provide some combat encounters that either directly countered that strategy (encouraging the player to be more than a “one trick pony”), but in some cases that used that same strategy against the player.

“I’m going to go back and make Chloe a throwing specialist after all. In fact, I’m going to see if I can combine that with the dual-wielding bug and make something like this: http://www.nuklearpower.com/2009/01/27/episode-1087-exalted-feat/“ – DGM

Going overboard on the spells was actually intentional – part of the joke – but in addition to making the UI more complicated than I’d like, they were also a nightmare to balance. Frankly, there simply wasn’t enough time or people to help test them all completely. I used a bunch of spreadsheets and calculations to come up with theoretical balance, but I’m still not 100% positive there are not some overly powerful spells (or spell combos) left in the game. And feats – I may have overdone it on the feats as well. I was very focused on trying to make characters very customizable during the game … perhaps overcompensating for the fact that the characters were pre-generated for the player (essential to the design, but still). Same problem.

“Spell – Shred the Dead’s descriptions says it does massive damage to all undead, but it only targets one. That is probably a good thing since that spell really does knock the unliving hell out of the dead…  I just entered the tomb and ran my little rat butt out of there as fast as I could. Those tomb skeletons are no joke. I defeated the first four, but think I should complete the tower first 🙂 Remember when I said that I could mow down anything in my way…..well forget that. I ain’t touching that tomb till I’m more “beefy”” – Brian

Another issue I hope I’ve learned from was due to how much of the game’s content had to be hand-coded. While slightly time-consuming on its own, the big problem came from human error that inevitably resulted from it. Doors, treasure chests, encounter IDs, quest items, all that – these should have been prime candidates for automating aspects of development. But when the game was small, hand-jamming these items into scripts and tables didn’t seem like a big deal. When the game got larger, the problem grew larger, but it was still a case of where (apparently) much of the game seemed to have been “done” already and the remaining tasks didn’t seem quite large enough to justify the effort to go into automation. Then, as hard-to-find bugs started coming to light, I learned a bit more about the true cost I’d pay for doing so much by hand. The results were… often confusing.

“Found a booby-trapped chest near where I hear the goblins talking, but there were no components so I couldn’t do anything.  I quit out of the disarm thing and tried again only to have it immediately blow up and be faced by 3 imps.” – Curtis

“FINALLY figured out a bug that’s been vexing me for a while. When you open the door that triggers the fight with 2 elementals and an imp, you get teleported back down 1 floor. I knew even I couldn’t be THAT bad at navigating. Even if I did end up in the wrong state once without trying to leave town. “ – DGM

So taking all this as a learning process, what sort of things did I learn from the giant testing phase for Frayed Knights 1, that I might apply to future titles?

#1- Simplify where possible. Yes, I resent this one, as I’m the kind of guy who likes gazillions of options. But the fact is that the fewer variables there are, the less chances there are of things going catastrophically wrong or just badly out-of-whack.

#2 – Automated Testing. This can be hard to do in games, but there were plenty of places which, in hindsight, I relied too heavily on spreadsheets, theory, and overworked testers to tell me the story where I could have had the game ‘exercise itself’ – in mock combats or whatever – and given me detailed results that would have exposed problems early and with greater clarity. At least at the end I built some tools to help me “error check” tables of data for obvious problems, and that was a good step. But a lot more could be done here.

#3 – Automated Tools. The ‘human error’ component of problems in a game the size of Frayed Knights: The Skull of S’makh-Daon is very significant, especially when the changes & fixes themselves can introduce new problems.  The answer is building tools that offload the purely mechanical aspects of game-building, eliminating human error from routine tasks and letting the designer (that’s me) focus on the important stuff: Like what SHOULD be triggered by the door opening, rather than the nuts and bolts of hooking up the triggered event. This would increase not only development speed, but also save oodles of time in the testing phases.

#4 – Better testing process & tester tools. I mixed too many different kinds of testing together into one batch during alpha and beta. I tended to drop testers into the game “cold” and then listen to their feedback. This is a very valuable kind of testing, as much of my concern was for the first-time player experience and making sure instructions, quests, and controls were presented clearly, and that players could figure their way through the game without help. Whenever they got stumped, I’d make a note to add in additional clues to help future players along. However, that’s only one kind of testing.  Maybe that’s a good way to use first-time testers, but after their initial “taste” I needed to give testers better instructions (and tools) to test specific aspects of the game – whether it was feat balance, particular dungeons, or whatever. While they had some tools for customizing their party for testing, I could have done better there, too. In fact, having automated feedback from the game sending information on the tester’s progress, issues, or whatever would be a useful tool for them and for me.

#5 – More playable earlier. I actually felt I did pretty good with this, as Frayed Knights was pretty playable from the get-go, thanks to my work on the pilot early on. But the alpha state was still pretty rough when external testers first started getting their hands on it.  It still hadn’t quite come together as a game yet.  I’m not sure what I could have done on this front to make it better, but I think more regular time dev sessions devoted to just playing through some piece of the game – with tools to compensate for the missing pieces (quest object dispensers, that kind of thing) might be a good addition.

#6 – Better-defined Framework: This goes hand-in-hand with #1, simplification.  A lot of the rules and elements in Frayed Knights were designed pretty ad-hoc. I may have coded up their architecture in such a way that I committed design-overkill, with careful forethought and planning, but from a game design perspective, I was probably a little too free-form and flexible. There weren’t enough governing principles — enough of a “skeleton” of rules that everything else should have been built from. As a result, a lot of the really cool parts of the game lost their distinctiveness. I retroactively made some changes later – such as with the four kinds of spells – to try and improve this, but the truth is the spell categories do blend a little too much. There are no guiding principles with trap components as to which components should be (generally) more challenging, more dangerous, or require more successes to disable. And so forth. A more rigid design framework – broad underlying rules and principles governing the mechanics – would probably help simplify testing (and provide additional opportunities for automated testing) a great deal. For example, you can actually perform balance-testing on the underlying rule – and it’s ‘limits’ (maximums and minimums) – and then test to make sure that all elements covered by that rule conform properly. That’s a lot easier than testing each element individually.

So the big question – am I actually using these ideas in Frayed Knights 2? Of course!  Maybe not as well as I should, but a lot of what I’ve been doing thus far is laying down the basic frameworks and tools to do exactly these thing, hopefully making both development and testing a lot easier and faster in the long run.

 

Filed Under: Frayed Knights - Comments: 9 Comments to Read



top