Over the years, I’ve been involved in dozens of game audits – the process whereby the content of the game (either on paper, or as a build) is assessed externally from the developer in order to help improve the project. I’ve been on both the kicking and the receiving team for audits – as the consultant brought in to provide feedback, and as part of the team getting that feedback. Game audits can be exceptionally useful. But they can also be a colossal waste of time and money. I would recommend that any team considers auditing their design or narrative materials – but I would also advise taking the feedback with a pinch of salt.
In this two-part series, I’ll offer some tips and warnings about auditing a game that may help developers going down this path. This week, I’ll look at how to set off in the right direction for a productive game audit – and what will happen if you don’t get it right.
Pitfall: Did They Get Everything They Needed?
The single biggest reason that a game audit fails miserably is when the auditor isn’t given everything they need to understand the game. Often this happens because the game design documentation is inadequate at capturing the development culture that the game is created by (which is unavoidable). Sometimes it happens because the design isn’t explicit about its reference titles, inviting a string of erroneous assumptions about what the game is supposed to be. Sometimes it happens because of a failure to take into account different target audiences. And sometimes it happens because the auditor is kept in the dark about information they actually need to do their job.
The worst of these I’ve been involved in was a Western-style cRPG project with a design entailing a span of many decades, and a central character passing through different stages of life. An audit was called by the publisher on the story materials – and the auditor was given the Narrative Design but not the Game Design documentation. As a result, they assessed the story materials as if the game was an action game, and provided feedback that was so wildly wrong-headed it was essentially impossible to get a useful direction from it. Fortunately in that case, I had subcontracted for an independent story review with an auditor who was given the design documents to take into account, but what the two audits revealed was the radical gulf between the understanding of the publisher and the understanding of the developer. This project was never completed.
Pro Tip: if you intend to audit design or narrative documentation, make sure the people in charge of those documents know about the audit and can prepare the documents with an external perspective in mind. Most game documentation is intended for the developer’s eyes only, with the decision processes undocumented because the team themselves don’t need a record of this. For audit, those decisions can be crucial, so make sure the paperwork includes context, reference titles and justifications for apparently odd decisions.
Pitfall: Are They the Right Person For the Job?
I’ve noticed that developers and publishers jump at the chance to get a “big name” game designer or writer involved in their project, thinking that commercial success is something that can rub off in the audit process like luck from kissing the Blarney Stone. It is my experience that while “big name” auditors can provide stellar feedback under the right circumstances, no game has enjoyed commercial success as a result of this kind of ‘star audit’.
One of the chief problems in this regard is that while most game designers consider themselves able to design any kind of game, no game designer is actually equipped to do so. If you have never worked on a role-playing game (tabletop or computerised), your shouldn’t be auditing one. If the game is targeting teenage girls, don’t get an audit from an expert in First Person Shooters. It seems obvious when you spell it out, but everyone has their strengths and weaknesses and the worst kind of audit is one that involves fitting a square peg into a round hole.
Although everyone who has built a name for themselves as a game designer or writer has (by definition) successful titles they can point to, what this usually means is that they have worked within a development culture capable of delivering games of that style, genre or for that particular target audience. They may well know what has worked for them – but do they know what doesn’t work for your style of game by virtue of their success on their style of game? The risk of getting in a superstar to audit is that what they really know is how to make games in a particular way – unless you’re certain you’re developing in an equivalent circumstance, their feedback is less likely to help you than critical input from someone specialising in the appropriate areas.
Pro Tip: Audit feedback is most useful when it can help you avoid mistakes, and for this prior experience within a particular field relevant to your project is more important than specific commercial success. The reasons specific titles succeed involve far more than just the game design or narrative materials. Market success is not a transferable skill – but experience of development problems is transferable. Hire auditors with breadth of experience wherever possible, and avoid “big name” audits except when the crossover in design is a near-perfect fit.
Next week: Bad Assumptions