The Craft of Game Design Cannot Be Measured By Any Metric
Wednesday, 20 July 2016
If game design is a craft, what becomes of it when game development is driven solely by financial metrics? Does any of the craft remain? Or are games reduced to mere commercial pit traps, luring in and monetising their unwitting victims?
A little over a decade ago, when my friend and colleague Richard Boon and I were writing 21st Century Game Design, I had predicted that this century in games was going to be characterised by a new focus upon understanding players, and that this would be attained by various models of player behaviour. I suggest (with the benefit of hindsight) that this general claim was correct, and that we have gone from an era where game design was dominated by dogmatic assumptions and self-satisfying design practices (although neither of these have gone away…) to one where understanding how players relate to games is an inescapable part of the videogame industry.
But we made one crucial error in that book. My assumption had been that modelling player behaviour entailed understanding how to satisfy play needs, which is to say, having a positive, inclusive, moral and practical relationship with players. But the dominant forms of player modelling right now have absolutely no need to understand how to satisfy players in any form, because the principal form of model we are using right now is analytic metrics – and these metrics are blind to any aspect of the mental states of the player whatsoever. If our image of game design in the 21st century was that the industry was going to be making money by creating games that deeply satisfied their players, what we are actually facing now is an industry that makes the majority of its money by simply analysing where the leaks are in their cashflow, and acting as digital predators to suck spare change out of players’ digital wallets.
It may be helpful to look at the key metrics at use today to verify what I’m claiming. Firstly, there are the measures of activity – Daily Active Users (DAUs), Sessions, Stickiness (DAU/MAU), Retention and its inverse, Churn. Then, the measures of monetisation – Conversion Rate (percentage of players making purchases), ARPDAU and ARPPU (Average Revenue Per Daily Active User, or Per Paying User). Also, game economy measures for Sources, Sinks, and the Flow Rate of in-game currencies, all geared towards engineering sufficient sparseness that players will be encouraged to pay money for advantages. And that’s what it’s all about: squeezing money out of players' impulses – although in analytics, there are no players, only users, just like the narcotics industry. As the company Game Analytics observe with the admirable unvarnished honesty that belongs to these thoroughly pragmatic commercial practices:
Successful free-to-play games create long-term relationships with users. Users that enjoy the experience enough are willing to pay to for a competitive advantage. A game needs to have strong retention to have time to build this relationship. (Emphasis added.)
One of the most coherent supporters of the free-to-play business model where such metrics dominate is Nicholas Lovell, author of The Curve and regular on the same speaking circuit as me. We first met at Develop Liverpool, many years back, and our paths still occasionally cross. He views the challenges of that side of the market as not so much about monetisation (he rankles at being called a ‘monetisation consultant’) as about retention, in accordance with the quote above. But I read very little from him about the craft of game design, and his recent talks have tended to be framed in terms of the keywords ‘Acquire, Retain, Monetise’, which sounds like a scaled down version of the Ferengi Rules of Acquisition. Nicholas continually insists our industry can self regulate itself away from abusive practices – but I still don’t see any sign of this, nor indeed do I detect much interest in doing so.
The focus on metrics over game design has brought the videogame industry closer to its less reputable but more profitable cousin ‘gaming’ – what's commonly known as gambling – and with it, we have a host of ethical questions about what we are doing, none of which can be merely presupposed. We urgently need a debate on monetisation practices to establish what ethical metrics consist of, but the industry does not want to have this talk. I offered a dynamite panel to GDC this year on this topic, but it was knocked out of contention instantly. The industry is afraid to have the conversation, but until we are ready to address questions about what metrics mean for game design as a craft, we have a serious unaddressed problem that affects the integrity of the games industry. Of course, in purely capitalistic terms there is no integrity, there is only money. But money is just another of our imaginary games – it just happens to be one that we all take very, very seriously, since we have lost our ability to feed ourselves without it.
One game designer who has taken a stand on the ethics of monetisation is former Free Realms creative lead Laralyn McWilliams, who quit a job out of disgust over the issues I’m highlighting here. In an interview back in 2014 entitled “The problem with ‘best practices’ in free-to-play”, Laralyn reports how designing for ‘friction’, which is to say, monetising player frustration, finally became something she couldn’t endorse:
…a designer came to me and said there was a spot where it got really rough; there weren't enough quests, and the grind was really terrible. He wanted to add five or ten quests to make it feel better…. But when I looked at our numbers that was the spot where we had our best monetisation. The awful feeling of that grind was getting people to spend money, so I had to say no to something that would make players happy because it would cut our revenue. At that point I said, ‘Nope,’ and I got out of social games.
Against the ruthless focus on the bottom line is the possibility, if nothing else, that game design can fulfil its calling as a craft, and that informed practitioners of that craft can satisfy the play needs of many different kinds of players. This does happen, even in the battleground of metrics, and developers that are willing to commit to doing so can build a loyal fanbase that supports them, and helps other players to find them. It’s a harder path, to be sure, because it means making commercial artworks that are worthwhile instead of just cranking the sausage machine of rehashed ideas. Nothing good comes without effort. But if we want to walk this path, it entails more than simply resisting the purely metrics-driven concept of commercial games.
Sadly, indie developers who have avoided going down the predatory monetisation path have tended to simply default to making what they like to play and then gambling upon finding an audience for it, which I view as a hugely risky way to pursue a career in game design. I’ve seen dozens (perhaps now hundreds) of developers fail doing this... it’s simply not a good enough plan to trust that – by chance – your play needs will align with enough players to magically make ends meet. As Rami Ismail of Vlambeer suggested to me when I accused him of giving this exact advice:
...I've told developers to make what they want to make - [but] never in that vacuum. My entire existence as a public figure exists because I was one of the very few prolific 2010-generation indies that was yelling about taking business seriously, engaging with publishers and marketing, and doing the work to make your game visible.
21st Century Game Design will be going out-of-print soon; its multinational publisher has withdrawn from publishing books about making games entirely, which in itself says something. Our first book’s core vision – that there are methods for game design, but there is no single, perfect method for game design – remains as true today as it ever did. Our deployment of that vision through a fusion of horizons between psychology of play and the history of videogames remains, I believe, an extremely fruitful way of understanding the craft of game design. Alas, the games industry didn’t choose this path. It choose instead an unholy schism between dogmatic indie design on the one hand, and pragmatic monetisation design on the other. Personally, I feel that the artworks we call games deserve more than this, but I appear to be in the minority. In a games industry divided between a stubborn individuality unable to reliably feed itself, and investment-glutted money farms, there seems little room left for cultivating the craft of satisfying players.
Agree? Disagree? I’d love to hear your comments! Have a blog? Any and all replies at other blogs will be promoted here to keep the conversation going – just let me know the link in the comments or on Twitter.
Incredible post Chris. I find this depressingly accurate in all ways.
I try and support the stubborn individuality types that do makes games that support my needs, and thankfully there's still a few AAA titles that do and haven't completely become 'money farms', even if they all show signs of moving in that direction of course.
Posted by: Rik Newman | Wednesday, 20 July 2016 at 16:14
Always a pleasure to hear from you! 'Depressingly accurate' seems to be a common response to this one... There has always been a tension between commerce and art, of course, but the current situation is particularly disturbing.
I am heartened by Overkill buying out their publisher and cancelling out their microtransactions, though - that is something we've never seen before. At this point, we could really use a good name for the fixed-price business model - something that can become a selling point for those games vouching to sell in that way. We need to start finding positive ways to describe the best commercial practices.
Posted by: Chris | Thursday, 21 July 2016 at 13:46
I guess I don't fully parse this argument or the response to Brian's followup.
Does this sentence summarize the key point "It choose instead an unholy schism between dogmatic indie design on the one hand, and pragmatic monetisation design on the other"? That there's two groups, one ignorant and the other overly focused on monetization?
Two groups! So far apart! That's less a fundamental tension than just a set of categories of developers. There are ignorant devs trying to become rich. There are rich devs that are happy to just make something and release it. Pick two variables and there's devs you can plot in 4 quadrants. ;-) But the truth is pretty multidimensional with people in almost every niche.
Big. The universe of game developers is now big. And wonderfully varied. It isn't that absolute or categorical statements are false. There's someone somewhere that fits each claim. For good or for evil.
But there's also a lot that's possible now that wasn't when we had more of an easily described monoculture. The variety alone is cause for optimism.
What's your particular desired utopia? There's probably at least one or two devs out there within spitting distance. Support them. Because denouncing the rest is a bit like the man on the Russian steppes who is irritated that ladies in Brazil don't dress in proper furs.
Posted by: Danctheduck | Tuesday, 26 July 2016 at 01:37
Always a pleasure to flush you out with a firestarter! We only ever talk these days as a result of such things - I must piss you off to get you to speak. :p
Right, let me accept everything you say in this comment as an artefact of the firestarter format which must simplify in a strident fashion in order to provoke opposing viewpoints to speak out. That might allow us to get to the interesting part of the discussion sooner.
The challenge I am offering the 'pro-metrics' camp - which you belong to - is either to explain what non-exploitative metrics look like, or to deny that any use of metrics are exploitative. That's the first challenge: show me the boundary of exploitation, and tell me how you know you haven't crossed it.
Furthermore, I want to know, and genuinely desire to uncover, how metrics can support the craft of game design, understood as attempting to satisfy players. I believe it might be possible. But no-one talks about this, which makes me doubt. Almost all talk of metrics begins with maximising effective monetization. And since such metrics are blind to the mental state of players, ex hypothesi, such use of metrics is contra to game design understood as a craft.
This is another option: to deny this specific claim of an irrecoverable tension between metrics and game design. But I suspect any attempt to deny this must tackle the first challenge, above.
You'll probably see a third path - you usually do. :) But I want to know how you see this either way.
You are not part of the dogmatic indie scene; you are fully situated in the commercial realities of game development at this time, and you use metrics to do it. But you have not abandoned the craft of game design - indeed, you are very much engaged in it. So how do you cross the disconnect? Do you view the monetisation as neutral to the game design? Surely not, because it clearly affects the designs. So you have a concept of non-exploitative metrics for game design: what is it?
Hope that is clear! And further hope that you will be able to extend this discussion.
All the best,
Posted by: Chris | Tuesday, 26 July 2016 at 12:05
PS: By the way, Dan, next week at DiGRA/FDG (the most prestigious of game studies conferences) I'm presenting a paper that completely grows out of your pushback on 'Is Fiction a Wrapper for Games?' back in 2013. You get a whole slide where you just mouth off at my half-cocked remarks. :p Just a reminder that your perspective is particularly important and valuable to me!
Posted by: Chris | Tuesday, 26 July 2016 at 12:08
Ha! Sorry that I tend to only reply to the inflammatory posts! Will try to correct in the future.
It is a bit tricky to know where to start. How about the role of empiricism in game design?
When I design I have a mental model of how I imagine my game will be played by players. This includes predictions about player emotions, learning, buying behaviors and a dozen other factors necessary to make a self-sustaining game in one of today’s various markets. I also make predictions about how markets will act. Platform desires, player designers, press desires.
Then we build the game, or at least we build an initial version of it.
Then we playtest the game to see if the my predictions worked out. Most of the time they don’t. In the best cases I’m only off by a factor or two. In the worse cases I’m off by several orders of magnitude. However, I may also find that players behaved in a manner that was actually more interesting than I predicted.
So we build another iteration of the game. Somehow, we need to connect the empirical reality of what the playtest suggests with what we predict will happen. This usually involves updating our models, sometimes radically. Often incrementally.
For some designers, this process can be frustrating. The reality of player behavior imposes constraints on their mostly imaginary vision. But I tend to see constraints as necessary to the process of design. And constraints based off observing real people playing the game tends to more often than not yield opportunities to impact the real *shared world of many people* vs the isolated imaginary world of a single person. We find new ways of playing that are more vibrant and interesting.
How are metrics useful when iterated on a game?
Game designers are information starved. With writing, we have an imperfect but competent mechanism for imagining how someone might feel reading a bit of text. In order to write, you must read. And thus you are forced to process a work in a somewhat similar fashion to how a potential reader might process. Game developers do not have this luxury. We build systems multiple times removed from a player’s experience. Write some code. Do a dozen other steps. Build an executable that someone somewhere runs. Knowing how people with react to what we make is hard.
So we use crutches. We create complex models of how players think. We use ‘proven’ patterns. We watch players and try to imagine what they are feeling. Then we try to backtrack all far removed information to whether or not a number in the bowels of a broken machine should be 2 or 4.
There are certainly classes of information we can extract more easily. Surface player emotions on individual playthroughs. Awesome. We can do that. But human behavior is broad. We see the need to sample behaviors across populations and discover central tendencies or outliers.
So metrics or analytics are that tool. They let us understand statistical patterns of behavior. Do they let us see inside the minds of our players? No. Nothing does yet. Do they replace in person playtests? No. Smart designers use multiple sources of insight.
But metrics do provide an amazing range of insight by allowing us to look at hard problems from a different direction. If players in an MMO are flooding forums with complaints about a change, how many people are impacted? How did playstyles change?
When balancing economies and progression systems, metrics are essential. You can’t do an in-person playtest of someone playing a game for 90 days. The old tools don’t work. And various forms of data collection do.
Maybe all this doesn’t need to be said. Maybe you are worried about something else entirely.
Are you worried about how metrics shines a light on bullshit design? Because a lot of design is unsubstantiated bullshit. We imagine people will play a game a certain way and then they don’t. Such an ego buster. Metrics beat us with bully numbers. They bluntly state our initial idea was flawed. Or even worse, the thing that people have been praising us for years doesn’t actually apply to anyone but some weird elite group of outliers that happens to give out chintzy feel good awards. Reality can be cruel when you live in a fantasy. But it also acts as a constraint that forces us to up our game and make something that works. Versus wandering blindly off a cliff in a feel good haze. Which I’ve done. (Lovely until you fall).
Are you worried that Bad Men use metrics in a reductive fashion to emphasize making money over art? Bad Men have been emphasizing making money over art for a very long time. For any golden era of games there were penny pinchers micromanaging creative decisions at a level that destroyed souls. Might I suggest that a new tool for getting data is not the actual problem. The team sets their goals. The tools just get them there.
Are you worried that we are using Dumb Metrics? That the dumb patterns dumbly followed by dumb practitioners result in dumb ideas and dumb games? Well it is true. And the solution is one that applies to all complex instruments used in the pursuit of art and beauty: Get Good.
I actually see metrics, competent design and building something positive that meets player needs as three complementary pursuits. I’ve asked “Well, what do players want and how does that align with business? And how does that align with art or craft?”
Here’s one answer. Many players want connection with meaning and community. They want mastery and agency. This leads to them enjoying an activity for a long period of time. That results in great retention metrics. And when deep needs are being met, people are willing to spend. Will I spend a buck on Pokemon lures to enhance a relaxing afternoon with my wife at the coffee shop? Yes. It makes for joyful light conversation. The game improves our relationship by creating a shared playful space.
Metrics track and tune all this. Is that evil? Just the opposite. I consider it doing great good for the world through competent design practices.
Posted by: Danctheduck | Thursday, 28 July 2016 at 00:56
Thanks Dan! This is almost everything I wanted. I'm going to repost this at the top level so it can be more widely read.
Posted by: Chris | Thursday, 28 July 2016 at 10:50
Metrics is just a tool, like a hammer. It can be used to create masterpieces of design or it can be used to beat money out of people. It depends on who is using the metrics and what they want out of their use.
Of course there are big money grubbing corporate types who only care about where the monetization comes in but equally there are people using metrics to identify and refine particular design issues with their games.
Metrics can tell you if most players find your levels too long or if there is a particular weapon or character that is unbalanced. It can tell you that very few people engaged in your latest event or the only played the special event level once and never again. Of course on its own this information is limited but it can be used in conjunction with design reviews and playtesting and other forms of feedback to quickly and accurately identify issues that need focusing on.
The main problem with metrics and the one that seems to drive blog posts like this is that there have been some highly publicised "success" stories where massive companies like Supercell or Zynga increased revenue by huge amounts by using metrics in a certain way. These sorts of stories have encouraged the much vilified sheep culture game designers down the same route of chasing monetization and the quick buck and so that's the only thing we ever hear about as a design community.
But metrics is much much more and the sooner everyone opens their eyes to see that and start using them (with other tools!) to create design masterpieces instead of glorified slot machines, the better the whole industry will be.
Posted by: Ben | Friday, 29 July 2016 at 14:30
I am wary these days of saying that something is 'just a tool'. Every tool is different. The hammer, that you mention, can be a weapon or a means of construction. A landmine is only a weapon. A doorstop is pretty harmless in any context. Every tool is different, it has a different effect on moral agency. My accusation here is that metrics as practiced in the games industry invite design away from satisfying player needs.
The real pushback against this, which Dan has provided brilliantly, is to show how metrics can be applied in support of the craft of game design. Getting to this was part of my motive in writing this piece - as the vast volume of discussion of metrics concerns monetisation, and there is very little talk of using metrics to support game design.
If you have knowledge of using metrics to support game design craft (understood as satisfying play needs), then I want to know what you do, how you do it, and how you ensure that you don't slip into exploitative territory. By all means share this knowledge if you have it - it's not enough to say "well some bad folks use metrics badly"... defend your craft! Let us know how to use metrics well.
Thanks again for getting involved,
Posted by: Chris | Saturday, 30 July 2016 at 10:25
I already provided a few snippets of how metrics can be used to aid the craft of game design without focusing solely on monetization. It is an important tool for insight into how a broad spectrum of people play games, rather than the small number of people used for testing.
Perhaps the hammer analogy is going down the wrong path. Metrics is more like a calculator or ruler - it's a tool for measuring things. What you choose to measure and how you use your results is up to you but just because many people use them in a certain way, it's not to say the tool can only be used to that end.
You keep coming back to the line of "the way metrics is used in the games industry" and this is precisely my point. Metrics is not the problem- the problem is the way it is used.
In my work I look at all sorts of non-monetization focuses metrics to improve certain areas of games. Engagement is an important one - are people equipping powerful accessories to their characters? If so which attributes do they focus on? How long do they spend on character creation? Do they mostly just blast through and ignore the finer details? If it's a Facebook game, what age, gender, nationality are they? This can help suggest certain themes that you might want to include in future updates or events.
People like to consider games design an art, which is one of the reasons they don't believe it can be quantified but it is not the art itself being measured; rather it is the reaction to it, how long is spent looking at it and how people interact with it.
Posted by: Ben | Saturday, 30 July 2016 at 10:50
One of the interesting effects of posting this piece is that it has encouraged metrics-wielding developers like yourself to present their own use of metrics, and thus to widen what is normally a very narrow view on these practices.
I'm open to the idea of 'virtuous metrics', but those developers taking automated measures for these purposes ought to make more of a fuss about it. I think the prevailing attitude is "those greedy people are doing such-and-such, and that's not us..." - but that only feeds into the prevailing negative impression by hiding some of the ways metrics are being used. We need more folks like yourself talking about how they use their metrics - and how they don't use their metrics too.
For me, I'm afraid most metrics-driven design still feels like taking a slightly too convenient shortcut, but then I'm sceptical of all aggregation techniques for dealing with human issues. The vast majority of psychology research has the same problem. But, despite the bile I summoned when I wrote this piece (necessary if it was to start conversations...), I don't actually oppose metrics - even monetisation metrics - I just want us to have better conversations about them, and recognise that we are changing our games because of our available tools, which is a more complex situation that it first appears.
Many thanks for continuing the conversation!
Posted by: Chris | Sunday, 31 July 2016 at 07:54
Totally agree. There needs to be a lot more conversation and education of the many various ways metrics can be measured and used. It's important to keep in mind though that analytics of any form in the games industry is in its infancy and developers and publishers are still learning what it is and what it can do.
An unfortunate side effect of the rapid growth of games metrics is that the shortage of qualified people experienced in games fields means many analysts are employed from other sectors, particularly "business intelligence" departments of financial services and all these people tend to think about is the money side of things.
It's equally important to realise that "metrics driven design" is actually a very rare thing and even more rarely is successful. Very few developers have dedicated analysts or can afford the costs of consultants. The issues you highlight are somewhat blown out of proportion by a few highly publicised successes of a handful of the biggest companies who like to show off but ultimately when you look deeper at their "success stories" they don't actually seem to know why what they did worked and are usually unable to reproduce it in future projects. Just look at supercell with their many many failed titles that don't even get through soft launch.
Posted by: Ben | Sunday, 31 July 2016 at 10:27
"when you look deeper at their 'success stories' they don't actually seem to know why what they did worked and are usually unable to reproduce it in future projects."
This made me chuckle, as this is broadly true for the entire history of the games industry! :) The only companies that have sustained success have created games that naturally imply further iterations.
I spoke in this piece about developers failing, but another story could be told of how success has brought down companies who were never able to make another successful game. I rather suspect that story goes far beyond metrics!
Many thanks for getting involved in the discussion! Much appreciated.
Posted by: Chris | Monday, 01 August 2016 at 07:37
Late addition to the comments here even though I read this when it was posted, 'twas shared in my newsletter too. Just wondered if you have watched Jonathan Blow's presentation on F2P design which has some crossover with what you're discussing here?
Posted by: Harbour Master | Thursday, 25 August 2016 at 08:59
Hi Harbour Master,
Thanks for the link, although I almost never watch internet videos. It's just not my thing. But I've looked at Blow's comments indirectly, and we're in broad agreement on some of the issues here.
Posted by: Chris | Tuesday, 30 August 2016 at 08:09