The Grind Mystery: Escalating Reward Schedules
Wednesday, 19 January 2011
Given the tremendous volume of psychological research that has been conducted on reward schedules, why is it that the most basic form used in digital games – that behind the grind in every computer RPG, and many other games beside – does not seem to exist in the literature?
I’ve written before about reward schedules in games (also known as schedules of reinforcement in the psychology literature), and as many will know these are considered for the most part to have four basic kinds: fixed ratio, variable ratio, fixed interval and variable interval. (If you are unfamiliar with these terms, please follow the previous link for an explanation, given in the second set of bullet points). In the past, I’ve suggested that the levelling mechanics from RPGs (now present in a growing majority of blockbuster commercial games) can be understood as a fixed ratio schedule. But this overlooks an important question.
If, as psychologists maintain, these four schedules are the basic effective forms of operant conditioning (that is, training for behaviour), why is it that the RPG levelling structure – which originated with Dungeons & Dragons – doesn’t use a fixed ratio at all, but in fact a ratio that gradually increases over iterations? For instance, a typical D&D experience point curve goes 0, 1000, 3000, 6000, 10000, 15000, 21000 etc. Let’s call this an escalating ratio schedule for convenience, or an escalating reward schedule. If psychologists are correct that the four basic schedules listed above are essentially fundamental, it seems quite anomalous that games (which are the example par excellence of reward schedules in action) should prefer an escalating schedule, and equally queer that the academic paperwork seems (as far as I have been able to ascertain) to completely overlook this form as significant.
What’s more, it’s not that the effectiveness of escalating schedules is unknown outside of game design, since animal trainers (as I discuss in the post on Positive Game Design) use escalating schedules as a matter of course to teach behaviour, giving jackpots for the first success, then rewarding every instance, then rewarding with decreasing frequency, before eventually rewarding at random. In the context of animal training, this is largely seen as starting with a fixed ratio (e.g. reward every time) then changing to variable ratio (i.e. random reinforcement). This fits with the literature, but it doesn’t fit so well with what is common in game design. Either the way escalating schedules function in games is maintained solely for reasons of tradition, and games would do better to use fixed followed by variable schedules (which I doubt, except from a certain perspective I’ll discuss shortly) or the escalating schedules are themselves significant forms of operant conditioning that are overlooked in the literature because the prior paradigm (established by B.F. Skinner et al) encourages psychologists to interpret in terms of its pre-existing ‘boxes’.
From my own experience of both animal training and game design, the escalating reward schedule is the ultimate form of operant conditioning when you want to motivate continued interest. It produces the same “post-reinforcement pause” (e.g. slightly reduced interest after levelling up) of a fixed schedule, hence the tendency to consider them under this banner, but they don’t allow the player to just repeat the same activity, because of the escalating requirement for the next reward (e.g. level). This helps maintain interest, since the player is encouraged to actively think about what they’re doing, which is itself rewarding. (It’s less clear to me why this should matter to my dog, who is less likely to be thinking in these terms, but still learns behaviour far more effectively with an escalating schedule).
It seems as if the merit of the escalating schedule is that in the early stages, the rewards come regularly, which helps the habit set in, while later on the rewards come further apart which gives more time to focus on the nature of the activity rather than having the activity overshadowed by the reward. Hence, in the case of teaching my dog to fetch, the rewards at the outset get him interested in what’s going on, then when the rewards come less frequently the activity itself (running after a ball or frisbee in this example) starts to become its own reward, and the food treats are merely the icing on the cake. In the case of players, the low experience point requirements for levelling at the beginning of a game (or the low target values in the case of Achievements) generate a high awareness of the rewards, then the escalating schedule increases the space between rewards to provide more time to think about how to achieve the goals or enjoy the process of attaining them.
It’s striking that almost all games using escalating schedules use an end point. Hence Dungeons & Dragons, World of Warcraft and almost any other RPG you’d care to mention with an experience system has a maximum level and/or a point at which the next level has a fixed value rather than an escalating value. The escalating schedule becomes counter-productive beyond the point that the player maintains interest in the next reward (e.g. level) – if, for instance, the amount of work required to reach the next reward seems insufficient to motivate proceeding. It is at at this point that the experience of escalating schedules ceases to be addictive fun and becomes grinding. Precisely how this term is used is as a pejorative to refer to conscious awareness of the ratio schedule, and hence the onset of boredom with respect to the process in question.
It’s equally striking that once the end point kicks in, games have supplemental systems based upon variable reward schedules at the back end. In the case of World of Warcraft, for instance, random rare treasure acquisition (“rare drops”) kick in near the maximum levels to maintain interest – which is exactly what animal trainers advocate when they talk about switching to rewarding randomly once the desired behaviour is established. This isn’t surprising since variable schedules are the most addictive, as any manufacturer of slot machines will tell you. The literature states that variable ratio schedules produce both the highest rate of responding and the greatest resistance to extinction, which is psychologist speak for how to make a slot jockey, or, for that matter, a WoW addict.
So despite the lack of discussion in the psychology literature concerning escalating schedules, the general way reward schedules are used in games does fit with the academic perspective on them: fixed ratio schedules to generate a high level of activity to begin with (to establish the habit) followed by variable ratio schedules to power up the degree of addiction. All that’s missing from the papers discussing this is the importance of the escalating ratio schedule (as opposed to the strictly fixed schedule), but from the point of view of game design this is a highly significant omission, since this pattern is utterly de rigueur, and not (I would suggest) for purely traditional reasons, since game designers would rather reinvent the wheel than borrow an existing wheel any day of the week. Game design patterns that persist do so primarily because they are successful in some manner pertinent to the games they are used in, and only rarely because of a culture of momentum.
The grind mystery, then, is why the escalating reward structures that produce the experience of grinding (which, despite the pejorative, are the basis of the core fun for an ever increasing number of games) are not considered in the psychological literature at all, given their effectiveness in games (and, I am positing, their greater effectiveness when compared to fixed schedules). Does this reflect an oversight on the part of psychological researchers? Or perhaps, as I have suggested here, it is because the gamer experience involves thinking about what to do and how to do it, and this parallel source of reward (i.e. fun) can be strangled out of existence by a game that doesn’t provide enough slack to allow for it. That’s the grind. And that’s why gamers complain about it, even though they continue to buy those games whose designs depend upon it for their success.
The XP curve is only part of the puzzle here. The other part is the XP rewards. Your advancement table is the same as D&D 3rd edition. But, looking at the rewards for killing a monster, those increase as level increases as well. Now, admittedly the increase in XP reward is linear compared whereas the XP requirements are ever increasing; however, note that fighting a monster 1 challenge rating higher gives 50% more XP, 2 higher gives you 100% more XP. At 15th level, fighting a creature with challenge rating 2 higher (CR 17) is not going to be nearly as challenging as doing so when at 5th level. Thus, there's some encouragement built into the system to fight bigger and bigger things. In a paper game, this also ignores the fact that the DM might give out additional XP for things like role-playing and non-combat encounters.
In a computer game, at least in modern MMOs, the increasing XP requirement is usually offset by a matched increase in XP from rewards. The goal is to get players to focus on level-appropriate challenges rather than scumming low-level stuff (and usually disrupting less experienced players). It may take 10x the XP to go from level 25 to 26, but the quests and monster xp is roughly 10x the value as well.
So, while I think your theory is interesting, I find it rarely works out that way in practice. Assuming a constant amount of effort, this almost approaches a fixed ratio system within some variation. A game that truly required an exponential amount of effort would eventually find many players getting frustrated and leaving when achieving a level became beyond their endurance.
Posted by: Psychochild | Thursday, 20 January 2011 at 19:11
Could it just be that the more advanced in the game a player is, the more he/she will tolerate grinding sections (so the longer they can last)? Because of all that's been achieved before, it can't have been for nothing...on the other hand, if you have to be grinding from the start, you may think "what's the point?"
So if is there's going to be a certain amount of grinding in a game, it makes sense to spread it so that those sections last increasingly longer as the game progresses.
Then the question remains, why to you need grinding in a game at all?
Maybe this escalating ratio schedule in game design - and the grinding that is often associated with it, comes traditionaly from the need in video games to make levels (or whatever progression system is in place) last longer, and artificially prolong the total playing time of the game through:
1- having to repeat the same difficult task over and over (because failing or dying), resulting in frustration (think of old platform games, vertical scrolling shoot them ups...)
2- having to repeat very similar and easy tasks (like collecting a certain number of items) - resulting in boredom.
I know this doesn't translate well into modern games, what is your opinion?
Posted by: Roman Age | Friday, 21 January 2011 at 01:23
Psychochild you beat met to the post.
The amount of the XP reward is indeed an important factor related to the amount of XP needed to level. With progressively higher levels, grinding low level mobs is a waste of time for a high level player. It's also an interesting risk/reward balance for the low level player who may want to take on a high level monster for a big boost.
A lot of what makes grinding boring is that it's not challenging - meaning you're way inside your comfort level.
Posted by: Tim Holt | Saturday, 22 January 2011 at 18:36
Psychochild (Brian): I'm not denying the role that being ahead or behind the curve has on the player - it is the case, as you suggest, that the curve allows the player to progress more rapidly by tackling greater challenge. But not every player does this, of course. Some simply grind the easy areas to get ahead of the curve, so they can tackle the game more easily. This ability to adjust the difficulty to the player's own preference is one of the great benefits of these kind of reward schedules for the player of computer RPGs. This point doesn't translate so well into the achievement schedules of other modern games, though, and these too use escalating forms.
Furthermore, this piece is about why the escalating structure doesn't appear in the literature, even though it is the basic form of positive animal training - and no-one could claim that the ability to adjust the difficulty mentioned above is a factor in this case. So I think the issue I raise here is an issue in its own right.
(Incidentally, I'm not sure what role fighting tougher monsters has in tabletop, since the GM sets the encounters here; it's rarely a case of the players choosing to go to such-and-such a place to earn more XP in D&D in my experience).
Your closing remarks are how I rationalised this in the past (see earlier post linked in this one): if you remain on the curve, it's approximately a fixed ratio. But this is a post-hoc justification, since in practice it is never a fixed ratio, especially in computer RPGs. The early awards pay out quickly and often. The later rewards get further apart. Players become aware of the grind in the latter stages, as the gap gets bigger. And that's what I'm alluding to here.
"A game that truly required an exponential amount of effort would eventually find many players getting frustrated and leaving when achieving a level became beyond their endurance."
Yes, and this is, in fact, what happens in these cases when the content runs out before the levels do (Armada on the Dreamcast had this problem in spades).
The modern solution to the problem (cf. WoW) seems to be to simply curtail the higher levels entirely and switch to using variable schedules at that point. Hence my earlier remark about WoW addicts being basically slot jockeys playing in teams. :)
Tim: "A lot of what makes grinding boring is that it's not challenging - meaning you're way inside your comfort level."
This may reflect how you play i.e. a challenge-oriented Conqueror style approach (I'm guessing from your comment - correct me if I'm wrong!). Other players will find grinding boring not because it isn't challenging, per se, but because it isn't interesting.
When I play a cRPG I'm always looking for my best options to level... I'm not looking for them to be challenging, at least not in the way this term is usually used. I would rather grind a repeated section for XP (especially in, say, a turn based game where I can perfect my methods for doing so) and crush the bosses with ease than "push the curve" and have to repeat sections of play over and over again after failing. So ironically, I'd rather repeat a lot of content voluntarily than risk repeating a lot of sections as a result of failure. I know from my player studies that I'm not alone in this - players play in both the way you allude to and in the way that I mention here.
As I mention above, the capacity to support *both* kinds of play is one of the key advantages to this kind of game mechanic *for the player*. But all this, while interesting and relevant to the wider discussion, is tangential to the above.
But you and Brian are right to raise this point as I should have made explicit reference to it here, and it does look like an oversight that I didn't. Thanks for bringing it up, both of you!
My main question still remains, though: why no discussion of the escalating schedules in the literature? One possible answer is Brian's that they approximate to linear on the curve and so are effectively fixed ratio. My answer is that you don't want fixed ratio in practice, because you want the player to have decisions to make about how to progress - such as whether they want more or less challenge. And this is best achieved by gradually opening up the gaps in the schedule as the game progresses, which I'm claiming is what actually happens in practice.
I wonder if anyone has data on time to level in WoW that could be brought into this discussion...
Roman Age: "Could it just be that the more advanced in the game a player is, the more he/she will tolerate grinding sections (so the longer they can last)?"
Undoubtedly once the habit of the game is established the player feels invested and therefore more willing to continue. I'm claiming this is part of why escalating schedules work so well - you hook the player in at the beginning with quick levelling and then, once they're hooked, you open it up and give them more material to explore as they want to.
"Maybe this escalating ratio schedule in game design - and the grinding that is often associated with it, comes traditionaly from the need in video games to make levels... last longer, and artificially prolong the total playing time of the game"
This is definitely a benefit that the system provides - it gives a reason for massively repetitive play, and that's what games need for long play windows (i.e. long total playing times).
But of course, this isn't the case in positive animal training which has no interest in prolonging the training. There, this form is used because it establishes the habits most effectively. And what I'm claiming is that is part of why this succeeds commercially in digital games - it establishes the habit in the player, and thus they become invested, exactly as you suggest here.
Great comments people! I welcome any further insights into this, and especially from any psychology students who want to make the case for escalating schedule being fixed schedule in more robust terms. Personally, I don't trust this easy out any more.
Posted by: Chris | Monday, 24 January 2011 at 10:19
Inwardly, I've come to label these types of external rewards as "incentive systems". I also prefer to avoid them wherever possible- in my head they are a crutch, an artificial substitute for inherently rewarding (i.e. fun) gameplay.
At the same time, I think there are a few vital game elements that get conflated with "leveling up":
-Loadout choices, i.e. the opportunity to customize your abilities in ways that alter the tactical gameplay. (Buy the shotgun and I'll be tearing around the level, looking to ambush enemies at point-blank range. Buy the sniper rifle and I'll spend half my time working to maximize my distance from anything hostile)
-Persistence, a layer of long-term logistical gameplay with a back-and-forth influence on the moment-to-moment action. (I'm low on healing items, so I play very cautiously to minimize their consumption)
-Grind avoidance. Obstacles in the game world whose challenges have been mastered become chores. Enabling the players to "skip" those obstacles at that precise point can be a huge boon for pacing. (I'm tired of having to play red-light-green-light with these fire vents, but I just got a freeze beam that can seal them shut as I run by!)
Posted by: Brooks Harrel | Tuesday, 25 January 2011 at 01:56
Brooks: your term "incentive systems" is a perfectly good synonym for "reward schedule". I prefer to use this latter term as it underlines the connection with Skinner's experiments, which I think it prudent to remain aware of.
While like you I would prefer to see gameplay that was able to be enjoyable as part of its very nature, I think there are advantages to the progression mechanics in computer RPGs that are unique and important - specifically, the one mentioned above concerning the ability for the player to control their own level of difficulty. But of course, this benefit is lost in achievement systems et al.
Of the three other points you raise, one of them (what you're terming "persistence" has implications similar to XP systems - there is a resource being consumed (in this case) that must be kept in supply. Some players eat up this kind of logistical play - some players despise it. I am more open to it with ammo (e.g. classic Resident Evil) than I am with healing items, where I find it is often a substitute for any coherent play (e.g. Dark Alliance, where the only skill was remembering to heal).
Thanks for commenting!
Posted by: Chris | Tuesday, 25 January 2011 at 10:09
hm, 'escalating' could be 'fixed' if the (progressive) perception of the interval is a log function. You find this in psychophysics a lot....
Posted by: Insightanalysis.wordpress.com | Tuesday, 15 February 2011 at 21:04
Heather: as it happens, none of these escalating reward systems are logarithmic related (nor are they Fibonacci sequences) - they are all either ad hoc sequences or compound arithmetic functions. So this argument won't fly - nice try though. :)
Posted by: Chris | Tuesday, 22 February 2011 at 10:51