How to create metrics that can't be "gamed?"

Question

I've seen managers create metrics that worked, and ones that had unintended side-effects. For example, one manager measured his IT team based on how quickly "help request tickets" were closed. Service suffered because everyone was closing tickets even before the customer could verify the problem was actually fixed...and often it was not. The metric changed to be based on satisfaction survey data, and suddenly people were staying on the phone for hours with each customer for even the most simple request.

At other companies I've seen managers who had some kind of ESP that let them create metrics that could not be "gamed," had no unintended side-effects, and encouraged the desire behavior.

What's the secret? How can I create metrics that drive behavior to create the desired result?

Answer 1

TL;DR

There is no secret. Magic pixie dust is a controlled substance.

You Get What You Measure

This is a truism in management of any kind. You can't prevent side-effects, because any phenomenon is changed by the very act of observation. At best, you can use multiple dimensions to refine the measurements so that you reach a better approximation of your goals.

Measure Multiple Dimensions

The problem you describe is a result of single-dimension thinking. If you measure exactly one dimension, then you are controlling or correcting for a single variable. If you measure ticket closures, then you get ticket closures...if that's all you're measuring, then that's all you get.

The solution is to apply reasonable controls to all significant variables, and then apply some common sense to the results. In your case, you might need to measure at least three dimensions:

1. tickets left open as a percentage of total tickets
2. mean or median time elapsed from ticket opening to ticket closing
3. customer satisfaction with ticket resolution

and then see if you get the results you wanted. If so, great; if not, refine (or redefine) your measurements.

Use Systems Thinking

You start from the implied assumption that help desk efficiency is the problem to be solved. Is it really? Question your assumptions.

If you have a defective or poorly-designed product, then your help desk is the wrong part of the process to fix. The correct solution would be to ship high-quality, intuitive products that require less support in the first place; your organization will gain efficiencies that way irrespective of how fast or effective your help desk folks are. You will also have happier customers overall, regardless of how happy those customers who open tickets may be.

Systems thinking isn't a silver bullet, but it is generally the right solution for most process problems. Look at the whole system, including development and quality assurance, not just the help-desk sub-process.

Answer 2

Metrics that track outcomes (e.g. customer satisfaction) are better than those that track activities (e.g. how quickly tickets are closed). In your example, I'd calculate aggregate customer satisfaction not just based on those who calls are answered but also based on wait times and dropped calls. Activity based metrics have unidirectional utility i.e. they mean something useful when red, not so much when green.

Answer 3

There is a cost with everything. Every metric you create will increase one behavior and decrease another to pay for it. Every metric has a cost/benefit. Analyze carefully.