Anecdote for when metrics go bad

More and more parts of our society are metric-obsessed these days.

While I’m in theory in favor of science, scientific approaches based on observation, data-driven decisions, etc.—I am also more and more cautious of the ways that metrics can be abused, be gamed, be interpreted incorrectly, and generally be insufficient proxies for the thing meant to be measured.   There are a bunch of possible reasons this can happen.

Just an anecdote, here’s a real phone call I just got:

“Hi, this is [X] calling from [Honda Dealership]. We’d like to thank you for bringing your car in for service the other day. You may be receiving a survey from Honda about your satisfaction. We just wanted to let you know that if you rate us 100% in all categories, then your next oil change will be free.”

Ah, yeah. Clearly the survey is from corporate Honda America or whatever; the phone call was from the dealer; and I’d guess that both the dealer as a whole and the individual staff at the dealer have various kinds of compensation pegged to these survey results.

(Incidentally, after seeing how over-priced the service was for the oil/filter change, air filter change in both cabin and engine, wiper refills, and checks of various lines and fluids — it’s very unlikely i’ll be going to the dealer again. It was at least 2x what it probably should have been, I am feeling serious pain in the wallet).

This kind of blatant manipulation certainly isn’t the only way that metrics-based approaches can end up misleading.  Talk to any teacher about “School Reform”, for issues with similar “performance-based compensation” — the problem isn’t just intentional abuse, but that the operationalized measurements may simply not validly measure what you really want to measure, for all sorts of reasons.  Inept application of statistical methods (which I think is awfully common even among scientists, let alone among Car Companies and Libraries) can compound the issue.

Then a larger philosophical issue is how choosing to look at things only in terms of quantitative measurements effects our judgements and perspectives. Someone recently recommended Michel Foucault to me on this topic, although I’m not sure which work of his would be most pertinent.

Use metrics for sure, but don’t be ruled by them, and don’t assume that just because something is reported to you in a quantitative fashion that automatically means it’s objective, accurate, or valid. (Or that any of those categories, ‘objective’, ‘accurate’, or ‘valid’, are simple yes-or-no, rather than questions of degree).


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s