The Effective CIO

What Are You Measuring? October 21, 2009

When you boil it all down, computing is about numbers.  Two of them, actually, 0 and 1.  Over the years, we’ve worked up from that, of course, so that you have to dig to find the 0s and 1s, but our obsession with numbers is deeply ingrained.  This has bad implications for those of who try to lead IT organizations.

Given our predilection for numbers, most people in IT like to collect them.  Storage usage, bandwidth, code size, database tables, and the like are obvious targets for the numerically fascinated.  But we also collect more abstracted numbers, like project success rates, hours spent doing things, call volumes, and satisfaction indices.  Sometimes, we place a lot of importance on these numbers and work hard to optimize them in some fashion.

Some numbers are necessary.  On the operations side of the house, disk utilization and processor loading are important metrics that drive good capacity planning models.  These numbers are easy to collect and understand because they relate back to physical resources that can be measured accurately.

Other numbers are a bit softer.  Many organizations try to quantify qualitative data, like customer satisfaction.  Gauging satisfaction is tough; there is no number that equates to “great” or “awful.”  That doesn’t stop us, however: our inherent love of numbers leads us to assign numbers to feelings and opinions. That’s not inherently bad when we make simple comparisons.  When one customer rates us a 9 and another decides we are a 1, there is a clear difference of opinion.

The problem with numbers is that they are so prone to manipulation.  Once you make the leap from adjective to value, it is way too easy to start doing arithmetic.  Suddenly, we are averaging those ratings, or worse, computing standard deviations and higher order statistical metrics.  These computed values are worthless, no matter how attractive they may seem.

Consider: if you are in a room with two people, one of which says “I love you” and another that says “I hate you,” the average in the room is not “We like you.” Depending on which way you turn, you are going to either get kicked or kissed.  Math and emotion simply do not mix.

In spite of this, many organizations use these numerical metrics to make business decisions and control compensation.  I’ve seen teams rejoice when customer satisfaction climbs from 3.3 to 3.5, as if the 0.2 difference has any significance.  They are beholden to the numbers and have lost track of the feelings and emotion behind them.

Part of being an expert with a tool is knowing when you shouldn’t use it.  We fancy ourselves to be experts with numbers; we should do a better job of applying them appropriately. In many parts of our businesses, we need to stop focusing on numbers and start listening to people. That’s where you’ll find the real answers and understand your real problems.

[tweetmeme source=”EffectiveCIO” alias=”http://bit.ly/cio120″ only_single=false]