jump to navigation

What Are You Measuring? October 21, 2009

Posted by Chuck Musciano in Leadership.
Tags: , ,

When you boil it all down, computing is about numbers.  Two of them, actually, 0 and 1.  Over the years, we’ve worked up from that, of course, so that you have to dig to find the 0s and 1s, but our obsession with numbers is deeply ingrained.  This has bad implications for those of who try to lead IT organizations.

Given our predilection for numbers, most people in IT like to collect them.  Storage usage, bandwidth, code size, database tables, and the like are obvious targets for the numerically fascinated.  But we also collect more abstracted numbers, like project success rates, hours spent doing things, call volumes, and satisfaction indices.  Sometimes, we place a lot of importance on these numbers and work hard to optimize them in some fashion.

Some numbers are necessary.  On the operations side of the house, disk utilization and processor loading are important metrics that drive good capacity planning models.  These numbers are easy to collect and understand because they relate back to physical resources that can be measured accurately.

Other numbers are a bit softer.  Many organizations try to quantify qualitative data, like customer satisfaction.  Gauging satisfaction is tough; there is no number that equates to “great” or “awful.”  That doesn’t stop us, however: our inherent love of numbers leads us to assign numbers to feelings and opinions. That’s not inherently bad when we make simple comparisons.  When one customer rates us a 9 and another decides we are a 1, there is a clear difference of opinion.

The problem with numbers is that they are so prone to manipulation.  Once you make the leap from adjective to value, it is way too easy to start doing arithmetic.  Suddenly, we are averaging those ratings, or worse, computing standard deviations and higher order statistical metrics.  These computed values are worthless, no matter how attractive they may seem.

Consider: if you are in a room with two people, one of which says “I love you” and another that says “I hate you,” the average in the room is not “We like you.” Depending on which way you turn, you are going to either get kicked or kissed.  Math and emotion simply do not mix.

In spite of this, many organizations use these numerical metrics to make business decisions and control compensation.  I’ve seen teams rejoice when customer satisfaction climbs from 3.3 to 3.5, as if the 0.2 difference has any significance.  They are beholden to the numbers and have lost track of the feelings and emotion behind them.

Part of being an expert with a tool is knowing when you shouldn’t use it.  We fancy ourselves to be experts with numbers; we should do a better job of applying them appropriately. In many parts of our businesses, we need to stop focusing on numbers and start listening to people. That’s where you’ll find the real answers and understand your real problems.

[tweetmeme source=”EffectiveCIO” alias=”http://bit.ly/cio120″ only_single=false]


1. Scott Duncan - October 21, 2009

Good points.

Have a related post on “mononumerosis” and “averages” at http://agilesoftwarequalities.blogspot.com/2009/09/mononumerosis-example-and-measurement.html

2. Susan Mazza - October 23, 2009

You are so right – measurements in and of themselves can be misleading. I’ve dealt a lot with the numbers on Employee Opinion Surveys. They can be helpful indicators, but what makes the biggest difference in improving scores is actively engaging with people and following up with actions that let them know you listened to and heard them. Lots of money and time is invested in all kinds of “programs” targeted at improving scores, yet what is really needed is more consistent straight and meaningful conversation.

3. jerryhelms - October 23, 2009

The Harvard Business Review published an interesting article on measuring customer satisfaction. The short version of the article: “Would you recommend this company to a friend?” 0 or 1 — yes or no. A whole new school of simple measurements has cropped up as a result of this article.


Jerry Helms
Non Stop Portals, Raleigh, NC

4. Wally Bock - October 28, 2009

Congratulations! This post was selected as one of the five best independent business blog posts of the week in my Three Star Leadership Midweek Review of the Business Blogs.


Wally Bock

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: