The Effective CIO

Fighting Fires November 18, 2009

Few phrases will sharpen the mind of an IT professional more than “we have an outage.”  Outages are to be avoided at all costs, the bane of our existence.  We all work diligently to build systems that will never fail.  We build in redundancy so that users will never know that a particular piece of equipment went off-line, or that someone kicked a cable out of the wall.  We are definitely a belt-and-suspenders crowd.

In a sense, that’s a shame.  Don’t misunderstand; I’m certainly not advocating more frequent outages as a way to spice up our day-to-day lives. But if you never have an outage, you’re missing a big opportunity: the chance to see your staff really shine.

Good IT people rise to the challenge of an outage.  Mindful of the impact, not wanting to disappoint customers, challenged by the technical problems, a good operations team will do amazing things to get their systems back up and running. It is a privilege just to watch them in action.

As much as your people make things look easy when all is going well, you are quickly reminded of how complicated their world really is when things run off the rails.  The many levels of abstraction coupled with the intricate details make it almost impossible for any one person to fully understand how all the pieces fit together.  A good team will play off one another, sharing information and supplying clues that collectively solve the problem.

How does this happen?  It certainly isn’t by chance.  Good operations teams develop a deep sense of ownership for the systems they tend.  It isn’t “a system;” it is “their system.”  Typically, they built it from the ground up, know every bit of software installed on it, and configured most of the settings themselves.  Like a mechanic and an automobile, a systems administrator forms bonds with their systems that will pay off when the chips are down.

To outsiders, this sounds a bit odd and even creepy.  But anyone who has been in operations understands this completely.  Each system is special and requires specific attention in unique ways.  You cannot typically step up and just take control, you have to know how and why each component was added and maintained.  Great operations people have this knowledge and use it to their advantage when needed.

Make sure you give IT people the chance to own their systems.  They need to be included in the design and development early on, integral to the decisions that drive the system design.  As the systems mature and develop, your people will acquire the knowledge that will really make them shine when things go wrong.  And may you never have a chance to see your people at their very best, when they are digging out from a disaster.

[tweetmeme source=”EffectiveCIO” alias=”http://bit.ly/cio133″ only_single=false]