Measuring knowledge work – when measures become targets

Posted by Shawn Callahan - November 13, 2006
Filed in Changing behaviour, Knowledge

The whole fraud is only possible because performance metrics in knowledge organizations are completely trivial to game. Joel on Software

Joel demonstrates a weakness of metrics in assessing the performance of software developers and somewhat cynically suggests that large management consulting firms actively exploit this weakness. Whenever I have a choice between malicious intent and incompetence I tend towards the latter as an explanation.  Personally I believe large consulting firms have these problems because they are steeped in particular mental models, namely the idea that organisations are machines and if you can’t measure it you can’t fix it. Fredrick Brooks understood this and painted a clear picture of the complexity of managing software development projects in his classic, The Mythical Man-Month.

So what is the weakness Joel uncovers? Companies want to improve performance (excellent objective) and consulting companies have methods to measure the current performance (it’s good to know where you stand). But here is the big mistake: the consulting company makes the measure of performance a target. And this is what happens:

Consulting company comes in, gets all the programmers in a room, tells them all about Function Points [the measure of performance] and stuff, and how productivity is REALLY IMPORTANT.

Programmers remember that scene from Office Space where Bob and Bob, the consultants, recommended all the people to get fired.

Programmers start writing a heck of a lot more function points. For example you can triple the number of function points in your code simply by round tripping everything through an XML file. Big waste of time, prone to bugs, does nothing, but each file you touch adds a function point. W00t!

Consulting company comes back, measures again, and lo and behold, with all the round trips through XML the function point count is up drastically. Consultant announces that Oil Company is now at 151.29% productivity. MISSION ACCOMPLISHED.

Here is an alternative approach.

  • Work with the leaders to get a broad picture of what they think performance is in the context of software development. Get them to paint that broad picture while resisting the temptation to fill in all the gaps.
  • Ask the software developers to define the performance measures and keep them to a minimum.
  • Be clear what will remain measures and what will be the targets
  • Add to the metrics approach a qualitative assessment of impact using Most Significant Change (see Zahmoo to learn about this technique).

This approach is based on the idea that any journey of change (actually I prefer the metaphor of the expedition) should be created three times. The first creation is in the minds of the leadership group (however you want to define that). The second creation of the journey is in the minds of the participants. Finally the journey is created for real as everyone sets off together and it’s here that everyone discovers that is never turns out the way they expected, so a willingness to monitor and adapt is essential.

[thanks to Les Posen for the pointer to Joel’s post]

About Shawn Callahan
Shawn is one of the world's leading business storytelling consultants. He helps executive teams find and tell the story of their strategy. When he is not working on strategy communication, Shawn is helping leaders find and tell business stories to engage, to influence and to inspire. Shawn works with Global 1000 companies including Shell, IBM, SAP, Bayer, Microsoft & Danone. Connect with Shawn on:

Comments Off

Comments are closed.