Be Careful What You Measure

Have you taken a hard look at your metrics recently?  Not the values and trend, but the underlying attribute being measured.  Is it really an effective reflection of some facet of business success?  A project by the federal government should teach us a couple lessons.

From the Cato-at-Liberty blog,

One of the behind-the-scenes initiatives of President Bush’s budget staff the past six years has been something called the Program Assessment Ratings Tool (PART) analysis. It’s an effort to measure the “effectiveness” and “efficiency” of nearly 1,000 federal programs. Each program is graded on how well it achieves its “goals.”

Sounds good.  About time we measure the effectiveness of programs, right?  I wonder what the results show.  In Tuesday’s Investor’s Business Daily op-ed section, Ernest Christian and Gary Robbins take a look at the results to date of the effort: 

Congress is about to wave its wand over nearly $1 trillion of additional “discretionary” spending that will, among other things, perpetuate or increase funding for nearly 500 expenditure programs that are not even “moderately effective,” according to the Office of Management and Budget. This includes more than 200 expenditure programs that have failing grades of D or F.  In these cases alone, the cost of government incompetence is over $250 billion per year.

The list of programs with the lowest grades might make any supporter of limited government point wildly and say, “Told you so!” This rogue’s gallery includes the Department of Housing and Urban Development’s pork-filled Community Development Block Grants, the Department of Education’s Even Start literacy program, and Amtrak.

There we go!  All kinds of waste just begging to be removed.  Don't even think about raising my taxes, and these are probably the last people I want in charge of my healthcare and retirement.  But I wonder what some of the more "efficient" programs are...

But what about the ones that received the equivalent of an A or B grade – those programs that are “effective” or “moderately effective”? That list includes homeless assistance grants, agricultural export subsidies, Indian housing loan guarantees, the non-insured crop assistance program, and corporate welfare programs like the Trade and Development Agency which subsidizes overseas demand for the products of various corporations.

Hmmm... uh oh.  The government is efficient at spending money on corporate welfare and helping out people that decided a government bailout was cheaper than buying insurance?

Sure, knowing when the government is losing money to fraud or mismanagment is important. But it makes more sense to determine whether these programs should exist at all before deciding what they should be “efficient” at doing. Besides, an efficient but unjustified wealth-redistribution program might actually be worse than an inefficient one. The former will likely be better at finding innovative ways of expanding the scope of its operations.

And that's also the lesson for businesses and other organizations.  Take the time to really understand what you are measuring, and whether a positive trend is really a positive.  Many of us know (and some of us have experienced) the saying of "you can sell yourself out of business" when you don't understand how cash flow plays into converting the top line into the bottom line.  I would add you can "streamline your way out of business" if you find ways to create waste very efficiently.

Comments

[John Hunter]

And be careful drawing conclusions from what you think the data says. This study provided identical search results to participants branded by Google, Yahoo... and the results showed people preferred the Yahoo results...

My guess is actually that the study picked up random variation. But lets say the study is accurate and repeatable and gives the same results. That would tell you that when you ask for people to provide you certain data they often fail to provide what you ask for - here they were asked to provide a ranking of the results (not the branding) but if since the only difference is branding (if it is an accurate, repeatable results) then people are not giving what was asked for but instead a branding preference.

This might seem silly in this example but companies often ask about satisfaction with x or y. While that data can be useful if you actually believe it is as specific as stated in the data you are wrong (each specific response is influenced by other factors).

When possible revealed preferences are preferable to stated preferences (ht Sean Stickle).

[Thomas Sortino]

Hmm... sounds like the Cato folks are suggesting to do 5S before starting improvement efforts - very sound advice indeed!