Tag Archives: KPI

BAD MetRICS

It’s that time of year where people are dusting off their strategic plans, hosting their parties and strategy workshops, and making sure the KPIs and metrics on their scorecards are the ones they want to be watching in 2021.

But most people really aren’t that religious about measurement systems, or tightly aligning specific actions with the needle they are most likely to move. The goal of “becoming data-driven” usually isn’t accompanied by the discipline and perseverance to make it happen, even though the payoffs are huge.

And none of us are immune to bad metrics, even when those things are really important. Sometimes, a metric is just too emotionally enticing to give up.

I use one bad metric myself, and no matter how bad I know it is, I keep using it to evaluate (one dimension of) my personal value. PSA: It is never good to tie your worth as a human to a metric (any metric). Gen Z may have more luck than us Gen Xers on this one.

My bad metric, the one I can’t emotionally detach from, is number of citations on Google Scholar. And the reason why I’m thinking about it today is because… I just achieved my 2020 goal of adding more citations than I added in 2019!

Here’s why this metric is so terribly bad:

  • Number of citations is a lagging indicator, and the lag can often be 3-5 years.
  • By the time the needle moves, it’s hard to figure out exactly what happened to make it move.
  • There are very few actions I can personally take to make that needle move.
  • Any actions I do take will be indirect. I can make people more aware of my papers, but can’t force anyone to cite one… so the actions I can take will influence reach, but not citations.
  • There’s an interesting social dimension to past number of citations. The more citations you have on a paper, the more likely you are to attract additional citations; similarly, the more citations you have, the better SEO you get on sites like Google Scholar. It’s a “fit get fitter” scenario.
  • I can’t monitor this metric on a weekly or monthly basis. If it dips, I won’t be able to respond by taking an action to restore growth.
  • I haven’t even thought about using this metric as a signal for when I should take action. Because of all the problems I listed above.
  • I didn’t do anything to achieve my 2020 goal. I just helplessly watched that number creep up, kept my fingers crossed, and (now) celebrated on December 19th when I (just barely) went over the wire before New Year’s.
  • The calendar is arbitrary anyway. What if I achieved the goal on January 5? Would I feel unaccomplished? Probably yes (this is pathetic)!
  • Ultimately, I am not in control of citations. I should have picked an intermediary metric that I am in control of… but that’s really difficult, and I’m not in academia any more so I really don’t even need to pay attention to this (another giant problem! Why am I still even paying attention? Attention is expensive!)

My Holiday wish for you is: Select your metrics carefully! Pick ones that are (ideally):

  • Not limited to lagging indicators with extraordinarily long lags
  • Monitorable on (at least) a weekly or monthly basis
  • Designed so that if you aren’t achieving your target level, you can immediately figure out where the problem is happening, and even know how to dig down deeper to figure out why that problem is happening
  • Triggers for action: every metric that’s not where you want it to be should be link to a thing you can do — that’s in your control — that you know has a pretty good chance of making that needle move the direction you want it to

When your metrics aren’t revealing, actionable, or in your control, you’ve just set yourself up for a special kind of paralysis the entire year.

KPIs vs Metrics: What’s the Difference? And Why Does it Matter?

Years ago I consulted for an organization that had an enticing mission, a dynamic and highly qualified workforce of around 200 people, and an innovative roadmap that was poised to make an impact — estimated to be ~$350-500M (yes really, that big). But there was one huge problem.

As engineers, the leadership could readily provide information about uptime and Service Level Agreements (SLAs). But they had no idea whether they were on track to meet strategic goals — or even whether they would be able to deliver key operations projects — at all! We recommended that they focus on developing metrics, and provided some guidelines for the types of metrics that might help them deliver their products and services — and satisfy their demanding customers.

Unfortunately, we made a critical mistake.

They were overachievers. When we came back six months later, they had nearly a thousand metrics. (A couple of the guys, beaming with pride, didn’t quite know how to interpret our non-smiling faces.)

“So tell us… what are your top three goals for the year, and are you on track to meet them?” we asked.

They looked at each other… then at us. They looked down at their papers. They glanced at each other again. It was in that moment they realized the difference between KPIs and metrics.

  • KPIs are KEY Performance Indicators. They have meaning. They are important. They are significant. And they relate to the overall goals of your business.
  • One KPI is associated with one or more metrics. Metrics are numbers, counts, percentages, or other values that provide insight about what’s happened in the past (descriptive metrics), what is happening right now (diagnostic metrics), what will happen (predictive metrics or forecasts), or what should happen (prescriptive metrics or recommendations).

For the human brain to be able to detect and respond to patterns in organizational performance, limit the number of KPIs!

A good rule of thumb is to select 3-5 KPIs (but never more than 8 or 9!) per logical division of your organization. A logical division can be a functional area (finance, IT, call center), a product line, a program or collection of projects, or a collection of strategic initiatives.

Or, use KPIs and metrics to describe product performance, process performance, customer satisfaction, customer engagement, workforce capability, workforce capacity, leadership performance, governance performance, financial performance, market performance, and how well you are executing on the action plans that drive your strategic initiatives (strategy performance). These logical divisions come from the Baldrige Excellence Framework.

Similarly, try to limit the number of projects and initiatives in each functional area — and across your organization. Work gets done more easily when people understand how all the parts of your organization relate to one another.

What happened to the organization from the story, you might ask? Within a year, they had boiled down their metrics into 8 functional areas, were working on 4 strategic initiatives, and had no more than 5 KPIs per functional area. They found it really easy to monitor the state of their business, and respond in an agile and capable way. (They were still collecting lots more metrics, but they only had to dig into them on occasion.)

Remember… metrics are helpful, but:

KPIs are KEY!!

You don’t have thousands of keys to your house… and you don’t want thousands of KPIs. Take a critical look at what’s most important to your business, and organize that information in a way that’s accessible. You’ll find it easier to manage everything — strategic initiatives, projects, and operations.

Modernizing the Balanced Scorecard

We covered Kaplan and Norton’s (1996) Balanced Scorecard in my ISAT 654 (Advanced Technology Management) class at JMU a couple weeks ago. Appropriately, the students recognized that the research into balanced metrics and the strategy maps underlying the method is nearly 20 years old. Hasn’t the Balanced Scorecard been modernized since then, they asked, to reflect changing concerns in a changing world – especially social, political and environmental concerns? After all, sustainability and social responsibility are major concerns these days, and any scorecard that’s worth its salt should incorporate these considerations, they argued.

Here are a few “modernizations” I found that might be of interest:

 


References:

Möller, A. & Schaltegger, S., (2005). The Sustainability Balanced Scorecard as a Framework for Eco-Efficiency Analysis, Journal of Industrial Ecology, 9(4), 73-83.

Sidiropolous, M., Mouzakitis, Y., Adamides, E., & Goutsos, S. (2004). Applying Sustainable Indicators to Corporate Strategy: The Eco-Balanced Scorecard. Environmental Research, Engineering & Management, 1(27), 28-33.