Quality and Innovation

It’s that time of year where people are dusting off their strategic plans, hosting their parties and strategy workshops, and making sure the KPIs and metrics on their scorecards are the ones they want to be watching as their new roadmap unfolds.

But most people really aren’t that religious about measurement systems, or tightly aligning specific actions with the needle they are most likely to move. The goal of “becoming data-driven” usually isn’t accompanied by the discipline and perseverance to make it happen, even though the payoffs of doing it right are huge. (Granted, it’s easy for business leaders to get warm fuzzy feelings from numbers that look right, even when they’re nowhere near accurate or trustworthy. This makes it even harder to help organizations manage their data with integrity.)

And none of us are immune to bad metrics, even when “doing it right” is really important. Sometimes, a metric is just too emotionally enticing to give up. For some, it’s conversions (measured so incompletely as to be unusable). For others, it’s ROAS (don’t even get me started about all the threats to validity, even at the back-of-envelope level). For me, it’s… citations.

No matter how bad I know it is, I keep looking at citations to evaluate (one dimension of) my personal value. PSA: It is never good to tie (even a tiny fraction of) your worth as a human to a metric (any metric). Gen Z may have more luck than us Gen Xers on this one.

I just can’t emotionally detach from number of citations on Google Scholar. And the reason why I’m thinking about it today is because… I just achieved my 2020 goal of adding more citations than I added in 2019!

Here’s why this metric is so terribly bad:

• Number of citations is a lagging indicator, and the lag can often be 3-5 years.
• By the time the needle moves, it’s hard to figure out exactly what happened to make it move.
• There are very few actions I can personally take to make that needle move.
• Any actions I do take will be indirect. I can make people more aware of my papers, but can’t force anyone to cite one… so the actions I can take will influence reach, but not citations.
• There’s an interesting social dimension to past number of citations. The more citations you have on a paper, the more likely you are to attract additional citations; similarly, the more citations you have, the better SEO you get on sites like Google Scholar. It’s a “fit get fitter” scenario.
• I can’t monitor this metric on a weekly or monthly basis. If it dips, I won’t be able to respond by taking an action to restore growth.
• I haven’t even thought about using this metric as a signal for when I should take action. Because of all the problems I listed above.
• I didn’t do anything to achieve my 2020 goal. I just helplessly watched that number creep up, kept my fingers crossed, and (now) celebrated on December 19th when I (just barely) went over the wire before New Year’s.
• The calendar is arbitrary anyway. What if I achieved the goal on January 5? Would I feel unaccomplished? Probably yes (this is pathetic)!
• Ultimately, I am not in control of citations. I should have picked an intermediary metric that I am in control of… but that’s really difficult, and I’m not in academia any more so I really don’t even need to pay attention to this (another giant problem! Why am I still even paying attention? Attention is expensive!)

My Holiday wish for you is: Select your metrics carefully! Pick ones that are (ideally):

• Not limited to lagging indicators with extraordinarily long lags
• Monitorable on (at least) a weekly or monthly basis
• Designed so that if you aren’t achieving your target level, you can immediately figure out where the problem is happening, and even know how to dig down deeper to figure out why that problem is happening
• Triggers for action: every metric that’s not where you want it to be should be link to a thing you can do — that’s in your control — that you know has a pretty good chance of making that needle move the direction you want it to

When your metrics aren’t revealing, actionable, or in your control, you’ve just set yourself up for a special kind of paralysis the entire year.