Over the past few years, Agile has gained popularity. This methodology emerged as a solution to manage projects with a number of unknown elements and to counter the typical waterfall method. Quality practitioners have observed the numerous similarities between this new framework and Lean. Some have speculated that Agile is simply the next generation’s version of Lean. These observations have posed the question: Is Agile the new Lean?
ASQ Influential Voices Roundtable for December 2019
The short answer to this question is: NO.
The longer answer is one I’m going to have to hold back some emotions to answer. Why? I have two reasons.
Reason #1: There is No Magic Bullet
First, many managers are on a quest for the silver bullet — a methodology or a tool that they can implement on Monday, and reap benefits no later than Friday. Neither lean nor agile can make this happen. But it’s not uncommon to see organizations try this approach. A workgroup will set up a Kanban board or start doing daily stand-up meetings, and then talk about how they’re “doing agile.” Now that agile is in place, these teams have no reason to go any further.
Reason #2: There is Nothing New Under the Sun
Neither approach is “new” and neither is going away. Lean principles have been around since Toyota pioneered its production system in the 1960s and 1970s. The methods prioritized value and flow, with attention to reducing all types of waste everywhere in the organization. Agile emerged in the 1990s for software development, as a response to waterfall methods that couldn’t respond effectively to changes in customer requirements.
Agile modeling uses some lean principles: for example, why spend hours documenting flow charts in Visio, when you can just write one on a whiteboard, take a photo, and paste it into your documentation? Agile doesn’t have to be perfectly lean, though. It’s acceptable to introduce elements that might seem like waste into processes, as long as you maintain your ability to quickly respond to new information and changes required by customers. (For example, maybe you need to touch base with your customers several times a week. This extra time and effort is OK in agile if it helps you achieve your customer-facing goals.)
Both lean and agile are practices. They
require discipline, time, and monitoring. Teams must continually hone their
practice, and learn about each other as they learn together. There are no magic
Information plays a key role. Effective flow of information from strategy to action is important for lean because confusion (or incomplete communication) and forms of waste. Agile also emphasizes high-value information flows, but for slightly different purposes — that include promoting:
Rapid, targeted, and effective action
The difference is easier to understand if you watch a couple cat videos.
This Cat is A G I L E
This cat is continuously scanning for information about its environment. It’s young and in shape, and it navigates its environment like a pro, whizzing from floor to ceiling. If it’s about to fall off something? No problem! This cat is A G I L E and can quickly adjust. It can easily achieve its goal of scaling any of the cat towers in this video. Agile is also about trying new things to quickly assess whether they will work. You’ll see this cat attempt to climb the wall with an open mind, and upon learning the ineffectiveness of the approach, abandoning that experiment.
This Cat is L E A N
This cat is using as LITTLE energy as possible to achieve its goal of hydration. Although this cat might be considered lazy, it is actually very intelligent, dynamically figuring out how to remove non-value-adding activity from its process at every moment. This cat is working smarter, not harder. This cat is L E A N.
Hope this has been helpful. Business posts definitely need more cat videos.
One of the heuristics we use at Intelex to guide decision making is former US President Truman’s advice that “imperfect action is better than perfect inaction.” What it means is — don’t wait too long to take action, because you don’t want to miss opportunities. Good advice, right?
When I share this with colleagues, I often hear a response like: “that’s dangerous!” To which my answer is “well sure, sometimes, but it can be really valuable depending on how you apply it!” The trick is: knowing how and when.
Here’s how it can be dangerous. For example, statistical process control (SPC) exists to keep us from tampering with processes — from taking imperfect action based on random variation, which will not only get us nowhere, but can exacerbate the problem we were trying to solve. The secret is to apply Truman’s heuristic based on an understanding of exactly how imperfect is OK with your organization, based on your risk appetite. And this is where loss functions can help.
Along the way, we’ll demonstrate how to do a few important things related to plotting with the ggplot package in R, gradually adding in new elements to the plot so you can see how it’s layered, including:
Plot a function based on its equation
Add text annotations to specific locations on a ggplot
Draw horizontal and vertical lines on a ggplot
Draw arrows on a ggplot
Add extra dots to a ggplot
Eliminate axis text and axis tick marks
What is a Loss Function?
A loss function quantifies how unhappy you’ll be based on the accuracy or effectiveness of a prediction or decision. In the simplest case, you control one variable (x) which leads to some cost or loss (y). For the case we’ll examine in this post, the variables are:
How much time and effort you put in to scoping and characterizing the problem (x); we assume that time+effort invested leads to real understanding
How much it will cost you (y); can be expressed in terms of direct costs (e.g. capex + opex) as well as opportunity costs or intangible costs (e.g. damage to reputation)
Here is an example of what this might look like, if you have a situation where overestimating (putting in too much x) OR underestimating (putting in too little x) are both equally bad. In this case, x=10 is the best (least costly) decision or prediction:
# describe the equation we want to plot
parabola <- function(x) ((x-10)^2)+10
# initialize ggplot with a dummy dataset
p <- ggplot(data = data.frame(x=0), mapping = aes(x=x))
p + stat_function(fun=parabola) + xlim(-2,23) + ylim(-2,100) +
xlab("x = the variable you can control") +
ylab("y = cost of loss ($$)")
In regression (and other techniques where you’re trying to build a model to predict a quantitative dependent variable), mean square error is a squared loss function that helps you quantify error. It captures two facts: the farther away you are from the correct answer the worse the error is — and both overestimating and underestimating is bad (which is why you square the values). Across this and related techniques, the loss function captures these characteristics:
Not all loss functions have that general shape. For classification, for example, the 0-1 loss function tells the story that if you get a classification wrong (x < 0) you incur all the penalty or loss (y=1), whereas if you get it right (x > 0) there is no penalty or loss (y=0):
# set up data frame of red points
d.step <- data.frame(x=c(-3,0,0,3), y=c(1,1,0,0))
# note that the loss function really extends to x=-Inf and x=+Inf
ggplot(d.step) + geom_step(mapping=aes(x=x, y=y), direction="hv") +
geom_point(mapping=aes(x=x, y=y), color="red") +
xlab("y* f(x)") + ylab("Loss (Cost)") +
ggtitle("0-1 Loss Function for Classification")
Use the Loss Function to Make Strategic Decisions
So let’s get back to Truman’s advice. Ideally, we want to choose the x (the amount of time and effort to invest into project planning) that results in the lowest possible cost or loss. That’s the green point at the nadir of the parabola:
What this means is — if we don’t plan, or we plan just a little bit, we incur high costs. We might make the wrong decision! Or miss critical opportunities! But if we plan too much — we’re going to spend too much time, money, and/or effort compared to the benefit of the solution we provide.
The trick is to FIND THAT CRITICAL LEVEL OF TIME and EFFORT invested to gain information and understanding about your problem… and then if you’re going to err, make sure you err towards the left — if you’re going to make a mistake, make the mistake that costs less and takes less time to make:
The moral of the story is… imperfect action can be expensive, but perfect action is ALWAYS expensive. Spend less to make mistakes and learn from them, if you can! This is one of the value drivers for agile methodologies… agile practices can help improve communication and coordination so that the loss function is minimized.
## FULL CODE FOR THE COMPLETELY ANNOTATED CHART ##
# If you change the equation for the parabola, annotations may shift and be in the wrong place.
parabola <- function(x) ((x-10)^2)+10
my.title <- expression(paste("Imperfect Action Can Be Expensive. But Perfect Action is ", italic("Always"), " Expensive."))
arrow.x <- c(10, 10, 10, 10)
arrow.y <- c(35, 50, 65, 80)
arrow.x.end <- c(6, 6, 6, 6)
arrow.y.end <- arrow.y
d <- data.frame(arrow.x, arrow.y, arrow.x.end, arrow.y.end)
p + stat_function(fun=parabola) + xlim(-2,23) + ylim(-2,100) +
xlab("Time Spent and Information Gained (e.g. person-weeks)") + ylab("$$ COST $$") +
annotate(geom="text", x=10, y=5, label="Some Effort, Lowest Cost!!", color="darkgreen") +
geom_point(aes(x=10, y=10), colour="darkgreen") +
annotate(geom="text", x=0, y=100, label="$$$$$", color="green") +
annotate(geom="text", x=0, y=75, label="$$$$", color="green") +
annotate(geom="text", x=0, y=50, label="$$$", color="green") +
annotate(geom="text", x=0, y=25, label="$$", color="green") +
annotate(geom="text", x=0, y=0, label="$ 0", color="green") +
annotate(geom="text", x=2, y=0, label="minutes\nof effort", size=3) +
annotate(geom="text", x=20, y=0, label="months\nof effort", size=3) +
annotate(geom="text",x=3, y=85, label="Little (or no) Planning\nHIGH COST", color="red") +
annotate(geom="text", x=18, y=85, label="Paralysis by Planning\nHIGH COST", color="red") +
geom_vline(xintercept=0, linetype="dotted") +
geom_hline(yintercept=0, linetype="dotted") +
geom_segment(data=d, mapping=aes(x=arrow.x, y=arrow.y, xend=arrow.x.end, yend=arrow.y.end),
arrow=arrow(), color="blue", size=2) +
annotate(geom="text", x=8, y=95, size=2.3, color="blue",
label="we prefer to be\non this side of the\nloss function") +
Now sometimes you need to make this investment! (Think nuclear power plants, or constructing aircraft carriers or submarines.) Don’t get caught up in getting your planning investment perfectly optimized — but do be aware of the trade-offs, and go into the decision deliberately, based on the risk level (and regulatory nature) of your industry, and your company’s risk appetite.
I really like Doug Buckley’s Facebook posts (he’s from http://hyperactive.to). Our connection was purely accidental – he tagged a picture of the back of my husband’s head on Facebook as his own, and after a short online debate (where he finally acknowledged that I was probably an expert in recognizing the back of my own spouse’s head) we friended one another. Doug posts great photos and images (like the one on the top left of this post), insights, quotes and music about 40 or 50 times a day. One of Doug’s recent gems was “Success is the ability to go from one failure to another with no loss of enthusiasm.” ~ Winston Churchill
The ability to move from one failure to another (presumably, using each as a valuable learning experience) with no loss of enthusiasm! Wow, I thought… not only is that the secret to innovation, but… I am SO not good at doing that. When I’m facing a failure, I do what any sane, logical person in the grips of Chinese handcuffs will do… I pull harder. I rearrange the deck chairs faster. I get really &$^#&^$& angry. Then I pretend like everything’s OK. And when I can’t deal with pretending any longer, I break down into tears (hopefully not around other people).
Then, in the words of Fred and Ginger, I pick myself up – dust myself off – and start all over again.
(I just read this again, and just so you don’t miss the point of that last sentence… after I dust myself off, I’m starting again on the negative pattern of trying even harder. What, you think I’d give up that easily?)
It’s a miserable approach, though, regardless of how noble it sounds. Pulling harder or pushing harder (whatever you’re doing) requires more effort and rarely generates better results. And if you’re pushing against someone else who’s not ready to see your light, or pushing on a project that other people just aren’t ready to play within the bounds of, well… good luck.
I’m a smart person. I’m solution oriented. I can make things happen!! As a result, I doggedly pursue my goals. And when I’m meeting with resistance (especially when that resistance doesn’t seem to make sense to me), I don’t respond very gracefully at all. (Sometimes I even turn psycho-chick, which makes me feel even more disturbed, because I’m pretty level headed in general and I wouldn’t act like that, would I?)
The illusion of control is an affliction that’s unique to humans. Bears looking for salmon will move on if their favorite spot in the river isn’t producing. If people were bears, we’d stick around, keep waiting, commit to a positive attitude, convene a quality circle or tiger team, rehash past data that proves the salmon used to be there (or extrapolate to show they will be there again, really), wish real hard that the salmon are still there, pretend nothing has changed, craft convincing arguments that the cost/benefit of moving to another place in the river is prohibitive (or my favorite, just cost neutral), curse the river, wish we’d never gotten into the habit of eating salmon in the first place, lose all motivation, lose sense of the meaning in one’s life without the salmon, or sit on the riverbanks weeping over the ephemeral salmon who just won’t show up no matter what we do. Pretty pathetic. Nowhere near as agile as moving to another spot in the river where the salmon may have moved on to themselves.
So this alternative theory turns the standard explanation on its head. Depressed people don’t end up lying in bed because they are undercommitted to goals. They end up lying in bed because they areovercommitted to goalsthat are failing badly. The idea that depressed people cannot disengage efforts from failure is a relatively new theory. It has not been much tested in research studies. However, the idea is well worth exploring. It fits well clinically with the kinds of situations that often precipitate serious depression — the battered wife who cannot bring herself to leave her troubled marriage, the seriously injured athlete who cannot bring himself to retire, the laid off employee who cannot bring herself to abandon her chosen career despite a lack of positions in her line of work. Seeing these depressions in terms of unreachable goals may be useful clinically, and may help us better understand how ordinary low moods can escalate into incapacitating bouts of depression.
To be innovative, we have to learn how to detach from failure quickly and move on with the next stage of our ideas with enthusiasm. If Rottenberg’s new hypothesis has merit, to escape depression we have to learn how to detach from failure quickly and move on to our next goals or the next phases of our lives – with enthusiasm.
Can a futuristic mental health intervention increase our personal innovative potential?
My hunch is yes. There are always other fish in the sea.