Quality and Innovation

(I originally wrote this article for the ASQ Software Division Newsletter compiled in the first quarter of 2009. I’m reproducing it here because I’ve found the ITEA criteria to be remarkably useful for all kinds of planning since I was introduced to it last year.)

For software professionals, particularly those of us who manage product development or development teams, it is important to track progress towards our goals and to justify the results of our efforts. We have to write effective project charters for software development just to get things moving, evaluate improvement alternatives before making an investment of time and effort in a process change, and ultimately validate the effectiveness of what we have implemented.

This past fall, I had the opportunity to serve as a preliminary round judge for the ASQ International Team Excellence Award (ITEA). My subgroup of judges met at the Bank of America training facility in Charlotte, North Carolina, where we split up into teams to evaluate almost 20 project portfolios. A handful of other events just like ours were held at the same time across the country, giving many people the opportunity to train and serve as judges. Before we evaluated the portfolios, we were all trained on how to use and understand the ITEA criteria, a 37-point system for assessing how well a project had established and managed to its own internal quality system. The ITEA criteria can be applied to any development project or process improvement initiative in the same way that the Baldrige criteria might be applied to an organization‘s strategic efforts. For software, this might include improving the internal processes of a software development team, using software improvements and automation to streamline a production or service process, and improving the performance or quality of a software product. (For example, I can envision the ITEA criteria being used to evaluate the benefits of parallelizing all or part of a software system to achieve a tenfold or hundredfold performance improvement.)

You can review these criteria on the web at http://wcqi.asq.org/2008/pdf/criteria-detail.pdf yourself. There are five main categories in the ITEA criteria: project selection and purpose, the current situation (prior to improvement), solution development (and evaluation of alternatives), project implementation and results, and team management and project presentation. An important distinction is in the use of the words Identify/Indicate, Describe and Explain within the criteria. To identify or indicate means that you have enumerated the results of brainstorming or analysis, which can often be achieved using a simple list of bullet points. To describe means that you have explained what you mean by each of these points. To explain means that you have fully discussed not only the subject addressed by one of the 37 points, but also your rationale for whatever decisions were made. Sustainability of the improvements that a project makes is also a major component of the ITEA criteria. Once your project is complete, how will you ensure that the benefits you provided are continued? How can you make sure that a new process you developed will actually be followed? Do you have the resources and capabilities to maintain the new state of the system and/or process?

The ITEA criteria can serve as a useful checklist to make sure you‘ve covered all of the bases for your software development or process improvement project. I encourage you to review the criteria and see how they can be useful to your work.

How you perceive quality is influenced by your expectations. And sometimes, your expectations are subconscious or emotionally driven.

For example, a product may have all the features you, as a consumer, could possibly want and need – and it might perform well too! But it still might not satisfy everyone, or generate the magnitude of sales that were originally projected. How could this be?

Understanding the psychology of quality and value, based on affect, provides insight into how this can happen. Merriam Webster’s Medical Dictionary defines affect as “the conscious subjective aspect of an emotion considered apart from bodily changes.” In short, affect describes how something makes you feel. For example, working on a task that you really enjoy promotes positive affect. Spending time with “de-energizers” who are negative, critical, and generally unhappy can create negative affect.

Research in psychology indicates that positive affect corresponds with the ability to solve problems more readily and effectively, while negative affect can impede problem solving, even for simple tasks. As a result, usability can be considered a function of the positive or negative affect that is generated when a user interacts with a product. This applies to all products, including software and web-based applications.

These studies also suggest that effective design translates to positive affect – meaning that before use, perceived quality and perceived value are more closely related to the perceived quality and value that will be experienced after use. Aesthetics thus play a role in promoting positive affect. As interpreted by Don Norman (2004) in Emotional Design, where many of the aforementioned studies are referenced,

“the emotional system changes how the cognitive system operates… [it is] easier for people to find solutions to the problems they encounter… [there is a] tendency to repeat the same operation over again is especially likely for those who are anxious or tense.”

An entertaining example is the ATM case, which I’ll write about tomorrow.

I’ve been thinking a lot about management fads lately, and ran into this 2005 article by Nick Carr, titled “Does Not Compute”. Here’s the part that caught my eye:

“A look at the private sector reveals that software debacles are routine. And the more ambitious the project, the higher the odds of disappointment. It may not be much consolation to taxpayers, but the F.B.I. has a lot of company. Software hell is a very crowded place.”

Carr continues by describing two examples of failed projects: a massive systems integration effort at Ford Motor Company, and a overzealous business intelligence initiative embarked upon by McDonald’s. Both projects were cancelled when the price tags got too big: $200M for Ford, $170M for McDonald’s. The catch is that failure is good, because when we fail we at least know one solution path that’s not workable – we just need to 1) understand that it doesn’t have to be expensive, and 2) have more courage to allow ourselves and our colleagues to fail without getting depressed or thinking our coworkers are idiots. This is often expressed as “fail early, fail often“. (But note that the assumption is that you persist, and as a result of the learning experience, ultimately meet your goals.)

Without an effective team culture, rational managers, healthy relationships with stakeholders, and capable programmers dedicated to continually improving their skills, all roads can lead to software hell. The process of getting there – which is hellish in and of itself – is the famed death march. This is where a software-related project, doomed to fail, sucks up more time, people, resources, and emotional energy at an ever increasing rate until the eventual cataclysm.

Carr also cites The Standish Report, which in 1994, asserted that only 16% of projects were completed on time, and budget, and meeting specifications. By 2003 the percentage had grown to 34% in a new survey. Other projects that were still completed ran, on average, 50 percent over budget. (And this is for the survey respondents who were actually telling the truth. I know a few people who wouldn’t admit that their project was quite so grossly over budget.)

One way to solve this problem is by focusing on sufficiency and continuous learning, starting the blueprint for a system based on these questions:

• What features represent the bare minimum we need to run this system?
• What are the really critical success factors?
• What do we know about our specifications now? What do we not know?
• What do we know about ourselves now? What do we want to learn more about?

Software development is a learning process. It’s a process of learning about the problem we need to solve, the problem domain, and ourselves – our interests and capabilities. It’s a process of recognizing what parts of building the solution we’re really good at, and what parts we’re not so good at. Let’s start small, and grow bigger as we form stronger relationships with the systems that we are developing. Having a $170M appetite sure didn’t get McDonald’s anywhere, at least in this case.

ISO 9241-11 defines usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.” The four elements that define usability within this context are as follows:

• both the users and goals must be explicitly identified,
• the intended context of use must be identified and understood, and
• the user can use the system in question
• to meet those stated goals.

These same four elements are implied by the ISO 8402 definition of quality: stated and implied needs are relative to specific users with specific goals, are dependent upon a context of use, and the entity in question is the system being defined and developed in response.

Usability is the extent, or the degree, to which the above criteria are satisfied. Here’s an example from software development to make this a little more concrete. The software development lifecycle, regardless of what incarnation you’re using (even waterfall),  inherently addresses usability through these four elements:

• the requirements process outlines the specified users, their goals, and the context of use
• the design process defines a specific technical solution to meet those needs, and
• the finished product provides evidence that the system can be used to achieve the goals of its users.

As a result, usability can be considered an implicit factor in software quality, ultimately reflecting how well the design process interpreted requirements within a specified context of use.

This morning for breakfast, I chose the “nutritious” option of a slice of buttered rye. After the obligatory ninety seconds, my freshly toasted bread popped out of the toaster. It was hot, with a little steam coming off the sides – pretty tantalizing for a frigid winter’s morning. But I had forgotten to take the butter out of the refrigerator earlier – arrhgrhh, I thought, now I’m going to have to cut cold butter and try to get it to spread on my rapidly cooling toast.

I hate cutting cold butter. The butter itself is unwieldy – it tends to act like a magnet (at least for me), consistently tipped the wrong way, repelling the knife at every attempt. It’s an altogether unpleasant experience, certainly not a complement to a rushed morning where you’re trying to get yourself ready for work and kids ready for school.

But — microwave that butter for about 15 seconds and everything changes. All of a sudden, the butter is at your command. The knife slices through effortlessly, like an airy ballet. The pat of butter, liberated from its stick, conforms to the shape of your toast with only a few strokes.

It’s my opinion that the software you use should feel the same as the experience of cutting the warm butter. But all too often, software feels far more like the cold butter. You try to take a decisive step with the tool, but the application jumps out from under your control. It falls on the ground, sometimes things get dirty, and it can become a struggle to work with the tool thereafter. That is, until you’ve had a while to adjust to the software, and it’s had time to adjust to room temperature and become more malleable. Just as the butter eventually adjusts to a temperate environment, with extended use, a user will adapt to software and be able to work with it (not always due to changes in the software, but to changes in the human-software sociotechnical system nonetheless).

Enter the Butter Test. The Butter Test is the equivalent of the “5-Second” test for user interface navigability – but for software applications, web pages, web applications, APIs, or any other software-related design. (There’s even a web app that helps you conduct a formal 5-Second test.) Whereas the 5-Second test gauges your user’s first impressions when they visit your web page, the Butter Test assesses how malleable the software is upon a user’s first encounter. To do the Butter Test, spend about five minutes with a new application. You don’t have to be alone; you can get a walkthrough from someone who’s more familiar with it. How does it feel? Do you feel like you’re struggling with a cold stick of butter? Does the software respond in a jagged, unpredictable way – forcing you to catch it before it falls on the floor? Or alternatively, is your first cut at using the software smooth? Do the results feel trustworthy, interpretable, and extendable (meaning you’re left feeling empowered to do more)? Using the Butter Test, your first impressions count.

The Butter Test is not just useful for subjectively evaluating full software applications. Just today, I used it to determine whether a taxonomy for a directory structure made sense. We needed a file structure for holding different types of data (with different levels of “ease to reproduce”) from different instruments. After being walked through the new structure, I could immediately figure out where new data would go, how we would adapt to novel data types and processed data products, and how to access the data without an a priori knowledge of the full directory structure. By learning a few rules, I could work easily with the entire collection of data – and I learned all this in less than five minutes. The new taxonomy passed the Butter Test with flying colors.

I have been using the Butter Test for about 15 years and it does not disappoint. Trust your instincts. If your software was toast and butter, would you be content or frustrated?

If you’re a software developer, software reuse is kind of like the Holy Grail.
You probably think it’s a good idea already. Plus, managers always want more of it.

But achieving software reuse in practice can be difficult. It’s one thing if you can use really stable, trusted code libraries and treat them as black boxes in your code. Unfortunately this only works if those black boxes don’t run into problems, even when you deploy them on hardware that’s different than what the original writers used. (Good luck.)

Here’s why I think software reuse is difficult.

First, we need to note the difference between explicit knowledge and tacit knowledge. Explicit knowledge is what you get from books, or memos, or hearing a story or experience that someone else shares with you. Tacit knowledge is what you get from experimentation, mentorship, or exploration. (For example, I have no explicit knowledge of the vi editor – but I have extensive tacit knowledge of it, because when I open the editor my fingers know what to do, darting back and forth between ESC and : and other punctuation marks that my brain is not consciously processing. Perl is kind of the same way for me.)

If software is the executable representation of knowledge, then developing software is the process of codifying (literally!) tacit and explicit knowledge to make it executable. That’s a huge challenge that requires the developer to learn, reflect, experiment, learn, reflect, experiment… and so on. It’s a process of learning that ultimately results in software – a tool that hopefully will earn a life of its own, perhaps extending beyond the time that the original developer works on it.

What do you get by reading and exploring someone else’s code? Explicit knowledge. What do you get by writing code yourself, by uncovering each new function one at a time, by realizing you didn’t know what you were doing the first time, by refactoring, by trying out a new design (or maybe two or three), by bouncing ideas off of your colleagues, or by going back to the first design you had a long time ago? That’s tacit knowledge. You need both to write good software.

Unless there’s some way to get that tacit knowledge of someone else’s code, or you don’t need the tacit knowledge anyway (ie. in the case of the “black box”), successful reuse will be challenging… if not impossible.

(Related Post –> Software Reuse Antipatterns)

In 2000, Scott Ambler wrote  an excellent article on the organizational aspects of software reuse. He talked specifically about patterns and antipatterns:

“A pattern is a common approach to solving a problem that is proven to
work in practice. Conversely, an antipattern is a common approach to solving a problem that leaves us worse off than when we started.”

Long (2001) built on Ambler’s work and made it more fun. (This is, in my opinion, one of the most interesting and entertaining articles about software reuse in existence. You can get the full text from the ACM if you have an account there.) Long calls antipatterns obvious, but wrong, solutions to recurring problems and characterizes four organizational approaches that don’t support successful software reuse. See if your organization is one of these:

• Abracadabra Model: A high-level manager is frustrated with a perceived lack of reuse and declares that “reuse will happen”. What Happens: Lots of talk, no action, silo development continues, managers start to panic, then the organization “de-evolves” into the next model.
• High Noon Model: A high-level manager is REALLY frustrated with a perceived lack of reuse and declares that “reuse will happen”. What Happens: Finger pointing, as everybody has a lot of reasons (many of them very good, and very accurate) about why reuse can’t possibly work. The de-evolution continues.
• Cost Cutter Model: A high-level manager is REALLY, REALLY frustrated with a perceived lack of reuse and declares that “reuse must happen to cut costs”. What Happens: Software people start to “force” reuse, immediate costs go up, upper management gets nervous, and more finger pointing happens (as everybody finds even more reasons now – including higher cost – for why reuse can’t possibly work.)
• Used Car Fiasco Model: Software group says “OK, we’ll try reuse.” One group has software it thinks about group can use, so it is made available as-is and with no support. What Happens: There are lots of bugs to fix. Reusers have to fix them because the originators don’t have the time or resources to solve the new group’s problems. The reusers get frustrated and then write the code themselves.

Note that in all models, the expectations and behavior of the managers doesn’t change. In the fourth model, the behavior of the software developers changes. At no point do the expectations of the software developers change – their mission is to do what they need to do to get the software written.

Tomorrow I’ll write about why I think software reuse is difficult. The antipatterns above provide a good foundation for that discussion.

---

Ambler, S. (2000). Reuse Patterns and Antipatterns. Dr. Dobbs Journal, February 1, 2000.
Long, J. (2001). Software Reuse Antipatterns. ACM SIGSOFT, Software Engineering Notes, 26(3), 68-76.