Quality and Innovation

According to dictionary.com, one of the definitions for rubric is “any established mode of conduct; protocol.” But the context you’ve probably heard this word in is education – where a grading rubric or a scoring rubric is used to evaluate a complex artifact like a student essay.

In my opinion, it’s time to move the concept of the rubric from the classroom into the mainstream, because it can be applied as a very practical general purpose quality tool! (Hear that, Nancy Tague? I think you should write about rubrics in your next edition of the very excellent book The Quality Toolbox. Let me know if you’d like me to help make this happen.)

A rubric is basically a grid with 1) levels of performance indicated along the top row, and 2) criteria or dimensions of performance listed down the leftmost column. Each cell of the grid contains a descriptive statement that explains how the level of performance in that column might be achieved for a specific dimension:

For example, here’s a rubric that one group constructed to evaluate the quality of the mind maps that they were producing. The performance levels are organized from high performance in the top left (smiley face giving a thumbs up) to low performance in the top right (smiley face that looks like he’s about to pass out):

The dimensions of performance are neatness and presentation, use of images/symbols, and use of color. The descriptive statements in each cell provide specific examples of how the performance level might be achieved, e.g. “has failed to include color in the mind map” is an indicator of a low performance level for the dimension of “use of color” – which is very understandable!

The concept of the rubric as a performance assessment tool is relatively new! Griffin (2009), in a brief history of the rubric, notes that since its introduction in 1981, “the scoring rubric has evolved into a more precise, technical, scientific-looking document. It carries a tone of certainty, authority, and exactitude.” However, she notes, the utility of a rubric will depend upon the thought and consideration that goes into its construction. “A rubric is a product of many minds working collaboratively to create new knowledge. It will, almost by definition, be more thoughtful, valid, unbiased and useful than any one of us could have conceived of being as we worked in isolation.”

Advantages of applying a well developed rubric include:

• Provides a common language for sharing expectations and feedback
• Helps to clarify and distinguish the differences between various performance levels
• Helps to focus an individual or group’s ATTENTION on relevant aspects of each desired quality characteristic or skill area
• Provides a mechanism to more easily identify strengths and opportunities for improvement
• Helps lend objectivity to an evaluation process that might otherwise be subjective

Disadvantages:

• Different rubrics may need to be devised for the different activities or artifacts that are to be evaluated using the rubric
• Not all evaluators will apply the rubric in exactly the same way – there is a subjective element at work here – so people may need to be trained in the use of a rubric, or perhaps it would be more effective in a group consensus context where inter-rater variability can be interactively discussed and resolved
• Creating a rubric can be time consuming
• The rubric may limit exploration of solutions or modes of presentation that do not conform to the rubric

Using Rubrics for Quality Improvement

Rubrics are already applied in the world of quality, although I’ve never heard them go by that name. The process scoring guidelines for the Baldrige Criteria are essentially rubrics (although the extra dimension of ADLI and LeTCI has to be considered in the mind of the examiner). The International Team Excellence Award (ITEA) criteria in the Team Excellence Framework (TEF) also forms a rubric in conjunction with the performance levels of missing, unclear, meets expectations or exceeds expectations.

I see a lot of ways in which rubrics can be developed and applied in the quality community to help us establish best practices for some of our most common project artifacts, such as Project Charters. Nancy Tague includes a Project Charter Checklist in The Quality Toolbox to help us create better and more complete charters… but what if we added a second dimension, which includes performance levels, and turned this checklist into a rubric? Any checklist could be transformed into a rubric. Furthermore, to develop a good rubric, we can brainstorm and rank all of the potential criteria in the left hand column, using a Pareto chart to separate the vital few criteria from the trivial many.

Are any of you already using rubrics for purposes outside training or education? I would love to start a list of resources to share with the quality community.

---

Reference: Griffin, M. (2009). What is a rubric? Assessment Update, 21(6), Nov/Dec 2009.

Note: There is a comprehensive site containing many examples of rubrics at http://www.web.virginia.edu/iaas/assess/tools/rubrics.shtm – however, they won’t open in Google Chrome.

I very recently took my ASQ CSSBB exam and passed! Here’s what I think helped me:

1. I studied for about 4 weeks (2 weeks very gently, 1 week much-more-work-because-the-exam-is-getting-closer, and 1 week of panicked, freaked out all nighters) using these great references that I wrote up tons of comments about.

2. I took about 10 pages of really good, concise notes. (I’ll share those with you sometime before the end of the year… want to write them up for public consumption.)

3. I brought about 15 super sharp #2 pencils just in case 14 of them broke. I made sure all the pencils actually SAID #2 on them, so the Scantron machine wouldn’t fail me.

4. I brought my SMART RULER. I’ve had this ruler since the late 1980′s, and every time I’ve taken a tough test, I’ve had my smart ruler with me in case I need to underline anything, or draw dividers between notes. I usually never have to USE the ruler. Usually, its presence is enough to make me do better on any exam.

5. They (the people who say such things) say that peppermint makes you smarter. So I got a new pack of Orbit peppermint gum and chewed it like I had obsessive compulsive disorder for all four hours. (Afterwards I found out that the peppermint thing isn’t really backed up by research, but I didn’t know that going into the exam, so I believed that the peppermint would make my brain work better, and that belief probably helped me out. Got to stack the deck in my favor… didn’t want those 4 weeks of studying NOT to pay off.)

6. When I wasn’t chewing gum, I was nibbling on a Reese’s peanut butter bar. Best 300 calorie investment ever made… the protein made my stomach stop growling so it wouldn’t bother the other test takers.

7. I also brought a couple very cold Diet Cokes, to wash down the peanut butter and the gum taste.

8. To appropriately address my superstitious nature, I wore my Ganesh necklace. In Hindu parlance, Ganesh helps break through obstacles, and I figured the exam that stood between me and CSSBB-hood was definitely an obstacle I wanted broken. (Hey, whatever works, right??)

Nicole

In July, Paul Graham posted an article called “Maker’s Schedule, Manager’s Schedule“. He points out that people who make things, like software engineers and writers, are on a completely different schedule than managers – and that by imposing the manager’s schedule on the developers, there is an associated cost. Makers simply can’t be as productive on the manager’s schedule:

When you’re operating on the maker’s schedule, meetings are a disaster. A single meeting can blow a whole afternoon, by breaking it into two pieces each too small to do anything hard in. Plus you have to remember to go to the meeting. That’s no problem for someone on the manager’s schedule. There’s always something coming on the next hour; the only question is what. But when someone on the maker’s schedule has a meeting, they have to think about it.

For someone on the maker’s schedule, having a meeting is like throwing an exception. It doesn’t merely cause you to switch from one task to another; it changes the mode in which you work.

I find one meeting can sometimes affect a whole day. A meeting commonly blows at least half a day, by breaking up a morning or afternoon. But in addition there’s sometimes a cascading effect. If I know the afternoon is going to be broken up, I’m slightly less likely to start something ambitious in the morning.

This concept really resonated with us – we know about the costs of context switching, but this presented a nice concept for how a developer’s day can be segmented such that ample time is provided for getting things done. As a result, we attempted to apply the concept to achieve more effective communication between technical staff and managers. And in at least one case, it worked extremely well.

Case: Ron DuPlain (@rduplain) and I frequently work together on technical projects. I am the manager; he is the developer. More than we like, we run into problems communicating, but fortunately we are both always on the lookout for strategies to help us communicate better. We decided to apply the “makers vs. managers” concept to meetings, to see whether declaring whether we were having a maker’s meeting or a manager’s meeting prior to the session would improve our ability to communicate with one another.

And it did. We had a very effective maker’s meeting today, for example… explored some technical challenges, worked through a solution space, and talked about possible design options and background information. It was great. As a manager, I got to spend time thinking about a technical problem, but temporarily suspended my attachment to dates, milestones and artifacts. As a developer, Ron got the time and attention from me that he needed to explain his challenges, without the pressure of knowing that I was in a hurry and just needed the bottom line. As a result, Ron felt like I was able to understand the perspectives he was presenting more effectively, and get a better sense of the trade-offs he was exploring.

We had the opportunity to meet on the same terms, all because we declared the intent of our meeting up front in terms of “makers” and “managers”. Thanks Paul – this common language is proving to be a powerful concept for achieving a shared and immediate understanding of technical problems.

Ron DuPlain forwarded me an interesting post from November 2008 (via @duanegran, I believe) called How much do websites cost? It’s a great comprehensive overview of the different kinds of web sites that can be built – the spectrum of customization, interactivity, and intent that dictate whether a web site will cost $200 or $2 million. But what really struck me about this article was one tiny little section that talks about value, emphasizing the relationship between quality, waste, and the changeability of human wants, needs, and desires:

2.) A small company site that has 5 to 10 pages describing the products/services offered by the company. $500 to $2,000 depending on how prepared you are, and also on how clear in your own head you are about what you want. Disorganization and changing your mind are both expensive.

Disorganization is expensive because it blocks action. When your house is disorganized, you waste time and energy trying to find stuff. When the processes you use in the workplace are disorganized, time and physical energy can be wasted engaging in non-value-adding activities, and mental and emotional time and energy wasted in unproductive communications. Wasting time and energy can generate short-term real costs (for example, moving parts or products around a factory or supply chain can delay time-to-market while costing more in fuel for transport), long-term real costs (e.g. reinforcing negative behaviors that lead to breakdowns in interpersonal relationships, teamwork, or morale) or opportunity costs.

Changing your mind is expensive for the same reason: it either blocks new action from taking place, or it eliminates the value that could have been added by prior work. A task is not actionable unless you have the 1) resources to do the job, 2) the information and interest to complete it, 3) the skills and capabilities to make it happen, as well as a clear idea of what needs to be done, and 4) an execution environment that supports getting things done. Changing your mind can erode #2 or #3.

To reduce the risk associated with development, and to control the costs of the project, find out:

• How much do you really know about what you want?
• What essential elements are you pretty sure you’ll still want or need in 3 months, 6 months, 1 year, 5 years?
• What parts do you have the resources, information/interest, capabilities/skills/clarity, and execution environment to get done now? (I call this RICE to make it easier to remember)

The lesson: to get higher quality and lower costs (that is, greater value), focus on those parts of a project that are least likely to change and do those first. This is, of course, if you have the luxury to be agile (highly regulated environments may impose restrictions or limits). Then stop – figure out what your experience working on those parts tells you about how you can approach the problem more systematically and effectively- and repeat the cycle until you iterate to the desired solution. This is the essence of applying organizational learning to your day to day tasks.

I’ve been thinking a lot about management fads lately, and ran into this 2005 article by Nick Carr, titled “Does Not Compute”. Here’s the part that caught my eye:

“A look at the private sector reveals that software debacles are routine. And the more ambitious the project, the higher the odds of disappointment. It may not be much consolation to taxpayers, but the F.B.I. has a lot of company. Software hell is a very crowded place.”

Carr continues by describing two examples of failed projects: a massive systems integration effort at Ford Motor Company, and a overzealous business intelligence initiative embarked upon by McDonald’s. Both projects were cancelled when the price tags got too big: $200M for Ford, $170M for McDonald’s. The catch is that failure is good, because when we fail we at least know one solution path that’s not workable – we just need to 1) understand that it doesn’t have to be expensive, and 2) have more courage to allow ourselves and our colleagues to fail without getting depressed or thinking our coworkers are idiots. This is often expressed as “fail early, fail often“. (But note that the assumption is that you persist, and as a result of the learning experience, ultimately meet your goals.)

Without an effective team culture, rational managers, healthy relationships with stakeholders, and capable programmers dedicated to continually improving their skills, all roads can lead to software hell. The process of getting there – which is hellish in and of itself – is the famed death march. This is where a software-related project, doomed to fail, sucks up more time, people, resources, and emotional energy at an ever increasing rate until the eventual cataclysm.

Carr also cites The Standish Report, which in 1994, asserted that only 16% of projects were completed on time, and budget, and meeting specifications. By 2003 the percentage had grown to 34% in a new survey. Other projects that were still completed ran, on average, 50 percent over budget. (And this is for the survey respondents who were actually telling the truth. I know a few people who wouldn’t admit that their project was quite so grossly over budget.)

One way to solve this problem is by focusing on sufficiency and continuous learning, starting the blueprint for a system based on these questions:

• What features represent the bare minimum we need to run this system?
• What are the really critical success factors?
• What do we know about our specifications now? What do we not know?
• What do we know about ourselves now? What do we want to learn more about?

Software development is a learning process. It’s a process of learning about the problem we need to solve, the problem domain, and ourselves – our interests and capabilities. It’s a process of recognizing what parts of building the solution we’re really good at, and what parts we’re not so good at. Let’s start small, and grow bigger as we form stronger relationships with the systems that we are developing. Having a $170M appetite sure didn’t get McDonald’s anywhere, at least in this case.

Technology is the “sum of ways in which social groups construct the material objects of their civilizations.” The things that we use – the “design artifacts” of the processes used to build them – are socially constructed to the same extent that they are technically constructed. The convergence of technological and social insights in the creation, construction and use of artifacts is sociotechnology.

For example, we typically build a bridge when there’s some expectation that people need to get from Point A to Point B, and there’s something they need to bypass along the way (e.g. a river, a canyon, another road). Failure to consider the social factors as well as the technical factors could lead to a “bridge to nowhere” – and we all know at least one person who’s had a problem with those. Non-technical factors pertaining to the environment in which an idea is created and implemented are crucial.

According to Bunge (1998), sociotechnology is the process of applying insights from the social sciences to design policies and programs. More specifically, this is how Gingras & Niosi (1990) explain Bunge’s perspective:

Ten years ago we were talking about the convergence of customer touch points: phone, fax, web, cell phones, email and regular mail. With handhelds and mobile devices becoming more and more ubiquitous, and services like Facebook becoming more integrated into our daily lives, the next convergence is between people and the technologies we use. The boundaries are becoming increasingly blurred, and the impact of this convergence on business must be explored. Reviewing research by pioneers like Tom Erickson is a good place to start. We are all becoming sociotechnical.

(Note: Sociotechnology is an important part of socio-technical design.)

---

Bunge, M. (1985). Philosophy of science and technology. Vol. 7 of Treatise on basic philosophy, Dordrecht: Holland.
Bunge, M. (1998), Social Science under debate. A Philosophical Approach. Toronto University Press: Toronto.
Gingras, Y. & Niosi, J. (1990). Technology and society: a view from sociology, in Georg Dorn and Paul Weintgartner (eds.) Studies on Mario Bunge’s Treatise, Poznan Studies in the Philosophy of Science, Amsterdam and Atlanta, 421-430. Retrieved from http://www.archipel.uqam.ca/506/01/On_Bunge.PDF
Nieto, C. C., Neotropica, F., & Durbin, P. T. (1995). Sustainable development and philosophies of technology. Society for Philosophy and Technology, Vol. 1, Fall 1995. Retrieved from http://scholar.lib.vt.edu/ejournals/SPT/v1n1n2/nieto.html – (Note: I added this one simply because I really like it, and it’s related to the discussion on sociotechnology.)

Software is the executable representation of knowledge. [1] As a result, I find that software development provides a fruitful basis for exploring how problem solving is done by diverse team members in a cooperative (or even combative!) context.

Here is one example. In June 1997, Tom Gilb wrote an article for Crosstalk on “Requirements-Driven Management”. He noted that his purpose was intended to discuss, among other things, “some of the current major problems in systems engineering.” I stumbled upon this article again over the weekend, and it’s still as relevant now as it was a decade ago.

Standards are a prerequisite for systematic continuous improvement; the means to achieve an improvement process, not the ends, according to Glib. Furthermore, he remarks that Deming’s perspective on continuous improvement establishes, in part, that the reason we should adopt standards is to normalize our project to other projects . Once we do that, we’ve opened the doors to be able to use industry best practices, derived from other projects who also applied the standards – so that we can “clearly see the effect of any changes experimentally introduced into a process and not have to worry too much about other potential factors that impact the results.”

Learning, learning, learning. It’s all about continuous improvement through continuous learning, and in presenting this, Gilb is essentially promoting the same philosophy as Alistair Cockburn’s Cooperative Game Manifesto for Software Development. Both see the learning process as the key to successful software development. (So why do we not focus on this aspect of development more?) Glib addresses the issue of learning directly:

“The time has come to recognize that projects are so large, so complex, so unpredictable, and so state-of-the-art new that we have no practical alternative except to maximize our learning process during the project and as early as possible in the project life. “

In this article, Gilb also presents “Evo” – short for “Evolutionary Project Management” – which has been used at IBM since 1970. It’s an implementation of Deming’s PDCA (Plan-Do-Check-Act) approach, and the author likens it to Humphrey’s Personal Software Process (PSP).

---

[1] This definition is credited to Eric Sessoms, who I consider a true artisan of software development. See some of his work at libraryh3lp.com. He likes the elegance of simplicity in his software.