Tag Archives: Measurement

The Rubric as a General Purpose Quality Tool

According to dictionary.com, one of the definitions for rubric is “any established mode of conduct; protocol.” But the context you’ve probably heard this word in is education – where a grading rubric or a scoring rubric is used to evaluate a complex artifact like a student essay.

In my opinion, it’s time to move the concept of the rubric from the classroom into the mainstream, because it can be applied as a very practical general purpose quality tool! (Hear that, Nancy Tague? I think you should write about rubrics in your next edition of the very excellent book The Quality Toolbox. Let me know if you’d like me to help make this happen.)

A rubric is basically a grid with 1) levels of performance indicated along the top row, and 2) criteria or dimensions of performance listed down the leftmost column. Each cell of the grid contains a descriptive statement that explains how the level of performance in that column might be achieved for a specific dimension:

For example, here’s a rubric that one group constructed to evaluate the quality of the mind maps that they were producing. The performance levels are organized from high performance in the top left (smiley face giving a thumbs up) to low performance in the top right (smiley face that looks like he’s about to pass out):

The dimensions of performance are neatness and presentation, use of images/symbols, and use of color. The descriptive statements in each cell provide specific examples of how the performance level might be achieved, e.g. “has failed to include color in the mind map” is an indicator of a low performance level for the dimension of “use of color” – which is very understandable!

The concept of the rubric as a performance assessment tool is relatively new! Griffin (2009), in a brief history of the rubric, notes that since its introduction in 1981, “the scoring rubric has evolved into a more precise, technical, scientific-looking document. It carries a tone of certainty, authority, and exactitude.” However, she notes, the utility of a rubric will depend upon the thought and consideration that goes into its construction. “A rubric is a product of many minds working collaboratively to create new knowledge. It will, almost by definition, be more thoughtful, valid, unbiased and useful than any one of us could have conceived of being as we worked in isolation.”

Advantages of applying a well developed rubric include:

  • Provides a common language for sharing expectations and feedback
  • Helps to clarify and distinguish the differences between various performance levels
  • Helps to focus an individual or group’s ATTENTION on relevant aspects of each desired quality characteristic or skill area
  • Provides a mechanism to more easily identify strengths and opportunities for improvement
  • Helps lend objectivity to an evaluation process that might otherwise be subjective

Disadvantages:

  • Different rubrics may need to be devised for the different activities or artifacts that are to be evaluated using the rubric
  • Not all evaluators will apply the rubric in exactly the same way – there is a subjective element at work here – so people may need to be trained in the use of a rubric, or perhaps it would be more effective in a group consensus context where inter-rater variability can be interactively discussed and resolved
  • Creating a rubric can be time consuming
  • The rubric may limit exploration of solutions or modes of presentation that do not conform to the rubric

Using Rubrics for Quality Improvement

Rubrics are already applied in the world of quality, although I’ve never heard them go by that name. The process scoring guidelines for the Baldrige Criteria are essentially rubrics (although the extra dimension of ADLI and LeTCI has to be considered in the mind of the examiner). The International Team Excellence Award (ITEA) criteria in the Team Excellence Framework (TEF) also forms a rubric in conjunction with the performance levels of missing, unclear, meets expectations or exceeds expectations.

I see a lot of ways in which rubrics can be developed and applied in the quality community to help us establish best practices for some of our most common project artifacts, such as Project Charters. Nancy Tague includes a Project Charter Checklist in The Quality Toolbox to help us create better and more complete charters… but what if we added a second dimension, which includes performance levels, and turned this checklist into a rubric? Any checklist could be transformed into a rubric. Furthermore, to develop a good rubric, we can brainstorm and rank all of the potential criteria in the left hand column, using a Pareto chart to separate the vital few criteria from the trivial many.

Are any of you already using rubrics for purposes outside training or education? I would love to start a list of resources to share with the quality community.


Reference: Griffin, M. (2009). What is a rubric? Assessment Update, 21(6), Nov/Dec 2009.

Note: There is a comprehensive site containing many examples of rubrics at http://www.web.virginia.edu/iaas/assess/tools/rubrics.shtm – however, they won’t open in Google Chrome.

The ITEA Criteria for Software Process & Performance Improvement

(I originally wrote this article for the ASQ Software Division Newsletter compiled in the first quarter of 2009. I’m reproducing it here because I’ve found the ITEA criteria to be remarkably useful for all kinds of planning since I was introduced to it last year.)

frangipani-flowersFor software professionals, particularly those of us who manage product development or development teams, it is important to track progress towards our goals and to justify the results of our efforts. We have to write effective project charters for software development just to get things moving, evaluate improvement alternatives before making an investment of time and effort in a process change, and ultimately validate the effectiveness of what we have implemented.

This past fall, I had the opportunity to serve as a preliminary round judge for the ASQ International Team Excellence Award (ITEA). My subgroup of judges met at the Bank of America training facility in Charlotte, North Carolina, where we split up into teams to evaluate almost 20 project portfolios. A handful of other events just like ours were held at the same time across the country, giving many people the opportunity to train and serve as judges. Before we evaluated the portfolios, we were all trained on how to use and understand the ITEA criteria, a 37-point system for assessing how well a project had established and managed to its own internal quality system. The ITEA criteria can be applied to any development project or process improvement initiative in the same way that the Baldrige criteria might be applied to an organization‘s strategic efforts. For software, this might include improving the internal processes of a software development team, using software improvements and automation to streamline a production or service process, and improving the performance or quality of a software product. (For example, I can envision the ITEA criteria being used to evaluate the benefits of parallelizing all or part of a software system to achieve a tenfold or hundredfold performance improvement.)

You can review these criteria on the web at http://wcqi.asq.org/2008/pdf/criteria-detail.pdf yourself. There are five main categories in the ITEA criteria: project selection and purpose, the current situation (prior to improvement), solution development (and evaluation of alternatives), project implementation and results, and team management and project presentation. An important distinction is in the use of the words Identify/Indicate, Describe and Explain within the criteria. To identify or indicate means that you have enumerated the results of brainstorming or analysis, which can often be achieved using a simple list of bullet points. To describe means that you have explained what you mean by each of these points. To explain means that you have fully discussed not only the subject addressed by one of the 37 points, but also your rationale for whatever decisions were made. Sustainability of the improvements that a project makes is also a major component of the ITEA criteria. Once your project is complete, how will you ensure that the benefits you provided are continued? How can you make sure that a new process you developed will actually be followed? Do you have the resources and capabilities to maintain the new state of the system and/or process?

The ITEA criteria can serve as a useful checklist to make sure you‘ve covered all of the bases for your software development or process improvement project. I encourage you to review the criteria and see how they can be useful to your work.

Google Measures Energy to Conserve Energy

Why measure? Because measurement compels behavior. I’ve written about this previously in my article on the Trash Guy, but now Google is taking note:

”Studies show that being able to see your energy usage makes it easier to reduce it.”

This is the driver for their new Google PowerMeter project, which envisions a future where access to energy informatics is through your desktop. The project, an initiative of Google.org (the philanthropic research arm of Google), provides this as their pitch:

London to Brighton Veteran Car Run“How much does it cost to leave your TV on all day? What about turning your air conditioning 1 degree cooler? Which uses more power every month — your fridge or your dishwasher? Is your household more or less energy efficient than similar homes in your neighborhood? … At Google we’re committed to helping enable a future where access to personal energy information helps everyone make smarter energy choices. To get started, we’re working on a tool called Google PowerMeter which will show consumers their electricity consumption in near real-time in a secure iGoogle Gadget. We think PowerMeter will offer more useful and actionable feedback than complicated monthly paper bills that provide little detail on consumption or how to save energy.”

I like it. I’ve always wanted to have a simple way to monitor my home energy usage that doesn’t require me to buy an expensive device like the Black & Decker EM100B Energy Saver Series Power Monitor– that probably doesn’t give me the granularity of information I’m looking for anyway.

Quality Metrics for Policy Evaluation?

The Center for Environmental Journalism (CEJ) recently posted an interview with Roger Pielke, Jr., an authority on (as CEJ calls it) “the nexus of science and technology in decision making”. The interview seeks to provide a perspective on how journalists can more accurately address climate change in the context of public policy over the next several years.

I was really intrigued by this part:

Reporters could help clarify understandings by asking climate scientists: “What behavior of the climate system over the next 5-10 years would cause you to question the IPCC consensus?” This would give people some metrics against which to evaluate future behavior as it evolves.

Similarly, you could ask partisans in the political debate “What science would cause you to change your political position on the issue?” This would allow people to judge how much dependence partisans put on science and what science would change their views. I would be surprised if many people would give a concrete answer to this!!

For the first question, Pielke is recommending is that we take an approach conceptually resembling statistical process control to help us figure out how to evaluate the magnitude and potential impacts of climate change. (Could we actually apply such techniques? It would be an interesting research question. Makes me think of studies like Khoo & Ariffin (2006), for example, who propose one method based on Shewhart x-bar charts to detect process shifts with a higher level of sensitivity – only tuned for a particular policy problem.) For the second question, I’m reminded of “willingness to pay” or “willingness to recommend” or other related marketing metrics. I’m sure that one of these established approaches could be extended to the policy domain (if it hasn’t been done already).

Inspection, Abstraction and Shipping Containers

On my drive home tonight, a giant “Maersk Sealand” branded truck passed me on the highway. It got me thinking about how introducing a standard size and container shape revolutionized the shipping industry and enabled a growing global economy. At least that’s the perspective presented by Mark Levinson in The Box: How the Shipping Container Made the World Smaller and the World Economy Bigger.

A synopsis of the story and a sample chapter are available; Wikipedia’s entry on containerization also presents a narrative describing the development and its impacts.

Here’s how impactlab.com describes it:

Indeed, it is hard to imagine how world trade could have grown so fast—quintupling in the last two decades—without the “intermodal shipping container,” to use the technical term. The invention of a standard-size steel box that can be easily moved from a truck to a ship to a railroad car, without ever passing through human hands, cut down on the work and vastly increased the speed of shipping. It represented an entirely new system, not just a new product. The dark side is that these steel containers are by definition black boxes, invisible to casual inspection, and the more of them authorities open for inspection, the more they undermine the smooth functioning of the system.

Although some people like to debate whether shipping containers were an incremental improvement or a breakthrough innovation, I’d like to note that a single process improvement step generated a multitude of benefits because the inspection step was eliminated. Inspection happened naturally the old way, without planning it explicitly; workers had to unpack all the boxes and crates from one truck and load them onto another truck, or a ship. It would be difficult to overlook a nuclear warhead or a few tons of pot.

To make the system work, the concept of what was being transported was abstracted away from the problem, making the shipping container a black box. If all parties are trustworthy and not using the system for a purpose other than what was intended, this is no problem. But once people start using the system for unintended purposes, everything changes.

This reflects what happens in software development as well: you code an application, abstracting away the complex aspects of the problem and attaching unit tests to those nuggets. You don’t have to inspect the code within the nuggets because either you’ve already fully tested them, or you don’t care – and either way, you don’t expect what’s in the nugget to change. Similarly, the shipping industry did not plan that the containers would be used to ship illegal cargo – that wasn’t one of the expectations of what could be within the black box. The lesson (to me)? Degree of abstraction within a system, and the level of inspection of a system, are related. When your expectations of what constitutes your components changes, you need to revisit whether you need inspection (and how much).

What is an Environmental Analysis?

An environmental analysis (or environmental assessment) is a decision-making tool, often applied in technology management to characterize the forces impacting an emerging technology or a new or existing product. The environmental analysis can help you determine the effects of a proposed project or policy, and to proactively assess the impacts of a developing or emerging product or discipline. An environmental analysis also provides a really useful structure for learning about an area or a theme new to you or your company and identifying what the “state of the art” is (e.g. petascale computing, nanotechnology, innovative composite materials).

To conduct an environmental analysis, you should investigate and outline:

  • CONTEXT. The technology of interest and the context in which it is/to be used
  • CHALLENGES. The challenges that are presently identifiable; what you know, and how it compares and contrasts with the unknowns
  • EXTERNAL ENVIRONMENT. How the competitive environment impacts the scenario. This can be done via SWOT analysis (strengths, weaknesses, opportunities, threats) and/or by examining Porter’s (1980) Five Forces (supplier power, barriers to entry, threat of substitutes, buyer power, degree of rivalry)
  • INTERNAL ENVIRONMENT. How themes influence and affect the scenario (e.g. via PEST analysis – political, economic, socio-cultural, technological impacts)
  • ALTERNATIVES. Examine alternatives to the scenario being evaluated, and investigate what criteria (e.g. values, beliefs, project constraints, technical constraints) might be used if you will choose between competing alternatives in the future

Where can you get data for an environmental analysis? In addition to searching through resources from newspapers, magazines and trade journals, check the following:

Organization for Economic Cooperation and Development (OECD)

  • The OECD statistics portal contains international databases on agriculture, education, development, finance, labor, science and technology, energy, globalization, productivity, welfare, transport
  • Their online library also contains environmental outlooks, news on economic policy reforms, and issues like work/life balance

World Economic Forum Global Competitiveness Report

  • In the Growth Competitiveness Index (GCI) issued by the World Economic Forum, which is measured for more a hundred countries every year, there are four dimensions of global competitiveness routinely assessed: institutions, infrastructure, the macroeconomic environment, and health and education.
  • Because technology has the potential to impact productivity at many levels, and because it is embedded in each of these areas, the effects of technological change are implicit in macroeconomic measures of competitiveness.
  • You can learn more about the Global Competitiveness Report on Wikipedia
  • Or use the Analyzer to explore the data

National Science Foundation Solicitations for Research Proposals – The NSF solicitations are an excellent place to learn about the state of the art in various fields. The solicitations explain what topics are the most interesting to the experts today, and what they are willing to pay to know more about. Often, the solicitations will explain the most recent trends that may be difficult to ascertain from the industry and academic literature.

Google Tracks Spread of Flu

google-orgIs the flu spreading across your state? You can find out using Google Flu Trends, which projects the spread of influenza based on how people are using Google to search for health information. Check out the movie illustrating how search data appears to correlate with flu data from the Center for Disease Control.

The reason this interests me is that Google is using a tracer – examining search patterns in terms of where the searches are originating from geographically to infer how diseases might be spreading. They are not tracking diagnosis information or other “hard” data which would affirm the presence of disease, only recognizing that people will tend to be more interested in the flu when they’re trying to figure out whether they have it! (The most useful aspect of the search data is that it appears to serve as a leading indicator for the CDC data, which has a two week lag.)

Are any companies out there using patterns in their Google searches on their websites to infer what consumers or constituents are most interested in at any given time? It would be interesting to see what other “real” things Google search data can serve as a leading indicator for. I could see this as a useful technique for diagnosing the “voice of the customer” in a novel way.

« Older Entries Recent Entries »