Quality and Innovation

On July 8th, Wall Street’s software failed (and the WSJ web site went down). United’s planes were grounded for two hours across the entire US. And this all happened only shortly after China’s stocks mysteriously plummeted. Odd coincidence, or carefully planned coordinated cyberattack? Bloggers say don’t worry… don’t panic. Probably not a big deal overall:

Heck, the whole United fleet was grounded last month too… NYSE is one stock exchange among many. The website of a newspaper isn’t important, and the Chinese stocks are volatile… we should not worry that this is a coordinated attack, especially of the dreaded “cyber-terrorist” kind…

The big problem we face isn’t coordinated cyber-terrorism, it’s that software sucks. Software sucks for many reasons, all of which go deep, are entangled, and expensive to fix. (Or, everything is broken, eventually). This is a major headache, and a real worry as software eats more and more of the world.

In a large and complex system, something will ALWAYS be broken. Our job is to make sure we don’t let the wrong pieces get broken and stay broken… and we need to make sure our funding, our policies, and our quality systems reflect this priority.

Once upon a time in the early 2000’s, I worked as a technology manager at a great big telescope called the GBT (not an acronym for great big, but rather Green Bank… where it’s located in West Virginia).

It cost a lot to maintain and operate that telescope… nearly $10M every year. About 10-15% of this budget was spent on software development. Behind all great hardware and instrumentation, there’s great (or at least functional) software that helps you accomplish whatever goals and objectives you have that require the hardware. Even though we had to push forward and work on new capabilities to keep our telescope relevant to the scientists who used it to uncover new knowledge about the universe, we had to continue maintaining the old software… or the whole telescope might malfunction.

It’s not popular to keep putting money into maintenance at the expense of funding innovation. But it’s necessary:

• Without spending time and money to continuously firm up our legacy systems, we’re increasing the likelihood that they will crash (all on their own), producing devastating impacts (either individually or collectively).
• Without spending time and money to continuously firm up our legacy systems, we’re also increasing the possibility that some rogue hacker (or completely legitimate domestic or foreign enemy) will be able to trigger some form of devastation that impacts the safety, security, or well-being of many people.

When we choose to support innovation at the expense of regular maintenance and continuous improvement, we’re terrorizing our future selves. Especially if our work involves maintaining software that connects or influences people and their devices. Isn’t that right, Amtrak?