Back in September 2007, Toyota Camrys began unexpectedly and unstoppably accelerating. Cars crashed. People were killed. Toyota put the blame on the physical causes like floormats and sticky accelerator pedals and even bad drivers.
Gradually the finger of blame began to point towards the software.
During the 2013 trial of Jean Bookout (unintended acceleration victim) vs Toyota Motor Corporation, expert witness Michael Barr revealed the cause of the deadly problem with the Camry’s software. After 18 months of investigation with his team of experts poring over Toyota’s code, Barr determined that the problem was... spaghetti code!
9 million cars recalled. $3 billion in legal settlements. 57 injuries. 89 deaths. All of it due to "spaghetti code." Untestable, unmaintainable spaghetti code.
Here are some quotes from Mr. Barr’s testimony.
The metrics, like the code complexity and a number of global variables, indicate the presence of large numbers of bugs. And just the overall style of the code is suggestive that there will be numerous more bugs that we haven't found yet.
Dirty stinkin' code leads to hidden dirty stinkin' bugs.
And the code complexity for Toyota's code is very high. There are a large number of functions that are overly complex. By the standard industry metrics some of them are untestable, meaning that it is so complicated a recipe that there is no way to develop a reliable test suite or test methodology to test all the possible things that can happen in it. Some of them are even so complex that they are what is called unmaintainable, which means that if you go in to fix a bug or to make a change, you're likely to create a new bug in the process.
Untestable and unmaintainable!
Q: Did Toyota employees refer to their software as spaghetti like? A: Yes
They knew it was spaghetti!
I can confirm that Toyota's source code has over is 11,000 global variables, they are saying that it is greatly intertwined in such a way that spaghetti -- the data -- if you want to follow a particular path, you know, where does the accelerator signal go, you have to trace through multiple files and multiple tasks to see where that data goes. And they're all linked together with these global variables. Some of which are 25, 30 characters long and some don't have vowels and some -- two of them are identical, except one has a P and one has a D, or a P and a B.
Poorly named variables. Tons of global variables. Clear violations of clean coding principals.
And my organizations that I've consulted with maintain a practice where they will not release a product if it has a code complexity of any function bigger than, a typical number is 30. Toyota's code actually has 67 functions that score over 50, which has been assessed as an untestable score. Now, within those 67, there are 12 in the 2005 Camry that have over 100, which is assessed at a level of unmaintainable. And one of those 12 unmaintainable functions is the approximately 1,300 line function that performs the calculation of mathematics to decide how open to make that throttle.
The function responsible for setting the throttle was unmaintainable!
I've dealt with code bases that had grown into beasts similar to the Totota Camry code base. It was not fun. And it was difficult to add unit tests and carve up and refactor the code, but I made it a priority simply because I knew bugs would be popping up all over the place if we didn't clean up the code. And all we were doing was serving advertisements! Toyota's code is controlling a goddam car!
Today’s vehicles contain around 100 million lines of code. Code that is directly responsible for people’s lives. You might imagine that much code would be well maintained, testable and so clean it squeaks. It should be but it’s not. We as software developers are responsible for this epidemic of filthy, sloppy, dirty spaghetti. For this scourge of unmaintainable balls of mud. For this spate of untestable houses of cards.
We must stand up for clean code. Clean code saves money! Clean code saves lives!