GROWTH BLOG: Learning from Failure: Conducting Postmortems
AKF Partners Logo Technology ConsultingScalability - We wrote the book on it ℠

Growth Blog

Scalability and Technology Consulting Advice for SaaS and Technology Companies

Normalization of Deviance and Software...Oh and Nasa

July 22, 2019  |  Posted By: Eric Arrington

waiting for challenger launch - normalization of deviance

It’s funny how clearly you can remember some events from your childhood. I remember exactly where I was on Jan 28th, 1986.

All the kids in Modoc Elementary School had been ushered into the Multi Purpose Room. It was an exciting day. We were all going to watch the Challenger Shuttle Launch. The school was especially excited about this launch. A civilian school teacher was going into space.

I was sitting right up front (probably so the teacher could keep an eye on me). I had on the paper helmet I had made the day before. I was ready to sign up for NASA. We all counted down and then cheered when the shuttle lifted off.

Seventy-three seconds in something happened.

There was an obvious malfunction. For once the kids were silent. Teachers didn’t know what to do. We all sat there watching. Watching as the Challenger exploded in mid air, taking the lives of all 7 crew members aboard.

How could this have happened? Some fluke accident after all that careful planning? This was NASA. They thought of everything right?

I recently picked up a book by Dr. Diane Vaughan called The Challenger Launch Decision. Vaughan isn’t an engineer, she is a sociologist. She doesn’t study Newtonian Mechanics. She studies social institutions, cultures, organizations, and interactions between people that work together.

She wasn’t interested in O-rings failing. She wanted to understand the environment that led to such a failure.

She realized that it’s easy for people to rationalize shortcuts under pressure. Let’s be honest, do any of us not work under a certain amount of pressure? The rationalization gets even easier when you take a shortcut and nothing bad happens. Lack of a “bad outcome” can actually justify the shortcut.

After studying the Challenger Launch and other failures, Vaughan came up with the theory for this type of breakdown in procedure. She called this theory the normalization of deviance. She defines it as:

The gradual process through which unacceptable practice or standards become acceptable. As the deviant behavior is repeated without catastrophic results, it becomes the social norm for the organization

In other words, the gradual breakdown in process where a violation of procedure becomes acceptable. One important key is, it happens even though everyone involved knows better.

normalization of deviance and software

Normalization of Deviance and What Happened at NASA

Prior to the launch, NASA became more and more focused on hitting the launch date (sound familiar?). Deviants from established procedures kept popping up. Instead of reevaluating and changing things, the deviants were accepted. Over time these deviants became the new normal.

Erosion between the O-rings had occurred before the date of the launch. It wasn’t a new occurrence. The issue was, erosion past the O-rings wasn’t supposed to happen at all. It was happening on every flight. The engineers scratched their heads and made changes but the erosion kept happening. They argued that yes, it was happening but it was stable so it could be ignored.

In other words, the O-rings didn’t completely fail so it was ok. A condition that was at one time deemed unacceptable was now considered to be acceptable. The deviance had become the new normal. This deviance led to the death of 7 people and scarred a bunch of my classmates for life (don’t worry I was ok).

Normalization of Deviance

Normalization of deviance doesn’t only happen at NASA. Their failures tend to garner more attention though. When you’re sitting on more than 500,000 gallons of liquid oxygen and liquid hydrogen the failures are spectacular.

Most of us don’t work in a job where a failure can cost someone their life. That doesn’t mean these principles don’t apply to us. Normalization of deviance happens in all industries.

There is a study of how the normalization of deviance affects healthcare. The author, John Banja, identifies 7 factors that contribute to normalizing unacceptable behaviors. These 7 factors are extremely relevant to us in the software industry as well. Here are his seven factors and some takeaways for the software world.

1. The rules are stupid and inefficient!

I am sure you have never heard this at your company before. A good alternative would be, “management doesn’t understand what we are doing. Their rules slow us down.”

In this situation the person violating the rule understands the rule. He just doesn’t think management understands his job. The rule was handed down by someone in management who doesn’t know what it’s like to be “in the trenches.”

Guess what? Sometimes this is true. Sometimes the rules are stupid and inefficient and are created by someone that is out of touch. What is the solution? Don’t ignore the rule. Go find out why the rule is there.

2. Knowledge is imperfect and uneven.

In this case, the “offender” falls under 3 possible categories:

They are unaware that the rule exists.

They might know about the rule but fail to get why it applies to them.

They have been taught deviants by other co-workers.

This is especially a problem in a culture where people are afraid to ask for help. This problem gets compounded with every new hire. Have you ever asked why a certain thing was done at a new job and heard back, “I don’t know, that’s just how things are done here”?

Foster a culture where it is acceptable to ask questions. New hires and juniors should feel empowered to ask “why.”

3. The work itself, along with new technology, can disrupt work behaviors and rule compliance.

We all do complex work in a dynamic environment. It’s unpredictable. New technologies and new environments can lead us to come up with solutions that don’t perfectly fit established procedures. Engineers are forced to come up with answers that might not fit in the old documented standards.  

4. I’m breaking the rule for the good of my patient!

We don’t have patients, but we can see this in our world as well. Substitute the word user for patient. Have you ever violated a procedure for the good of the user or ease of integration with a colleague?

What would be a better solution? If it’s a better way and you don’t see any negative to doing it that way, communicate it. It might be beneficial to everyone to not have that rule. Have a discussion with your team about what you are trying to do and why. Maybe the rule can be changed or maybe you aren’t seeing the whole picture.

5. The rules don’t apply to me/you can trust me.

“It’s a good rule for everyone else but I have been here for 10 years. I understand the system better than everyone. I know how and when to break the rules.”

We see this a lot as startups grow up. Employee #2 doesn’t need to follow the rules right? She knows every line of code in the repo. Here is the problem, developers aren’t know for our humility. We all think we are that person. We all think we understand things so well that we know what we can get away with.

6. Workers are afraid to speak up.

The likelihood of deviant behavior increases in a culture that discourages people from speaking up. Fear of confrontation, fear or retaliation, “not my job”, and lack of confidence make ignoring something even though it’s wrong easier.

Let’s be honest, as developers we aren’t always highly functioning human beings. We are great when our heads are down and we’re banging on a keyboard but when we are face to face with another human? That’s a different set of tools than most of us don’t have in our quiver.

This especially difficult in a relationship between a junior and senior engineer. It’s hard to a junior engineer to point out flaws or call out procedure violations to a senior engineer.

7. Leadership withholding or diluting findings on system problems.

We know about deviant behavior, we just dilute it as we pass it up the chain of command. This can happen for many reasons but can be mostly summed by “company politics.” Maybe someone doesn’t want to look bad to superiors so they won’t report the incident fully. Maybe you don’t discipline a top performer for unacceptable behavior because you are afraid they might leave.

You also see this in companies that have a culture where managers lead with an iron fist. People feel compelled to protect coworkers and don’t pass information along.

How Do You Fix It

This happens everywhere. It happens at your current job, at home, with your personal habits, driving habits, diet and exercise; it’s everywhere. There are 3 important steps to fighting it.

Creating and Communicating Good Processes

It’s simple, bad processes lead to bad results. Good processes that aren’t documented and/or accessible lead to bad results. Detailed and documented processes are the first step to fixing this culture of deviance.

Good documentation helps you maintain operational consistency. The next step is to make sure each employee knows the process.

Create good processes, document them, train employees, and hold everyone accountable for maintaining them.

Create a Collaborative Environment

This is especially true when creating new processes. Bring the whole team in to discuss. People should feel some ownership over the process they are accountable for.

Remember, normalization of deviance is a social problem. If a process is created as a group then the social need to adhere to it as a group is more powerful.

This also solves problem #1 Rules are Stupid. If the team makes the rules then they will be more likely to follow them.

Create a Culture of Communication

The key to fighting normalization of deviance is to understand that everyone knows better. If employees are consistently accepting deviants to accepted procedures then find out why.

A great way to see this is in action is to watch what happens when a new hire comes to the team with an alert. How does the team react? Do they brush them off? If so, then you probably have a team that is accepting deviant practices.

Employees should feel empowered to “hit the e-stop” on their processes and tasks. Employees, especially juniors, should be encouraged to question the established order of things. They need to feel comfortable asking “why?”.

Conventional wisdom needs to be questioned. They will be wrong most of the time. This will give you an opportunity to explain why you do things the way you do. If they are right then you make the procedure better. It’s a no lose situation.

Key Takeaways

As you can tell, most of the solutions are the same: Communication. Creating a culture of communication is the only way to keep from falling into this trap. Empower your employees to question the status quo. You will create stronger teams, better ideas, and improved performance.

There is only one way to catch normalization of deviance before it sets in: Create a culture of honesty, communication, and continuous improvement.

Sometimes it’s hard to judge this in your own culture. I call this “ship in a bottle” syndrome. When you’re in the bottle it’s hard to see things clearly. AKF has helped hundreds of software companies change their culture. Give us a call, we can help.

Next: Hiring the Right Employees


Most Popular: