The partial meltdown of TMI-2 at Three Mile Island in 1979 is one of the best known crisis situations within the US and was the source of several books, and at least one movie.  It also generated two theories relevant to crisis management.

Charles Perrow’s Normal Accident Theory (NAT), described in his book Normal Accidents, states that the complexity inherent to tightly coupled technology systems makes accidents inevitable.  Perrow’s hypothesis is that the tight coupling causes interactions to escalate rapidly and without obstruction.  “Normal” is a nod to the inevitability of such accidents.

Todd LaPorte, who founded the Berkeley school of High Reliability Theory, believes that there are organizational strategies to achieve high reliability even in the face of such tight coupling.  The two theories have been debated for quite some time.  While the authors don’t completely agree as to how they can coexist (LaPorte believes that they are complimentary and Perrow believes that they are useful for the purposes of comparison), we believe there is something to be gained from them.

One paradox from these debates becomes intuitively obvious to our pursuit of high availability and highly scalable systems:  The better we are at building systems that avoid problems and crises, the less practice we have in solving problems and crises.  As the practice of resolving failures are critical to our learning, we become more and more inept at rapidly resolving these failures as their frequency decreases.  Therefore, as we get better at building fault tolerant and scalable systems, we get worse at resolving crisis situations that are almost certain to happen at some point.

Weick and Sutcliffe have a solution to this paradox that we paraphrase as “organizational mindfulness”.  They identify 5 practices for developing this mindfulness:

1)      Preoccupation with failure.  This practice is all about monitoring IT systems and reporting errors in a timely fashion.  Success, they argue, narrows perceptions and breeds overconfidence.   To combat the resulting complacency, organizations need complete transparency into system faults and failures.  Reports should be widely distributed and discussed frequently such as in our oft recommended “operations review” process outlined within the Art of Scalability.

2)      Reluctance to simplify interpretations.  Take nothing for granted and seek input from diverse sources.  Don’t try to box failures into expected behavior and act with a healthy bit of paranoia.

3)      Sensitivity to operations.  Look at detail data at the minute level, such as we’ve suggested in our posts on monitoring.  Include the usage of real time data and make ongoing assessments and continual updates of this data.  We think our book and our post on monitoring strategies have some good suggestions on this topic.

4)      Commitment to resilience.  Build excess capability by rotating positions and training your people in new skills.  Former employees of eBay operations can attest that DBAs, SAs and Network Engineers used to be rotated through the operations center to do just this.  Furthermore, once fixes are made the organization should be quickly returned to a sense of preparedness for the next situation.

5)      Deference to expertise.  During crisis events, shift the leadership role to the person possessing the greatest expertise to deal with the problem.  Our book also suggests creating a competency around crisis management such as a “technical duty officer” in the operations center.

We would add that every operations team should use every failure as a learning opportunity, especially in those environments in which failures are infrequent.  A good way to do this is to leverage the post mortem process.

In an article on the NY Times Deal Book All That Twitters May Not Be Gold, Analysts Say they state “…the Web 2.0 model of building a product and then figuring out how to monetize it has been largely debunked.”  We agree that There Is No Substitute For Profitability.  The article continues, stating “People who sign up for free services tend to resent a company for trying to wring revenue from the business later.”  I’m not sure that is necessarily the case especially ad based models that are not invasive.  I can’t argue that in this economy the sooner any company starts generating revenue the better.

Juan Enriquez, a leading authority on the economic and political impacts of life sciences, gave his annual talk at TED this year and had some great insights. His presentation was called “Beyond the crisis, mind boggling science and the arrival of Homo evolutis.” Besides his perspective on the singularity, as in “The Singularity is Near” by Ray Kurzweil, he had an interesting take on the economy.  If you don’t check out the entire 18 min video, go to min 7:00.  He talks about how venture backed startups are 0.02% of the GDP investment but generate 17.8% of the output.  As he states, these companies are the future of the U.S. economy.

Here is an article discussing venture funding during this economy and includes one of our recently funded clients.  

(Phoenix, AZ) February 19, 2009 - For start-up companies vying for increasingly scarce venture dollars, talk is cheap. According to AKF Partners, a boutique advisory services firm that works primarily with hyper-growth start-ups, many of which have recently secured early-stage financing, VCs will not seriously consider an investment in any business that cannot demonstrate a quick and viable path to profitability.

Venture investment has always been risky, but in a world where assets traditionally viewed as ‘safe’ are imploding all around us, the risk in start-up companies hasn’t increased substantially, says Kevin Fortuna, a partner with AKF.  Venture capitalists, just like everyone else, are belt-tightening in this environment, but they are still on the hunt for great ideas and great management teams, and they are pulling the trigger if the deal is right.

So what are the venture firms looking for?  Professional investors want to see a great idea, a great team one with a track record for building shareholder value and, above all else, they want to see that the company is executing on a plan that will quickly lead to profitability and scale, continues Fortuna.  In particular, investors want to see that a technology platform can scale well past the point of profitability. Our clients recognize that, so they’re attracting venture investments while others are failing.

In the past month alone, three clients of AKF Partners, Outbrain, Achieve CCA, and Snooth, have secured early-stage venture financing. In its role as a key advisor and partner, AKF is helping these and other hyper-growth companies align their strategy and operations and get the right financial backing, so they can succeed in today’s challenging environment.

Snooth.com, the world’s largest wine site, which grew 400 percent last year and has more than 500,000 unique visitors a month and over two million reviews, has been working with AKF Partners as a key advisor.  

“When we started, we had great organic growth and all the traffic we could handle,” said Philip James, Founder and CEO of Snooth.com. “But when we needed to take the company to the next level, we turned to AKF Partners.  Kevin Fortuna at AKF Partners had the established contacts to get to the right people at the right level, and his advice on strategy and positioning has been invaluable. It’s also great to have access to AKF’s unbeatable technical and scalability expertise.”  

Today isn’t necessarily a bad time to be starting a company. During the last deep recession in the 1970s, companies like Microsoft, Fed-Ex and Costco were started. But you need to get the wheels turning and show that the model is working before venture firms will be giving you money to expand, sums up Fortuna.

The key to managing through this downturn is focus, hard work, and capital. After a certain point, if a company has the right ingredients, then survival will pay off.

General Douglas MacArthur was a quote machine. One of my favorite is “War’s very object is victory, not prolonged indecision. In war there is no substitute for victory.” What’s true in war is true in business. Swap “war” with “business” and “victory” with profitability and you have a statement we all should heed: ““Business’s very object is profitability, not prolonged indecision. In business there is no substitute for profitability.”

Why do we keep forgetting this? What’s wrong with us that we continue to focus on businesses metrics such as unique or concurrent users and growth in both? None of these matter if they don’t directly relate to profitability. Don’t fool yourself folks – don’t think that getting the “traction” of users is the hardest part of the business. Don’t think that you’ll just layer on advertising later and have a profitable business.

If you aren’t building in profitability from day 1 you are creating a cash incinerator. The folks at Google are still trying to figure out how to make YouTube profitable. Great idea? Maybe…if it ever becomes profitable. Anyone can build something to burn cash. After all, it’s just a free hosting environment for videos. Going to business without a plan for profitability is like going to war without a strategy to win. The only difference is that you’ll burn cash rather than lives.

I had a power outage at my house this morning and was without electricity for a while.  We all come to expect that now and then from our utility providers.  This morning I happened to be Johnny-on-the-spot and called right away.  The agent who answered was polite enough when he acknowledged that he didn’t know anything about the outage because no one else had reported it yet.  I always knew this was the case but never really thought about how ridiculous it was to have your customers be your first line of monitoring.  How can companies, year after year, look at their processes and decide that they are doing fine by waiting for the customer complaints to come rolling in before they react to a problem.  

This seems ridiculous for a utility company but even more so for a SaaS or Web 2.0 company.  Unfortunately this occurs all the time.  Technology staffs state things like “we know we have a problem when we start getting emails from customers.”  If that’s your first signal you are too late!  Get some monitoring setup on your system and services to warn you about potential issues so you can either react to prevent a problem or proactively notify your customers.  Contrary to what some belief customers are there to use your service not debug or monitor your application.

Think how much better I would have felt about my utility company had the agent who answered stated that they knew I was one of a hundred homes in the area without power and that they had dispatched a crew to fix the problem, initial estimates were 2 hours.

This is the second on our two part post on the Principles of War and our interpretation of them relative to the business world.  The 9 US Principles of War (derived from von Clausewitz’s essay “Principles of War” and his book “On War”) are Objective, Offensive, Mass, Economy of Force, Maneuver, Unity of Command, Security, Surprise and Simplicity.  We will discuss the last four of these in this post.

Unity of Command.  The US Armed Forces definition is that for each objective, there should be a single owner or commander and that the forces necessary to achieve that objective should be placed under the authority of that commander.  In the business world, this does not mean that you should slice your technology, client services and product teams into separate groups under each general manager or objective owner.  Rather, it means that the person responsible for achieving some business objective should have the authority to direct the resources necessary to achieving that goal.  These resources could be set up in project teams that respond to the needs of the objective owner, or they could be “dotted lined” to the individual.  The key here is that for any objective there should be a clearly defined and empowered owner of the objective.  

Security.  Security enhances freedom of action by reducing vulnerability to hostile acts, influence or surprise.  While you may jump immediate to “information and technology security” implemented as policies within firewalls and such, we believe this has a much broader meaning.  Portions of this principle speak to your actions and efforts to get early warning of competitive threats – not just the threats afforded by hackers and the like.  What are you doing in an ethical fashion to find out how your competitors are responding to your actions?  How do you monitor the strategies and products of your competitors?  How well do you know your competition?

Surprise.  Strike the enemy at a time or place or in a manner for which he (or she) is unprepared.  This principle is the hardest to achieve within the business world, but can have incredible results when it is in fact achieved.  Surprise is achieved when a struggling computer company releases a portable personal music device such as Apple did with the iPod.  Sony, the creator of the portable music device phenomenon was taken completely by surprise and had previously never even considered Apple a competitor.  Surprise, therefore, does not have to be a principle you apply to those you currently consider to be your competitors – it can be applied to markets tangent to the ones in which you currently operate.  Surprise can also manifest itself as a change in approach such as Nintendo’s approach with the Wii.  While Microsoft and Sony focused on more complex graphics and processors, Nintendo took the surprise approach of using less sophisticated graphics and processing power (thereby offering a lower initial price) and focused their approach on a revolutionary game controller (the Wii Nunchuk and motion system). 

Simplicity.  Prepare clean, uncomplicated plans and concise instructions that ensure thorough understanding.   This is perhaps the most easily understood within the business context of all the principles of war.  Simply stated, you need to make sure your plans, orders, and objectives are understood by all parties and are unambiguous.

Summarizing the nine principles from our two posts, then leave us with 9 Principles of Business in a slightly restated fashion:

1. Create Clearly Defined Aggressive But Achievable Goals (Objective)
2. Be First to Market and Aggressive in Your Implementation (Offensive)
3. Align Your Companies Organizations with Your Objectives (Mass)
4. Employ Your Teams and Organizations in Accordance with Their Capabilities (Economy of Force)
5. Maintain Business Flexibility but Do Not Oscillate Constantly (Maneuver)
6. Clearly Define Ownership of Objectives and Empower those Individuals (Unity of Command)
7. Sense and Respond to your Competition (Security)
8. Surprise your competition with your timing or approach (Surprise)
9. Constantly Communicate and Simplify Your Plans (Simplicity)

Many authors have previously described the relationship between business and war and we believe that the most successful businesses approach their operations as would General Douglas MacArthur when he claimed that “In war, there is no substitute for victory”.

Carl von Clausewitz offered several tenets of war in his essay “Principles of War” and later expanded upon those in his book “On War”.  Many armed forces throughout the world have taken portions of these tenets and adopted them for their own use.  This post is the first in a two part series relating the 9 US Armed Forces Principles of War to your everyday business activities, strategy and tactics.  The 9 US Principles of War are Objective, Offensive, Mass, Economy of Force, Manuever, Unity of Command, Security, Surprise and Simplicity.  We will discuss the first 5 in this post and the next 4 in a subsequent post.

Objective.  The US Armed Forces definition is to direct every military operation toward a clearly defined, decisive and attainable objective.  We think this is pretty self explanatory and includes concepts about which we’ve previously blogged such as the need to set aggressive but achievable goals.  The most important aspects of “Objective” as applied to your business are for your goals to be clearly defined, well understood, measurable and attainable.

Offensive.  The military definition is to seize, retain and exploit the initiative.  The business definition here is found by looking at what Offensive implies – specifically that it’s all about time to market and getting the right features, products and services out and adopted first.  Being first offers the best chance at achieving virility within the market, and creating a viral marketplace or product is the military equivalent of seizing the high ground.

Mass.  The military definition is to mass the overwhelming effects of combat power at the decisive place and time.  Mass here in military terms is different from the concentration of forces which may not be desirable.  Combat power refers to all the aspects of military power from infantry and armor, to field artillery and other combat multipliers. The business equivalent is to ensure that your business units are aligned with your greater business objective and that they are contributing to it properly.  Your technology, product, marketing and finance teams should all realize and be contributing to the core objectives necessary to win your business battle.  If you wish to win quickly, they cannot be marching to separate agendas and they should not be fighting with each other.

Economy of Force.  This one can be confusing, but within the military definition is a reference to “No part of the force should be left without a purpose”.  The military definition also hints that every part of the force should be used in the most effective way possible.  Goals and objectives are again part of this, but more importantly you should be able to answer the question of whether you are using the right team for the job at hand.  Not only should you ensure that every organization has a purpose directly relating to your most important initiatives, you need to ensure that they are the best team to have those specific goals and objectives.  Client Services and Customer Support teams might be useful in helping to QA new products but allocating them 100% to such an endeavor is probably not the most leveraged use of their time.  Conversely, forgetting to include Customer Support or Client Services in any product rollout is a failure to employ a very important part of your “combat power” in achieving product success.  While its useful for engineers to understand customer needs and complaints, allowing more than 5 to 10% of their time to be taken up by such activities is a costly endeavor relative to your future product needs.

Maneuver. Place the enemy in a position of disadvantage through the flexible application of combat power.  This one relates to how flexible you are in your product delivery lifecycle, and whether you are set up to respond to your competitors actions in the marketplace.  This IS NOT an argument that you should abandon products in flight and constantly change your strategy.  Constant change in strategy is a clear indication of a management team incapable of defining a winning path and it’s a early indication of likely future failure.  You should be flexible, and changing features or making course corrections a few times a year is appropriate.  Ensuring that your product delivery processes allow you the flexibility to change (with the additional cost that implies) is critical to success.  But constant change is not a strategy – it’s a recipe for disaster.

If you are one of those people who think that a startup consists of a couple of years of hard work followed by a large payout, think again. Most venture backed companies never make it to that payout and those that do often take much longer than two years to do it. Furthermore, those “years”, be they 2 or 10, are very stressful years packed with a roller coaster ride of emotions that any theme park would be proud to call its own.

Life in a startup, especially an early stage startup, is full of ups and downs. On Monday you are on top of the world, having signed a new customer and released an enhancement to your base product; champagne corks pop and party hats are worn by all. By Wednesday you fear that the most recent release from one of your competitors will be the end of your business and you hold an all night product roadmap session to re plan how you will spend the rest of your product manpower and budget through the course of the year.

In our experience, the entrepreneurial CEO most often has the greatest emotional amplitude in the highs and lows and the resulting incredible mood swings are not healthy for the company on either the euphoric peak or the abysmal trough. Behaviors and actions vary so widely in many young CEOs that the organization simply cannot react optimally to them. For instance, during the peak, the CEO talks about the IPO, high-fives and knuckles the team and takes them out for drinks. Within three days, people are hearing of layoffs, cut backs and certain business death. During the highs, the CEO talks about what a stellar team he or she has and during the lows he or she complains how no one really works hard enough and how no one cares about the success of the company.

Sound familiar?  If it does, listen closely.  YOU MUST STOP!  If you want to lead, then you must be the pillar of the company in both good times and bad.  Don’t let yourself believe that you get a pass on this advice just because its “the way you work”.  Don’t be an enabler of yourself in behaviors that run counter to creating shareholder value.  You must absolutely be the firm guiding hand in good times that reminds the company that there is still a fight for survival going on even as the celebration reaches its crescendo and you must embody the hope that the company seeks in bad times by reassuring them of the strategy and how exactly it will be successful against the competition.

While we agree that flexibility of strategy especially in the ability to sense and respond to competition is a key to success, we argue that changing strategy weekly is a lack of strategy and you are suffering from IotD (Idea of the Day) as we discuss in a previous article.   Making course corrections through the year to capitalize on a competitor’s weaknesses or to defend against a competitor’s attacks is absolutely appropriate.  Holding all night sessions the day after a competitor’s release, especially given that the development cycle for any product course correction will likely dwarf the hasty strategy development time is ludicrous and fraught with perils such as a lack of competitive information and a complete lack of an appropriate product discovery phase.

What are the lessons for each of you in this?  For the entrepreneurial CEO, recognize this ride is going to happen and monitor your reactions to the highs and lows.  Be the leader for your organization and keep a level head in front of your teams so they can remain focused on delivery.  If you’ve been complaining lately about how nothing ever gets done and no one cares enough about the company, look inward and ask if you aren’t the one causing this problem.  For the CTOs, recognize this behavior in your boss and don’t jump on board the rollercoaster.  Try to get your CEO to read this article, and if that fails, just keep reminding them to stay the course.  The upside if you resist is that you are being the leader your team needs to buffer them from the whipsaw and tomorrow the CEO will probably change his or her mind back again.  For the engineers, ask during the interview how focused the team is and how long projects are given to complete.  If they brag that they change product strategy everyday to keep up with the competition consider this a warning sign.  If they are contemplative, understanding the true cost of changing direction, and reactive only after a well thought out discussions, then consider those as good signs of a mature leadership team.

Ever wonder how situations like Tyco and Enron happen?  The argument of whether certain people are just born to lie, cheat and steal is best left to psychologists, theologists and sociologists.  Instead, we want to explore how leaders can negatively or positively influence a company’s culture.  Our position is that overlooked and unchecked small ethic violations will over a number of years lead to increasingly larger ethics problems and ultimately mushroom into a catastrophic breach of law or regulation.  It’s not enough to simply act ethically yourself; you must also police the ethics of your organization as what you allow you teach and what you teach becomes your standard.

One of the biggest mistakes a company can make (the founder or CEO) is not establishing early and clearly what the expected behaviors are for employees.  Is it okay for you to take a pen home?  Can you use the company computer to write personal email?  Can you accept shirts or other gifts from vendors?

Does your company have a policy regarding any of this?  If not, they should.  Every employee brings their own set of moral standards to work with them but like any society there needs to be an agreed upon set of clearly defined principles that govern employee behaviors.  The absence of such principles creates moral ambiguity or worse,  a moral-least-common-denominator, where the lowest set of morals becomes the standard.

When we were young executives it was quite common for our primary vendors to offer us Super Bowl tickets, or tickets to other high value sporting events.  We never took them even if our company at the time allowed such actions.  The larger and more established companies we worked for earlier in our careers had strict guidelines around what was acceptable and these guidelines governed our behaviors even as we went to companies that had not yet created such policies.  The reason such “no gift” policies exist is that the vendors giving such gifts do so to get closer to you and to sway your future decisions. The acceptance of gifts from partners can be seen as abuses of position and power.  If an engineer making $100,000 a year takes a pen or a pair of tickets to a baseball game, why shouldn’t a CEO making $1,000,000 a year use her company administrative assistant to pick up her children from daycare or make a stop to see her family in the company jet?  Without shareholder sanction and appropriate accounting, these actions from the engineer to the CEO are abuses of position and power and set a precedent by way of example for the remainder of the organization.

Following this scenario from the individual contributor to the corner office is a chart that trends from small dollar items to large ticket items in a graph that moves from lower left to upper right outlining abuses of position, responsibility and power.  We do not know Dennis Kozlowski, Kenneth Lay, or Jeffrey Skilling but we are fairly certain that the time they got caught was not the first time they lied, cheated or abused their positions for personal gain.  More likely, they had moral issues throughout their entire career but no one ever checked them on it and in fact they might have actually been promoted or praised for it.  With each promotion they took larger steps over the line that in time took them into what were clearly illegal acts including theft from shareholders.

An example, which may appear unrealistic, is actually very similar to one that we have seen before and more than once.  A small company allows a senior executive to mislead the Board of Directors about revenue projections and working capital and routinely practices “smoothing” of results – the movement of financial performance between quarters to paint the picture of a more easily managed business.  The management team gets praised for operating results that are not completely correct.  The CTO knows of this behavior, at least in the abstract sense if not entirely clear about the details, and sees this behavior accepted.  In effect, the CTO has been conditioned that such behavior is accepted and he does not possess enough moral certitude to maintain the higher ground.  When it comes time to add more servers to the site instead of purchasing legal copies of the operating system and application server, he copies it and does not pay for it.  Again, the individuals involved get rewarded.  The engineers are now taught that such theft is okay and they in turn start making illegal copies of software for personal use.  What ultimately happens is that the company gets investigated by the Business Software Alliance and has to pay three times the original price for the software because of penalties, the BOD catches on to the senior management’s games and replaces the CFO, and across the entire organization moral starts to suffer and key people leave.

Our argument is not that you should not accept pens or shirts from vendors (although that is a pretty good position to take).  Rather, it is that you should draw clear boundaries for everyone and explain both those boundaries and the reasons for them to your employees.  Furthermore, you should create a culture where it is acceptable to question actions based on ethics without repercussion.  Doing so early and often, and rewarding people for doing so will help incent a culture of ethical excellence.  People can make mistakes, and inadvertently make a decision due to haste that can be perceived as ethically perilous without malicious intent.  Perception of an ethical violation is important here as the perception alone will incent the wrong culture, so to the previous point you should reward people for asking questions about certain decisions solely based on the notion of ethics.  This can be a very difficult thing to do, as it is easy to feel angered if someone asks you if a decision is ethically sound – it feels as though they are questioning your morality and that is never fun.  Nevertheless, if you can explain the reasons or alternatively you are willing to say “You know what, you are right – I did not think of it that way so let’s change the decision” you will go a long way to creating the appropriate culture for ethical success.

Every promotion you receive will present new challenges to your character.  Draw immovable boundaries for yourself team today and live by them.  Teach those boundaries and their reasons to your team not only through direct discussion, but through your actions and expectations of the team.