13 Bankers, by Johnson and Kwak, is an interesting but at times text-bookish read on the history, successes and failures of banks in the US from the time of the founding fathers to the recent mortgage induced meltdown.  The book makes a compelling point that effective bank regulation cannot happen while former bankers are so tightly tied to American politics (witness Paulson, Geithner and several other former bankers as recent Secretaries of the Treasury).  Given the size and influence of the remaining 6 large banks, the authors argue that the banks have effectively created an oligarchy.  The authors defend their position through recent events; even after one of the worst financial meltdowns of all time we have yet to reign in the level of risk taking of these modern goliaths.  Banker compensation is at a near all time high, high risk financial products continue to be developed with little or no oversight and the taxpayer has funded all of this with below market loans in the form of TARP.  In every other meltdown in US banking history, we have swung to tightly regulate and control the banks while in this one we simply loaned them money and went back to business as usual.  The authors implicitly argue that Jefferson’s (who was against a central bank for fear of it having too great an influence over government) worst fears have been realized.

The book is informative, and it got me to think about governance overall – especially over technology functions within companies.  Just as the existing board structures in coordination with the government seem to be ill equipped to properly govern and regulate modern banks, so too are too many product companies ill equipped to govern their technology efforts.   CEOs often do not have a deep technical background and while we’ve written that they do not necessarily need to be technical, we’ve also stated that they should ensure that they have the appropriate internal technical talent and outside technical governance.  I’m not talking about the tired old topic of internal IT governance here, though that is still important.   This is about ensuring that the governance of the company is correct and representative of shareholder interests.

Looking at the boards of most companies private and public companies, you will not find many successful technologists as directors.  These boards are likely to have former CEOs, CFOs and CMOs but precious few of them will have former CTOs and CIOs.  Given the level of spend within technology companies, what could be more important than ensuring that spending and focus is appropriate to the needs and the direction of the company?  Given that the product is what creates shareholder wealth, what could be more important than having someone with relevant technical product experience on the board of directors?

A lack of understanding of the risk of certain financial products (especially in the case of debt derivatives like CDOs) on the part of boards of directors and regulators is at least partially to blame for the most recent financial failures.   If former bankers sitting on these boards of directors can’t understand the firm’s financial products, how can a board devoid of technologists have a chance of understanding the ins and outs of developing the company’s products?

While clearly not of the magnitude of our current bank governance problem, it is at least a problem that should be solved as technology continues to play a larger role in each of our lives, our companies and our futures.

The Big Short puts the cloudy mechanics underlying the real estate bubble into easily understood layperson’s terms.  Michael Lewis is perhaps one of the greatest story tellers of our generation, and his ability to describe complex situations simply while telling a human interest story in a novel like fashion is in my opinion unmatched.

The Big Short explains the world of CDOs and CDSs, while simultaneously telling the stories of how they came to be and how they contributed to both the creation and popping of the bubble.  Lewis also does a great job of explaining how both Moodys and Standard & Poors failed to properly protect investor interests.  The punchline:  Debt that would likely have a high default rate and therefore low ratings was  repackaged into new securities and given new, higher ratings.  The theory by bankers, analysts and rating agencies was that by bundling several states worth of mortgages into a single instrument, risk would go down because not all states would have high rates of default at the same time.

Lewis, and this book have taken a great deal of heat and criticism from articles he published in 2007.  His comments echoed those of Alan Greenspan, and are supported in a fashion by the efficient market hypothesis.    Lewis is cited as having been a huge supporter of derivatives as an efficient mechanism to redistribute risk.  But The Big Short isn’t a change in position, an attempt to argue that he didn’t make those statements or an acknowledgement that he was wrong.  It’s just a well told story, in typical Lewis fashion, of one of the worst financial events in recent memory.

As we describe in our book and as it is outlined in the ITIL toolkit, all organizations can benefit greatly from the separation of Incidents and Problems.  Incidents are customer impacting events in your production environment, or as the ITIL defines them “an event which is not part of the standard operation of a service and which causes or may cause disruption to or a reduction in the quality of services and Customer productivity”.   Problems are the cause of one or more incidents.

The separation of these are important as most of us wish to quickly resolve incidents (reduce or minimize customer impact) while permanently resolving the underlying problems causing them.  The actions we take to resolve an incident may include workarounds or band-aids to restore service while the team works to eliminate the root cause of the problem.  We strive to restore service in whatever way possible as quickly as possible while working to find true root cause for the service disruption.

There is another important piece we typically recommend to our clients and that is to map incidents to customer complaints or customer cost.  This cost may include the real cost of handling customer contacts through phone, chat and email.  It also should include the risk of customer departure, engineering cost in workarounds or permanent fixes, overall customer satisfaction and lost opportunity of working on fixes v. other revenue enhancing features.

We know that a problem may cause one or more incidents and that an incident might be caused by one or more problems.  But that information alone isn’t enough to prioritize, with limited resources, what we attack first in short, medium and long term product and architecture changes.  Because not every incident costs us the same to fix, we need to identify what 20% of incidents drive 80% of our problems (assuming that the Pareto Principle applies).  At the very least, we should be working on those incidents and associated problems that are high in customer cost and risk relative to other incidents and problems.

By adding Customer Cost (the “C” in the P-I-C process) to our operations morning meetings, and evaluating it alongside incidents and their problems we can help make better decisions.   Classifying the severity of the incident by this “C” and using that classification to drive effort and resolution aligns your engineering operations with your  business objectives.

Somewhere in the mix of my father, the United States Military Academy and the Army I learned two important self tests for leadership: the “Man in the Mirror” test and the “Would I” test.  While I am human and have failed these tests and their exacting standards from time to time, I think they have on balance served me well.

The Man in the Mirror test is:  Can you look yourself in the eye (in the mirror) after you’ve made this decision?  The question is profound and very powerful.  It alone, if asked at the appropriate time, might keep you from making otherwise foolish ethical choices and poor leadership decisions.  Would it have kept Lay and Skilling, Maddoff or Kozlowski  from  violating their fiduciary responsibilities?  While I can’t answer this question, I do believe that if a person is “on balance good” and hasn’t succumbed to greed induced sociopathic behaviors, then this test will help keep them straight if asked at the appropriate times.  As we’ve written before, building tests such as these into your daily routine can help you from taking the long journey of small steps towards unethical behavior.

The “Would I” test is much more focused on the concept of “Leadership by Example” that we discuss in The Art of Scalability.  This test is also simple.  You just ask yourself “Would I be willing to do what I am asking this person to do?”  Maybe you are asking someone to work during their anniversary.  Perhaps you are asking someone to skip a child’s sporting or school event.  Maybe you’ve just given them some last minute assignment that will cause them to work all night, like completing a presentation for you to give at a conference tomorrow.  “Would I” is not an excuse to be lenient on people, or not to hold people to aggressive standards.  Rather, it is a test to determine if you are truly meeting the expectations that you hold of those around you.

In one respect, the “Man in the Mirror” and “Would I” tests fly in the face of concepts such as Strengths Based Leadership.   They exist to keep us from failing due to shortsightedness, a lack of introspection and aggressive or excessive egoism.  They are rooted in the concept that we sometimes fail because we fail to see how our actions will be perceived or acted upon by others.  Leaders are humans, and leaders make mistakes.  The “Man in the Mirror”, if employed often, helps us to avoid dangerous ethical pitfalls and the “Would I” test helps us avoid burning out our teams.

Unless you’re one of the incredibly lucky sites where the traffic spikes 100x overnight, scalability problems don’t sneak up on you. They give you warning signs that if you are able to recognize and react to appropriately, allow you to stay ahead of the issues. However, we’re often so head down getting the next release out the door that we don’t take the time to realize we’re experiencing warning signs until they become huge problems staring us in the face.  Here are a few of the warnings that we’ve heard teams talk about in the past couple of months that were clearly signs of problems on the horizon.

Not wanting to make changes – If you find yourself denying request for changes to certain parts of your system, this might be a warning sign that you have scalability issues with that component. A completely scalable system has components that can fail without catastrophic impact to customers. If you’re avoiding changes to a component because of the risk of problems this is a warning sign that you need to re-architect to eliminate or at least mitigate the risk.

Performance creep – If after each release you need to add hardware to a subsystem or you accept a performance degradation in a service you could have a scaling issue approaching quickly. Consistently increasing consumption of CPU or memory resources in a service with each release will lead you into an unsustainable situation. If today you’re comfortably sitting at 40% CPU utilization and you allow a modest 10% degradation in each release you have less than nine releases before you are well above 100% but the reality is you won’t get close to that without significant issues.

Investigating larger hardware – If you’ve started asking your vendors or VAR about bigger hardware you’re heading down the path of scalability problems. The scale of more computational resources per dollar is not linear, it’s closer to cubic or even exponential scales. Purchasing more expensive hardware might seem like the economical way out when you compare the cost of the first hardware upgrade versus developer time but run the calculation out several iterations. When you get to a Sun Fire™ E25K Server with Oracle Database 10g at a $6M price tag you might feel differently about the decision.

Asking vendors for advanced features – When you start exploring advanced options of your vendor’s software you’re likely heading down the path of increased complexity and this is a clear warning sign of scalability problems in your future. Besides potentially locking you into a vendor which lowers your negotiating power it puts the fate of your company in someone else’s hands, which wouldn’t make me sleep very well at night. See our post on using vendor features to scale for more information.

Watch out for these or similar warning signs that scalability problems are looming on the horizon. Dealing with the problems today while you have time to plan properly might not get you an award for being a firefighter but you’ll more likely deliver a quality product without costly interruption.