Archive for the ‘CEO’ Category

Please Be Quiet

Monday, August 2nd, 2010

I’ve noticed lately that more companies are putting up signs in hallways and cube farms requesting that people avoid having conversations in these areas. While having a nice quiet work environment makes sense to me as a developer, doesn’t this completely void having people work beside each other? The ad hoc/hallway/water cooler/coffe machine conversations or ones overheard when cube mates are chatting about a new feature are one of the primary benefits of having people work in small open environments.

I haven’t done any sort of scientific study but it seems that these sort of “please be quiet” signs are more prevalent at larger companies. These are the same ones that are trying to mimic the small startup with agile development processes or open work spaces to compete in a fast moving SaaS marketplace. Imitating the actions without understanding the purpose or allowing old school corporate policies to overrule are surefire ways to tank the initiative.

A parallel to the “please be quiet” sign is allowing corporate IT to dictate the architecture of the SaaS offering based on a corporate standard that works for the ERP system. Running Oracle ERP on a 16-way system might be the vendor recommended, preferred approach but for scaling a SaaS offering this is a quick way to run up the costs and ensure lower availability. We often use the analogy of goldfish and thoroughbreds for comparing small, cheap 1U servers with large, expensive multi-processor boxes with lots of memory. The goldfish (small, cheap servers) are inexpensive to purchase and replaceable while the thoroughbred (large servers) are expensive to purchase/maintain and cause big impacts when they go down.

The take away to all of this is that if your part of a corporate initiative to run an internal startup or deliver a Software as a Service from inside a larger organization, don’t allow corporate policies to prevent your success. The differences in approaches, architectures, organizations, and offices have a purpose and should not be discounted as non-critical to the success of your initiative.

Fail Early

Monday, July 12th, 2010

I saw this Nike commercial with Michael Jordan talking about how much he had failed in his career and this got me thinking about the debate of whether or not entrepreneurs should “fail early, fail often” as the saying goes. A quick Google timeline shows how popular this term has grown in our vernacular over the past two decades. No doubt following the trend of high tech entrepreneurial failures.

Digging deeper into some of the more popular entrepreneurial technology bloggers, I found one of the earliest references to failing early from Jeff Atwood from Coding Horror. In May ’06 he stated:

“The best software developers embrace failure — in fact, they’re obsessed with failure. If you forget how easy it is to make critical mistakes, you’re likely to fail. “
And Jeff continues in the same post:
“I say the more failed projects in your portfolio, the better. If you’re not failing some of the time, you’re not trying hard enough. You need to overreach to find your limits and grow. But do make sure you fail in spectacular new ways on each subsequent project.”
Next up that same year in September, Matt from Signal Vs. Noise, the 37Signals blog, quoted Jonathan Ive, VP of Industrial Design, Apple:
“Getting Real is all about iterations too. “Be excited to be wrong because then you’ve discovered something new” is a neat way to put it (btw, so is Fail early, fail often).”
Fast forward to February 2009, Jason, also from Signal vs. Noise states:
“It’s true: Everything is a learning experience. Good and bad, there’s something to be learned. But all learning isn’t equal. I’ve found that if you’re going to spend your time pondering the past, focus on the wins not the losses. The lessons learned from doing well give you a better chance at continuing your success.”
Hmmm, it seems like there has been a change in attitude towards failure. As a final example we have Matt from Signal vs Noise again in December 2009 stating:
“The idea that you should “fail early, fail often” is bogus. Plans are guesses. Interruption is the enemy of productivity”
What are we to make of this change in advice from some of the most popular names in entrepreneurial advice? Well this study from Harvard published in 2008 might explain some of it. In it the authors claim that entrepreneurs with a track record of success are much more likely to succeed than first-time entrepreneurs or those who have failed before. The primary reason for this is that the successful entrepreneurs exhibit persistence in selecting the right industry and time to start new ventures.
“… all else equal, a venture-capital-backed entrepreneur who succeeds in a venture (by our definition, starts a company that goes public) has a 30% chance of succeeding in his next venture. By contrast, first-time entrepreneurs have only an 18% chance of succeeding and entrepreneurs who previously failed have a 20% chance of succeeding.”
Alas don’t give up hope for redemption from failure, the study does show entrepreneurs who have previously failed have a slightly higher subsequent success rate than first-timers, suggesting that they learn something from their failure.
My take on all this failure talk is that both camps are correct, depending on the magnitude. Failing on small things like a wrong product feature should be done early and often because no matter how much you think you know your users, you nor they know exactly how or what they will use in the future. But you should avoid if at all possible failing on the big things like a product line. These large failures take too much time, energy, and resources that can never be recouped. If you do happen to fail big you can at least take comfort from the fact that there is something to be learned. And that lesson is to fail at the small stuff so that you don’t fail at the big stuff.

Outsourcing Engineering or Operations

Monday, April 26th, 2010

Our clients very often have questions over Why, What and How to outsource software development efforts, infrastructure, hosting, etc.  Readers of our book or frequent readers of our blog will notice that the questions are similar to those we ask in our “Build v. Buy” analysis.   The decision of what to outsource isn’t significantly different than determining when to buy rather than build.

Why outsource?  There are three very good and common reasons to outsource engineering efforts.

1)      You want to reduce your average cost of engineering and outsourcing may provide a way to do that (especially “offshoring”).  The right kind of outsourcing can reduce your unit cost of labor for engineering efforts.  But before you outsource, you should understand the full cost per unit developed of your engineering efforts so that you can measure and validate your cost benefit.

2)      You have near term capacity needs to increase engineering capacity that you cannot meet with current hiring practices.  If you need to 3x the size of your engineering team in 2 months, you probably need outside help.

3)      You fear that the engineering capacity need will be short lived and do not want the risk of hiring W2 employees.  Sometimes (2) and (3) are bundled together.  If you don’t have follow on work for some new system or product, you probably don’t want to hire and then fire employees.

The “What you should outsource” is very often mistaken as “why one should not outsource”.  There are almost always things you can outsource, and very often there are things you absolutely should not outsource.   We typically discuss 4 areas with our clients to help them understand what can and what should not be outsourced.

1)      Don’t outsource things that create strategic competitive differentiation for your company.  Having a third party develop the thing that differentiates you from your competitors is giving away the secret sauce.  It’s hard enough to protect intellectual property – if you simply give it to someone else you might as well just give it away.  Now probably not everything you do differentiates you from competitors.  For instance, if you run an ecommerce site you might determine that your product proposal system is a differentiator while search is not.  Outsource search, keep the development of your product proposal and analytics system in house.

2)      Don’t outsource product definition.  If you are in a product business, you really can’t outsource the definition of the product that makes you money.  We’ve seen customers try and it’s not pretty.

3)      Don’t outsource your architecture or standards.  Tightly coupled with product definition is the need to set the standards and architecture by which the platform abides.   You may believe that the beauty is in the idea or the specification of the product but if it takes off it will need to scale.  Few outsourcers are adept at defining scalable platforms because the largest and best companies simply don’t outsource that – ever.

4)      Don’t outsource areas where you need rapid response and flexibility.  These things might not be competitive differentiators – but if you expect a turn on a dime response in specific areas you aren’t likely to get those with a contractual relationship.

Finally we come to “How you should outsource”.  Here again, we have three common rules for our clients.

1)      Manage the outsourcer.  That means that you need to add employees to manage the outsourcer and the projects, which in turn means that the actual cost of outsourcing is higher than what the outsourcer has quoted.  Keep this in mind when considering outsourcing to dollar average costs down.

2)      Expect conflict.  Rarely do we see outsourced projects that don’t have conflict between internal engineering teams and the outsourced team.  Expect it and be prepared to manage it quickly.

3)      Deliver standards with specifications.  If you expect something to be 99.99% available, scale to 10x the current volume and deliver new functionality be very specific and demand proof.  We’ve even helped negotiate contracts where payment happens after proof in the production environment rather than delivery.

Summary:  Look to outsource when you want to manage the risk of growth or contraction and to lower your engineering costs.  Always expect that you will have to aggressively manage your outsourcer and always deliver specific standards of operation with your product specifications.  Never outsource areas that strategically differentiate your company or product offering or where you need strategic or tactical flexibility.

Crisis Management – Normal Accident Theory and High Reliability Theory

Wednesday, November 18th, 2009

The partial meltdown of TMI-2 at Three Mile Island in 1979 is one of the best known crisis situations within the US and was the source of several books, and at least one movie.  It also generated two theories relevant to crisis management.

Charles Perrow’s Normal Accident Theory (NAT), described in his book Normal Accidents, states that the complexity inherent to tightly coupled technology systems makes accidents inevitable.  Perrow’s hypothesis is that the tight coupling causes interactions to escalate rapidly and without obstruction.  “Normal” is a nod to the inevitability of such accidents.

Todd LaPorte, who founded the Berkeley school of High Reliability Theory, believes that there are organizational strategies to achieve high reliability even in the face of such tight coupling.  The two theories have been debated for quite some time.  While the authors don’t completely agree as to how they can coexist (LaPorte believes that they are complimentary and Perrow believes that they are useful for the purposes of comparison), we believe there is something to be gained from them.

One paradox from these debates becomes intuitively obvious to our pursuit of high availability and highly scalable systems:  The better we are at building systems that avoid problems and crises, the less practice we have in solving problems and crises.  As the practice of resolving failures are critical to our learning, we become more and more inept at rapidly resolving these failures as their frequency decreases.  Therefore, as we get better at building fault tolerant and scalable systems, we get worse at resolving crisis situations that are almost certain to happen at some point.

Weick and Sutcliffe have a solution to this paradox that we paraphrase as “organizational mindfulness”.  They identify 5 practices for developing this mindfulness:

1)      Preoccupation with failure.  This practice is all about monitoring IT systems and reporting errors in a timely fashion.  Success, they argue, narrows perceptions and breeds overconfidence.   To combat the resulting complacency, organizations need complete transparency into system faults and failures.  Reports should be widely distributed and discussed frequently such as in our oft recommended “operations review” process outlined within the Art of Scalability.

2)      Reluctance to simplify interpretations.  Take nothing for granted and seek input from diverse sources.  Don’t try to box failures into expected behavior and act with a healthy bit of paranoia.

3)      Sensitivity to operations.  Look at detail data at the minute level, such as we’ve suggested in our posts on monitoring.  Include the usage of real time data and make ongoing assessments and continual updates of this data.  We think our book and our post on monitoring strategies have some good suggestions on this topic.

4)      Commitment to resilience.  Build excess capability by rotating positions and training your people in new skills.  Former employees of eBay operations can attest that DBAs, SAs and Network Engineers used to be rotated through the operations center to do just this.  Furthermore, once fixes are made the organization should be quickly returned to a sense of preparedness for the next situation.

5)      Deference to expertise.  During crisis events, shift the leadership role to the person possessing the greatest expertise to deal with the problem.  Our book also suggests creating a competency around crisis management such as a “technical duty officer” in the operations center.

We would add that every operations team should use every failure as a learning opportunity, especially in those environments in which failures are infrequent.  A good way to do this is to leverage the post mortem process.

Monetization is king

Tuesday, March 17th, 2009

In an article on the NY Times Deal Book All That Twitters May Not Be Gold, Analysts Say they state “…the Web 2.0 model of building a product and then figuring out how to monetize it has been largely debunked.”  We agree that There Is No Substitute For Profitability.  The article continues, stating “People who sign up for free services tend to resent a company for trying to wring revenue from the business later.”  I’m not sure that is necessarily the case especially ad based models that are not invasive.  I can’t argue that in this economy the sooner any company starts generating revenue the better.