AKF Partners

Abbott, Keeven & Fisher PartnersPartners In Hyper Growth

Category » CEO

Availability as a Feature

It doesn’t matter if you run a commerce site, a services product (such as a SaaS offering) or simply use your homepage to distribute information:  The table stakes for playing online is high availability.  So many companies just take for granted that they will be highly available because they have multiple instances of systems and multiple copies of their data.  This assumption of availability will likely, at the very least, cost you significant pain and in the extreme cost you either significant market share or close your doors as a business.  Customers expect the unachievable – 100% availability.  At the very least you need to give them something close to that.  What will happen to you if you have a data center failure?  How about if a DBA accidentally drops a critical table in your production database?  What will you do when that marketing campaign triggers a near overnight doubling of traffic?  What happens when that new feature has a significant performance bug and gets adopted so quickly that it brings your entire site to its knees?

We often tell our clients that they should treat high availability as a feature.  Unfortunately, it is a somewhat expensive feature that requires constant investment overtime to achieve and maintain. It is a must have feature that will only differentiate your firm if you have competitors who do not make similar investments; when competition exists, customers are more likely to leave a site for a competitor due to availability and performance issues than nearly any other reason.  If you don’t believe us on this topic, just go ask the folks at Friendster.

Treating availability as a feature means measuring availability from a customer perspective rather than a systems or device perspective.  How many times did customer requests not complete?  In this regard, availability now becomes a percentage of failed transactions against an expected number of transactions.   We define an approach to accomplish this in our first book “The Art of Scalability”.  Every executive in the company should “own” the availability metric and understand its implication to the business.    You should track how much you invest in availability over time and significant decreases in engineering or capital should be questioned as it may be an early indicator that you are under investing and a harbinger of hard times to come.

One of the most common failures we see is to assume that disaster recovery is something that only big companies need.  Make no mistake about it, disasters do happen and given enough time they will happen to you.  Data centers catch on fire, have water (sprinkler) discharges that ruin equipment, have complete power equipment failures that take hours to fix and are prone to damages from vehicles, earthquakes, employees and tornados.  In our past lives as executives and current roles as advisors we’ve seen no less than 4 data center fires, 2 data centers incapacitated from earthquakes and tornados and one data center leveled by a truck running into it.  You are never too young to invest in DR.

And DR need not break your bank.  The dials of RTO (recovery time objective) and RPO (recovery point objective) allow you to determine how much you will invest.  Perhaps you simply replicate your databases to a smaller set of databases at a remote datacenter and have a copy of each of your systems there with an additional copy ready “in the cloud”.  While you won’t be able to run production from that data center, you may be able to leverage the cloud to add capacity for relatively low cost by cloning the cloud based systems.  Such a solution has a fast recovery point objective (you lose very little data) and a moderate recovery time objective (several hours) for very low comparative cost.  Of course, you would need to test the solution from time to time to show that it is viable, but it’s a cheap and effective insurance policy for the business.

So remember – availability is your most important feature.  Customers expect it always and will run away from you to competitors if you do not have it.  Create an availability metric and ensure that everyone understands it as a critical KPI.  Evaluate the company spend against availability quarterly or annually as an additional indicator of potential problems.   Assume that disasters happen and have a DR plan regardless of your company size.


Comments Off on Availability as a Feature

Mergers and Acquisitions Revisited

We wrote a post last Sept about successful acquisitions. In that post we first struggled with how to actually define a “successful” acquisition. After that we postulated that there were two primary methods of achieving what would in general be considered a successful outcome from an acquisition.

The first method is by overwhelming the acquisition’s culture and turning it into the acquiring culture as fast as possible. I called this the GE-approach because of all the acquisitions I saw while at General Electric during the 90’s, this appeared to be the dominant strategy. The second approach is to leave the acquisition completely alone and let it run autonomously. The only tie to the acquiring company is through financials. Reading an article recently I was shocked but pleased to see that academic research had arrived at similar strategies for successful mergers and acquisitions.

Clayton Christensen, Harvard professor and author of books such as The Innovator’s Dilemma, wrote an article recently in HBR with several other authors about the new M&A playbook. In this article the authors state that studies indicate that mergers and acquisitions fail over 70% of the time. Much has been written and studied about this but from the perspective of attributes of the deal. Christensen et al suggest that the problem isn’t attributes of the deal but that executives fail to match candidate acquisitions to the strategic purpose of the deal.

The article states that there are two reasons to acquire a company, to boost your company’s current performance or to reinvent your business. To extend your business but not fundamentally change how you compete, an executive should buy a company with resources aligned with the current business and fold them in, letting the acquisition eventually die. This is what we described as “overwhelming the acquisition’s culture” or the GE-approach. To reinvent your business, executives should seek companies that have a different business model put resources into it and let it grow. This is what I see as our approach of leaving the acquisition alone letting it continue to grow, perhaps providing financial resources from the parent company.


“Internal Customer”: The “C” Word of SaaS Companies

If you are a technology organization within a Software as a Service (SaaS) company, there is no such thing as an “internal customer”.

If you are a technology organization within a Software as a Service (SaaS) company, there is no such thing as an “internal customer”.  We often see this anachronistic IT phrase thrown around in web X.0 companies by executives and engineers who simply have not adopted the new SaaS mindset.  Do you think you’ll hear the left offensive tackle of an NFL team refer to the quarterback as his “internal customer”?  The quarterback consumes services (energy to block opponents) of the left tackle – so why wouldn’t he be a customer?  The answer is simple – because the notion of a customer relationship is different than the notion of a relationship within teammates.

The first reason why your teammate isn’t your customer is because he or she is, well, your TEAMMATE.  Customers are someone for whom you produce a service or product and teammates are someone with whom you work to accomplish a goal.  The difference between working FOR someone and working WITH someone is HUGE.  This difference creates a contextually activated identity that forces you to think about customers in a different light than you would a teammate.  Very often, as we’ve written before, this can result in affective (role based or bad) conflict between teams.  Affective conflict is bad and it destroys shareholder value.  Working as a team is important and customers aren’t part of your team.

The next reason that your teammate isn’t your customer is that the customer is always right.  Your teammate isn’t always right.  You need to debate certain points as a team to come to better solutions.  This isn’t affective conflict, it is cognitive conflict and if handled properly it is good and helps to create shareholder value.

The most important reason there isn’t a customer relationship here is that your teammate isn’t paying!   “Servicing” your teammate (uggh…that’s an ugly term) doesn’t create shareholder value.  Working as  a team to delivery a service or product to your  “real” customer is what creates shareholder value.  One design, one approach, one ruthless drive as a team to get across the goal line is what is necessary to thrive and succeed.

So stop using the ugly “internal” C word in your SaaS company.  It doesn’t have a place there.  Let the old world, internal IT folks continue to provide services to their internal customers.  Start acting like a team, designing and building services rather than software.


Successful Acquisitions

How do you successfully integrate an acquisition? Most academic research on this subject seems to suggest a rational choice perspective which focuses on either strategic fit or organizational fit.

The strategic fit camp emphasizes strategic analysis and negotiation prior to the acquisition whereas the organizational fit research emphasizes the integration of day-to-day operations post acquisition. There has even been recent research on the impact of the acquisition process itself, which makes sense given that first impressions are established during this phase. I’ve had the good fortune of seeing acquisitions from all different perspectives, as the acquired, as the acquiree, as a consultant of the acquiree, etc. I think there are two models that result in successful acquisitions.

First off defining a successful acquisition is tough. Whether or not the acquiring company writes off the purchase cost in a few years might be one way but these write offs can include goodwill which is the amount of the purchase price paid above the value of the target’s identifiable net assets. There are of course accounting rules for testing impairment of goodwill that may require it to be written off. Another way of testing this might be by calculating Return On Investment (ROI) for the cost of the acquisition. A third way of determining success would be whether the acquired company’s product offerings continue 3 or 5 years later. If the acquiring company has discontinued the acquisitions products the purchase might be considered a failure. Lastly, I tend to think of a successful acquisition like Justice Potter Stewart tried to explain what is obscene by saying “I shall not today attempt further to define the kinds of material I understand to be embraced … but I know it when I see it.”

Whether there is a clear definition or I know it when I see it, there appear to be two paths to get there. The first is what I term the GE-approach because of all the acquisitions I saw while at General Electric during the 90’s, this appeared to be the dominant strategy, although I’m sure many other companies have similar methods. It’s simply an approach of overwhelming the acquisition’s culture and turning it into a GE culture as fast as possible. They did this by sending current GE employees into key positions at the acquisition and sending the acquisitions employees to lots of GE training. Within the first year the acquisition was on the GE strategic planning calendar, using GE financial systems, and following GE’s HR guidelines…they were a GE company by that point.

The second approach is to leave the acquisition completely alone and let it run autonomously. The only tie to the acquiring company is through financials. The parent company acts like a board of directors, approving annual budgets and determining the reinvestment ratio. The acquiree is left to their own to manage the day-to-day operations, processes, etc.

If you’ve seen other successful acquisitions handled differently let us know your thoughts.

Comments Off on Successful Acquisitions

Chief Innovation Officer – Organization #Fail

If I get another phone call from another recruiter about a “Chief Innovation Officer” search, I may just hang myself from my hotel light fixture with my belt.  Don’t get me wrong – I’m not angry at the recruiter (other than for wasting some of my time) – I’m mad at the CEO or person advising the CEO who come up with such a lame title.  Why am I so upset over the name? Pull up a chair and grab a cup of coffee.  You don’t want to miss this rant.

My first gripe is with the name of this “position”.  The name just sucks and flies in the face of everything we know about innovation.  In some companies, the Innovation Officer might just be a rebranded VP or SVP of product.  I’m almost okay with that – except for the fact that in this case it is a complete misrepresentation of what the person is doing.   If the person is something other than the head product person, well, that’s when this position really is nothing more than a shareholder wealth incineration device.

This moronic title seems to imply that there is a person “responsible” for innovation within a company.  Anyone who believes that one person can somehow be responsible for innovation is clueless about whence innovation comes.  You can no more “control” innovation than you can domesticate a rabid dog hyped up on meth.  You can certainly “kill” innovation, just as you can put down that rabid junkie dog – and giving someone a business card with Chief Innovation Officer on it is like putting a 240 grain slug in that dog.  Goodbye junkie dog and goodbye innovation.  The dog will probably die a more humane death however, as the Chief Innovation Officer will kill innovation under the crushing weight of power point presentations and spreadsheets.

Innovation simply can’t be controlled.  It can be “captured”, it might be “harvested” and it certainly can be “found”.   In some cases, it might even be able to be “directed” with the right questions and incentives.   But “controlled”?  Give me a break.  Capturing, harvesting and finding are all by the company culture – not a single individual.  And as we all know, that culture is most often affected by the CEO and to a lesser degree by each level below the CEO.  This is one of those things, then, that the CEO can’t delegate.  It’s not a position – it’s a shared responsibility.  How do you incent people to innovate?  How do you find little pockets of innovation that hide within the organization but never become apparent because your processes require it to be presented within a 40 page Power Point deck?

My partner, Tom Keeven, knows how.  Tom is a man of varied interests.  One interest is running a pizza parlor in Carmel.  His employees come to him with suggestions all the time.  “Let’s start a delivery service”, “Why don’t we make a low carb pizza dough?”, “Let’s advertise in local hotels”.  Tom doesn’t tell them to run off and build a presentation – he listens to them with great interest.  He rewards them for great insights.  He tells them to experiment and try things out.

Google allows engineers to work on interesting problems with great benefit to the innovation process.  Granted, few of these innovations have paid off and the resulting shareholder dilution is very large.  But the process of bottoms -up, “grass roots” innovation produces results in terms of number of innovations brought to market.  Sooner or later one of them will pay off – the question is “at what cost”.

In summary, innovation isn’t an organization, it’s not a title and it’s certainly not something controlled from the top.  There are things you can do to nurture it and maybe even slightly direct it – but there is little you can do to control it.   You can’t hire a person to control it and you are just wasting your time if you expect to try to create an organization to “innovate”.  If you want your team to innovate, work on creating a risk tolerant culture and work to incent your team for their innovation efforts.  Lower the cost of innovation by eliminating ridiculously complex and burdensome innovation hurdles like executive presentations.  Change the ridiculous titles of your innovation officers to something useful and stop wasting shareholder wealth.


Please Be Quiet

Are you allowing your product or service to fail because someone else doesn't understand what it takes for you to succeed?

I’ve noticed lately that more companies are putting up signs in hallways and cube farms requesting that people avoid having conversations in these areas. While having a nice quiet work environment makes sense to me as a developer, doesn’t this completely void having people work beside each other? The ad hoc/hallway/water cooler/coffe machine conversations or ones overheard when cube mates are chatting about a new feature are one of the primary benefits of having people work in small open environments.

I haven’t done any sort of scientific study but it seems that these sort of “please be quiet” signs are more prevalent at larger companies. These are the same ones that are trying to mimic the small startup with agile development processes or open work spaces to compete in a fast moving SaaS marketplace. Imitating the actions without understanding the purpose or allowing old school corporate policies to overrule are surefire ways to tank the initiative.

A parallel to the “please be quiet” sign is allowing corporate IT to dictate the architecture of the SaaS offering based on a corporate standard that works for the ERP system. Running Oracle ERP on a 16-way system might be the vendor recommended, preferred approach but for scaling a SaaS offering this is a quick way to run up the costs and ensure lower availability. We often use the analogy of goldfish and thoroughbreds for comparing small, cheap 1U servers with large, expensive multi-processor boxes with lots of memory. The goldfish (small, cheap servers) are inexpensive to purchase and replaceable while the thoroughbred (large servers) are expensive to purchase/maintain and cause big impacts when they go down.

The take away to all of this is that if your part of a corporate initiative to run an internal startup or deliver a Software as a Service from inside a larger organization, don’t allow corporate policies to prevent your success. The differences in approaches, architectures, organizations, and offices have a purpose and should not be discounted as non-critical to the success of your initiative.


Fail Early

You've probably heard it before that you should "fail early and fail often" but should you really?

I saw this Nike commercial with Michael Jordan talking about how much he had failed in his career and this got me thinking about the debate of whether or not entrepreneurs should “fail early, fail often” as the saying goes. A quick Google timeline shows how popular this term has grown in our vernacular over the past two decades. No doubt following the trend of high tech entrepreneurial failures.

Digging deeper into some of the more popular entrepreneurial technology bloggers, I found one of the earliest references to failing early from Jeff Atwood from Coding Horror. In May ’06 he stated:

“The best software developers embrace failure — in fact, they’re obsessed with failure. If you forget how easy it is to make critical mistakes, you’re likely to fail. “

And Jeff continues in the same post:

“I say the more failed projects in your portfolio, the better. If you’re not failing some of the time, you’re not trying hard enough. You need to overreach to find your limits and grow. But do make sure you fail in spectacular new ways on each subsequent project.”

Next up that same year in September, Matt from Signal Vs. Noise, the 37Signals blog, quoted Jonathan Ive, VP of Industrial Design, Apple:

“Getting Real is all about iterations too. “Be excited to be wrong because then you’ve discovered something new” is a neat way to put it (btw, so is Fail early, fail often).”

Fast forward to February 2009, Jason, also from Signal vs. Noise states:

“It’s true: Everything is a learning experience. Good and bad, there’s something to be learned. But all learning isn’t equal. I’ve found that if you’re going to spend your time pondering the past, focus on the wins not the losses. The lessons learned from doing well give you a better chance at continuing your success.”

Hmmm, it seems like there has been a change in attitude towards failure. As a final example we have Matt from Signal vs Noise again in December 2009 stating:

“The idea that you should “fail early, fail often” is bogus. Plans are guesses. Interruption is the enemy of productivity”

What are we to make of this change in advice from some of the most popular names in entrepreneurial advice? Well this study from Harvard published in 2008 might explain some of it. In it the authors claim that entrepreneurs with a track record of success are much more likely to succeed than first-time entrepreneurs or those who have failed before. The primary reason for this is that the successful entrepreneurs exhibit persistence in selecting the right industry and time to start new ventures.

“… all else equal, a venture-capital-backed entrepreneur who succeeds in a venture (by our definition, starts a company that goes public) has a 30% chance of succeeding in his next venture. By contrast, first-time entrepreneurs have only an 18% chance of succeeding and entrepreneurs who previously failed have a 20% chance of succeeding.”

Alas don’t give up hope for redemption from failure, the study does show entrepreneurs who have previously failed have a slightly higher subsequent success rate than first-timers, suggesting that they learn something from their failure.
My take on all this failure talk is that both camps are correct, depending on the magnitude. Failing on small things like a wrong product feature should be done early and often because no matter how much you think you know your users, you nor they know exactly how or what they will use in the future. But you should avoid if at all possible failing on the big things like a product line. These large failures take too much time, energy, and resources that can never be recouped. If you do happen to fail big you can at least take comfort from the fact that there is something to be learned. And that lesson is to fail at the small stuff so that you don’t fail at the big stuff.

Comments Off on Fail Early

Outsourcing Engineering or Operations

A quick summary of AKF Partners' approach of what, why and how to outsource engineering efforts.

Our clients very often have questions over Why, What and How to outsource software development efforts, infrastructure, hosting, etc.  Readers of our book or frequent readers of our blog will notice that the questions are similar to those we ask in our “Build v. Buy” analysis.   The decision of what to outsource isn’t significantly different than determining when to buy rather than build.

Why outsource?  There are three very good and common reasons to outsource engineering efforts.

1)      You want to reduce your average cost of engineering and outsourcing may provide a way to do that (especially “offshoring”).  The right kind of outsourcing can reduce your unit cost of labor for engineering efforts.  But before you outsource, you should understand the full cost per unit developed of your engineering efforts so that you can measure and validate your cost benefit.

2)      You have near term capacity needs to increase engineering capacity that you cannot meet with current hiring practices.  If you need to 3x the size of your engineering team in 2 months, you probably need outside help.

3)      You fear that the engineering capacity need will be short lived and do not want the risk of hiring W2 employees.  Sometimes (2) and (3) are bundled together.  If you don’t have follow on work for some new system or product, you probably don’t want to hire and then fire employees.

The “What you should outsource” is very often mistaken as “why one should not outsource”.  There are almost always things you can outsource, and very often there are things you absolutely should not outsource.   We typically discuss 4 areas with our clients to help them understand what can and what should not be outsourced.

1)      Don’t outsource things that create strategic competitive differentiation for your company.  Having a third party develop the thing that differentiates you from your competitors is giving away the secret sauce.  It’s hard enough to protect intellectual property – if you simply give it to someone else you might as well just give it away.  Now probably not everything you do differentiates you from competitors.  For instance, if you run an ecommerce site you might determine that your product proposal system is a differentiator while search is not.  Outsource search, keep the development of your product proposal and analytics system in house.

2)      Don’t outsource product definition.  If you are in a product business, you really can’t outsource the definition of the product that makes you money.  We’ve seen customers try and it’s not pretty.

3)      Don’t outsource your architecture or standards.  Tightly coupled with product definition is the need to set the standards and architecture by which the platform abides.   You may believe that the beauty is in the idea or the specification of the product but if it takes off it will need to scale.  Few outsourcers are adept at defining scalable platforms because the largest and best companies simply don’t outsource that – ever.

4)      Don’t outsource areas where you need rapid response and flexibility.  These things might not be competitive differentiators – but if you expect a turn on a dime response in specific areas you aren’t likely to get those with a contractual relationship.

Finally we come to “How you should outsource”.  Here again, we have three common rules for our clients.

1)      Manage the outsourcer.  That means that you need to add employees to manage the outsourcer and the projects, which in turn means that the actual cost of outsourcing is higher than what the outsourcer has quoted.  Keep this in mind when considering outsourcing to dollar average costs down.

2)      Expect conflict.  Rarely do we see outsourced projects that don’t have conflict between internal engineering teams and the outsourced team.  Expect it and be prepared to manage it quickly.

3)      Deliver standards with specifications.  If you expect something to be 99.99% available, scale to 10x the current volume and deliver new functionality be very specific and demand proof.  We’ve even helped negotiate contracts where payment happens after proof in the production environment rather than delivery.

Summary:  Look to outsource when you want to manage the risk of growth or contraction and to lower your engineering costs.  Always expect that you will have to aggressively manage your outsourcer and always deliver specific standards of operation with your product specifications.  Never outsource areas that strategically differentiate your company or product offering or where you need strategic or tactical flexibility.

1 comment

Crisis Management – Normal Accident Theory and High Reliability Theory

The partial meltdown of TMI-2 at Three Mile Island in 1979 is one of the best known crisis situations within the US and was the source of several books, and at least one movie.  It also generated two theories relevant to crisis management.

Charles Perrow’s Normal Accident Theory (NAT), described in his book Normal Accidents, states that the complexity inherent to tightly coupled technology systems makes accidents inevitable.  Perrow’s hypothesis is that the tight coupling causes interactions to escalate rapidly and without obstruction.  “Normal” is a nod to the inevitability of such accidents.

Todd LaPorte, who founded the Berkeley school of High Reliability Theory, believes that there are organizational strategies to achieve high reliability even in the face of such tight coupling.  The two theories have been debated for quite some time.  While the authors don’t completely agree as to how they can coexist (LaPorte believes that they are complimentary and Perrow believes that they are useful for the purposes of comparison), we believe there is something to be gained from them.

One paradox from these debates becomes intuitively obvious to our pursuit of high availability and highly scalable systems:  The better we are at building systems that avoid problems and crises, the less practice we have in solving problems and crises.  As the practice of resolving failures are critical to our learning, we become more and more inept at rapidly resolving these failures as their frequency decreases.  Therefore, as we get better at building fault tolerant and scalable systems, we get worse at resolving crisis situations that are almost certain to happen at some point.

Weick and Sutcliffe have a solution to this paradox that we paraphrase as “organizational mindfulness”.  They identify 5 practices for developing this mindfulness:

1)      Preoccupation with failure.  This practice is all about monitoring IT systems and reporting errors in a timely fashion.  Success, they argue, narrows perceptions and breeds overconfidence.   To combat the resulting complacency, organizations need complete transparency into system faults and failures.  Reports should be widely distributed and discussed frequently such as in our oft recommended “operations review” process outlined within the Art of Scalability.

2)      Reluctance to simplify interpretations.  Take nothing for granted and seek input from diverse sources.  Don’t try to box failures into expected behavior and act with a healthy bit of paranoia.

3)      Sensitivity to operations.  Look at detail data at the minute level, such as we’ve suggested in our posts on monitoring.  Include the usage of real time data and make ongoing assessments and continual updates of this data.  We think our book and our post on monitoring strategies have some good suggestions on this topic.

4)      Commitment to resilience.  Build excess capability by rotating positions and training your people in new skills.  Former employees of eBay operations can attest that DBAs, SAs and Network Engineers used to be rotated through the operations center to do just this.  Furthermore, once fixes are made the organization should be quickly returned to a sense of preparedness for the next situation.

5)      Deference to expertise.  During crisis events, shift the leadership role to the person possessing the greatest expertise to deal with the problem.  Our book also suggests creating a competency around crisis management such as a “technical duty officer” in the operations center.

We would add that every operations team should use every failure as a learning opportunity, especially in those environments in which failures are infrequent.  A good way to do this is to leverage the post mortem process.


Monetization is king

In an article on the NY Times Deal Book All That Twitters May Not Be Gold, Analysts Say they state “…the Web 2.0 model of building a product and then figuring out how to monetize it has been largely debunked.”  We agree that There Is No Substitute For Profitability.  The article continues, stating “People who sign up for free services tend to resent a company for trying to wring revenue from the business later.”  I’m not sure that is necessarily the case especially ad based models that are not invasive.  I can’t argue that in this economy the sooner any company starts generating revenue the better.