It’s not uncommon to hear from our clients that they have multiple versions of their application running, many of them dedicated to a single “special customer”. While such implementations destroy the economics of a SaaS company, sometimes we must bend a few rules and accept certain risks early in our company’s lives to stay afloat. A startup that is desperate for cash will do just about anything to attract and retain their bigger customers, especially when this customer interaction is lead by sales staff unfamiliar with the hidden costs of customizations.

As technologist, maintaining multiple versions of our SaaS software makes us cringe. We all know that this is a recipe for a maintenance nightmare. Multiple versions require – bug fixes, patches, and features to be tested and deployed on different versions as well as dedicated hardware or virtual instances to run the different versions. The cost of this can quickly double a technology organization’s budget when additional QA, developers, and operations staff are accounted for as well as the environments. In fact, when running a SaaS platform with dedicated instances (remember the original ASP days?) one can quickly create the worst of both the hosted and packaged software worlds and sink a company.  So how do you know when to say “No” and how do you do it?

The way to make decisions, whether build vs buy or feature prioritization or customizations, is to gather the appropriate data to establish a well reasoned argument without exhibiting “analysis paralysis”. For customizations this general includes the upside in terms of future revenue, the downside in terms of development, environment, and maintenance costs, and the risk. No matter what the sales reps are saying there is no guarantee that with the customization you will win the business or without it you won’t. This is the reason for adding a risk variable (beta) that attempts to quantify the likelihood of success or failure.

If the decision is to say yes, then spend some time thinking about what level of customization will occur and how you can possibly reuse the customization. According to authors Pine and Gilmore in “The Experience Economy” there are four types of customizations, transparent, collaborative, adaptive, and cosmetic. Transparent is when the customer isn’t aware that their version is different from others, such as internationalization. Collaborative is when the product or service works with the customer to achieve a custom solution. Adaptive is allowing the customers to change the product or service themselves, such as choosing personal feeds. Cosmetic is re-skinning the product without modifying the functionality.

If you’ve decided to say no to the customer’s request for customization don’t just say “no”. Think about how you can use a carrot and/or stick approach to guide the customer to where you would like them. Sticks are things like additional costs. If a customer wants to stay on a previous version and you’ve decided that’s not acceptable to you consider charging them progressively higher rates until they either leave or make the upgrade. If they are going to leave any way then the temporary additional revenue will help offset the loss. A carrot approach might be refusing to add new features to a custom solution. Eventually the customer, hopefully, will want the newest features and acquiesce on the customization. Instead of just saying “no” emply carrots and sticks in an attempt to move your customer to where you both will be happy.

It’s probably no surprise to our readers that many old packaged software companies are attempting to take their software and hence their business models “online”.  And why not?  The model is attractive and benefits accrue to both the providers of service through software and those who outsource portions of what was once bothersome internally hosted software.  The providers benefit from economies of scale in hosting that generate attractive profits for the provider and savings for the customer, lower maintenance costs resulting from custom customer deployments, predictable revenue streams fostered through closer customer contact, more frequent and smaller releases that reduce risk and faster implementation times that result in faster profit recognition.  Customers benefit from outsourcing non-core IT functions, providers who specialize in delivering specific services, lower capital expenditures and faster deployment times.  SaaS is both a desert topping and a floor wax!  It’s the cure for cancer and the answer to the riddle of life!

But what many of these companies don’t realize is that the way one architects a product and runs a company focused on service delivery is simply different than the approach of a company focused on delivering software.  Customers expect that you are going to give them higher availability and fewer headaches.  Software alone simply won’t meet this goal; it is imperative that one design SaaS systems holistically which in turn requires skills in both infrastructure and software architecture (or “systems” architecture).   The cost leverage necessary to both increase profit margins and decrease customer cost typically requires multi-tenancy which has its own share of headaches.  Fault isolation and rollback capabilities are a must to minimize customer impact and mitigate rapid deployment risks.

It is not enough to simply bundle up an application in a hosted fashion and label yourself a “SaaS” company.  If you don’t work aggressively to increase availability and decrease your cost of operations, someone with greater experience will come along and simply put you out of business.  After all, your business is now about SERVICE – not SOFTWARE.  This is a fundamental mind-shift that some companies simply can’t overcome or maybe simply don’t recognize.  This isn’t to say that a good engineer or product manager can’t be equally good at developing packaged and SaaS applications, but it does mean that the approach is completely different.

Stop trying to figure out how to leverage your existing assets with minimal work and start thinking about having two different products.  Or, determine which business you want and kill the other one off.  If you decide to keep both products alive, you can share services and code between these platforms, but you should not do so at the expense of optimizing your SaaS solution.  Attempting to satisfy both with a single architecture will likely result in you failing at both.

I saw this Nike commercial with Michael Jordan talking about how much he had failed in his career and this got me thinking about the debate of whether or not entrepreneurs should “fail early, fail often” as the saying goes. A quick Google timeline shows how popular this term has grown in our vernacular over the past two decades. No doubt following the trend of high tech entrepreneurial failures.

Digging deeper into some of the more popular entrepreneurial technology bloggers, I found one of the earliest references to failing early from Jeff Atwood from Coding Horror. In May ’06 he stated:

“The best software developers embrace failure — in fact, they’re obsessed with failure. If you forget how easy it is to make critical mistakes, you’re likely to fail. “

“I say the more failed projects in your portfolio, the better. If you’re not failing some of the time, you’re not trying hard enough. You need to overreach to find your limits and grow. But do make sure you fail in spectacular new ways on each subsequent project.”

“Getting Real is all about iterations too. “Be excited to be wrong because then you’ve discovered something new” is a neat way to put it (btw, so is Fail early, fail often).”

When asked by a team what they should prepare for an engagement with us, we usually tell them to not prepare anything. Instead of PowerPoint slides showing the architecture, network, etc we prefer for people to jump to the white board and draw. One of the primary reasons is that we often find people debating how the architecture actually exists. How does your architecture diagrams or institutional knowledge reflect reality of the software?

In the May issue of Computer, there is an article, “Evolving Software Architecture Descriptions of Critical Systems”, by Tom Mens, Jeff Magee, and Bernhard Rumpe, in which the authors’ state:

An explicit architecture description is important but not sufficient to manage the complexity of developing, maintaining, and evolving a critical software-intensive system.

The authors continue explaining that the architecture description must be accurate and traceably linked to the actual implementation in software so that changes in the architecture are reflected in implementation and vice versa.

If your team has spent a bunch of time creating an architecture that will scale, all that effort is wasted when the software implementation doesn’t abide by the architecture. Because of the ever evolving nature of complex software systems it is admittedly difficult to keep the architecture description and software artifacts aligned.  The authors of the article suggest that evolving architecture descriptions requires co-evolution of different viewpoints such as the structural and behavioral. To this I completely agree but they address the solution to this issue from the aspect of Architecture Description Languages (ADLs). The problem with this approach is that I don’t know of many, if any, SaaS companies using ADLs. Therefore, in order to accomplish this co-eveolution of software artifacts and architecture descriptions we have to seek a different solution.

To ensure that architectural changes are reflected in the software we typically suggest that companies rely on architecture principles. We’ve dedicated an entire chapter in The Art of Scalability to this subject but I’ll try to summarize it here. Architectural principles are a set of ideas that the team has determined when used as guidelines during the design and development of the software will yield a scalable, available, and cost effective system. Principles should help influence the behavior and the culture of the team. We often use the SMART acronym to describe good principles as being Specific, Measurable, Achievable, Realistic, and Testable.

So how about the other direction, how do we ensure the architecture description accurately reflects the software? By using JAD and ARB processes, which we’ve covered in detail before on this blog as well as in the book, we can help ensure that software artifacts that deviate from the established architecture are discussed and noted by the appropriate individuals and teams.

Assuming you have a fair coin, that is just as likely to land heads as tails, the probability of getting a heads on a single flip is 50%.  Now two questions.  First, what is the chance of getting two heads in a row?  Second, what is the chance of the next flip being heads?  While these two questions seem similar they are very different.  To answer the first question, two heads in a row, we can look at all the possible combinations of two coin tosses:

(H,H) (H,T) (T,H) (T,T)

With four possible outcomes one of which is our two heads (H,H), we can easily compute that we have a 25% chance of getting two heads in a row. Another way of computing this is by multiplying the probability of getting a head on each coin toss. Because each toss is independent the likelihood of getting a head is 50% for each. We therefore have 50% * 50% = 25%.

This gets us to the second question, what is the probability that the next flip is a heads. As mentioned above, and contrary to what gamblers and sportscaster often believe, each flip is independent there is no “law of averages” that would indicate heads or tails is more likely. The first flip and the second flip and the third and each subsequent flip are independent of each other. However, we do expect that as the number of flips gets large the porportion of heads and tails approaches 50%. Another way to look at this is to go back to our diagram above.  If our first flip was a heads which of the scenarios could exist for our second flip? The answer is the scenarios of (H,H) (H,T) because they both have ‘H’ as their first flip.  Therefore the chance that the second flip is a tails is 1 out of 2 or 50%.

Back to our hot fix scenario. Let’s ignore the probability that our fix actually solved the problem and just focus on the likelihood of a problem occurring each day this week. We can get into prior probabilities in a later post. As we discussed above, independent events maintain their probability for each event so the probability of the problem occuring today is 50%, that it occurs tomorrow is 50%, and so on. A different question is, what is the probability that the problem will not occur three days in a row? Let’s first look at this visually using N = No Problem and P = Problem.

(N, N, N) (N, N, P) (N, P, N) (P, N, N) (N, P, P) (P, N, P) (P, P, N) (P, P, P)

From the figure above we can see that there is only 1 out of 8 scenarios where we have No Problem three days in a row. This equates to 1/8 = 0.125 or 12.5%. We can also solve this as we did above by multiplying our probability each day 50% * 50% * 50% = 12.5%.

One final note about independent failure events. We often tell clients to avoid synchronous calls because if one service fails (because of hardware or software) it causes others to fail. If you have 99% uptime on one service and 99% uptime on another but they are both required to service a request, the total system availability is 99% * 99% = 98% unless of course they happen to fail at the exact same time every time.  This is what we call the multiplicative effect of failure.