- I can finally focus on product development and software engineering and not worry about this infrastructure stuff.
The notion that IaaS providers like Amazon have eliminated your worries about infrastructure is only partially true. You may not need to understand everything about designing an infrastructure with bare metal but you need to make sure you understand how your virtual configuration in the Cloud will affect your product. IaaS helps us to quickly deploy infrastructure for our products but it doesn’t eliminate the need for good high availability and fault tolerance design. There are several levers you can pull within an IaaS console and design decisions that will impact your products performance. To ensure good design and configuration, its ultra important that your SaaS product engineering team is made up of talent that has expertise in distributed application architecture design, infrastructure, security, and networking. Having this knowledge will help you design a high performing, fault isolated product for your business.

- Going to the cloud pretty much guarantees me high availability because of auto scaling.
Going to the cloud will provide you with the ability to scale quickly as load increases but it will not provide you with high availability. For example, if you have a monolithic code base that you deployed and you are pushing to production on a regular basis, there is a pretty good chance you will introduce a defect at some point that impacts the availability of your entire service and business. We advise our clients to split their applications appropriately, deploy the services to separate instances, and, assuming you are using Amazon, configure them to run across multiple zones within a region at a minimum (preferably across regions). This allows you to focus dedicated teams to the individual services and reduce the likelihood of introducing a defect that takes down your system.

- The Cloud will be cheaper than a collocation or managed hosting provider.
There are several factors that need to be considered before you can confirm that is cost effective. You should look closely at load on your servers. If your servers are not serving traffic around the clock, it may be better from a cost perspective for you to buy and maintain your own infrastructure in a collocation or in an existing data center you may have. The economics of this decision is changing rapidly as IaaS pricing is declining due to the competition in the industry. A simple spreadsheet exercise will help determine if the move to Cloud would be cost effective for your business.

- The Cloud isn’t secure so we better not use it.
The cloud isn’t necessarily what makes or breaks security around your SaaS product. Many believe that public cloud services like Amazon’s EC2 service isn’t secure. First off, you are far more likely to experience a security breach because of an employee’s actions (either intentionally or unintentionally) than caused by an infrastructure provider. Secondly, Amazon likely has invested much more in security at various layers, including their physical data centers, than most companies we see who have their own data centers. Amazon has designed the infrastructure to isolate customer instances and you can also choose to take advantage of Amazon Virtual Private Cloud that can be configured to create an isolated network. There are various options for encrypting all of your data as well. This only touches the surface for security design options you have and they continue to be enhanced everyday. You can see why it’s important to staff your team with an engineer who has experience in this space.

If you are looking to move to Cloud, don’t rush into the decision. Do your homework and make sure it’s right for your business. Make sure you have the talent that has experience with the technology that will get you there and run your operations. Once you make the leap, you will have to live with it for a while.

The Art of Scalability

Scalability Rules

The allure of IaaS & PaaS is firmly rooted in the concept of enablement. Self-service is an amazing if you are a DBA, developer or even the end user of your product. The cloud may not be suitable for your needs but don’t let that stop your organization from thinking strategically about bringing those processes & technologies “in-house” for scalability reasons. Implemented correctly, the infrastructure and platform you are building should enable the users and not hinder them, as is sometimes the case. Reducing the number of dependencies between technology teams for launching products is not only good for cycle time of product launches but also critical in scaling up.

Consider making enablement part of your technology team’s DNA and you will likely see that employee morale, productivity and other metrics like NPS will rise.

The Art of Scalability

Scalability Rules

A great opportunity came for me in 2004 to join a still young PayPal. It was by no means a startup anymore but the entire engineering organization fit into the same medium sized room at that time. It was then that I first learned how important culture is and how vital it is to maintain it as you scale up. Over the course of 9 years I did my best to help hire many team members that would not only help the business achieve its’ goals but maintain & grow that unique culture.

As I embark on a new chapter in my career here at AKF Partners I’m excited to work with extremely talented people both at AKF and with the organizations we serve. I’m excited to learn about the people & cultures that make these companies so successful and to help them scale.

However, priority #1 is clear enough: don’t dilute, add to it!

The Art of Scalability

Scalability Rules

The biggest mistake companies make implementing Agile and thus the cause of most of their problems is they don’t understand that Agile is a business process, not a software development methodology. Thus, the business owners or their delicates, product managers, must be involved at every step.

We’ve argued before that Agile teams must sit together because communication degrades at a rate of square the distance. Not having product managers with the Agile team involved in the entire process and (if you’ve moved from a Waterfall methodology) not having detailed specifications, is the worst possible scenario. Developers either need someone siting beside them to help with product decisions (Agile) or a detailed spec to work from (Waterfall).

Agile is a business process which requires the business to be involved in the product development process. It does not mean you get to stop writing specs and not be involved.

The Art of Scalability

Scalability Rules

1. The Future of IaaS and Paas
2. JAD and ARB
3. Dealing With Shared Services
4. Be A Leader
5. Fault Isolation

Not too surprising, our readers are interested in how to scale via clouds, better architectures, and improved processes. They are also interested in leadership, since without that scaling a great organization becomes impossible.

Let us know what your favorite post is. You can find a list of all our posts here.

The Art of Scalability

Scalability Rules

Although I didn’t say the words “get onboard,” I might as well have. I can tell you this doesn’t work well. My team was zapped. At that point they were no longer empowered and didn’t have any ownership of the change and when that happens a team’s performance will be suboptimal. Yeah, I could use the excuse that my senior management dictated what they wanted and I could drum up a bunch of other excuses but the fact of the matter is I should have raised a flag with my leadership and encouraged better collaboration for input from the engineers, QA testers, product managers, and others.

There were two key learning points that I was able to take away from my experience. Technology leaders should ask themselves two simple questions before they start to drive change in their organizations.

1) How do I make sure that my team has input and ownership of the change we are about to make?

Sit down with the team and work with them on coming up with solutions and their recommended timing. Engineers love to problem solve and they are very good at it. Provide the team with the problem that needs solved and any constraints and let them determine the solution. For example, in the situation I described above, we needed to improve our time to market. Many products and features across the organization were being delivered late, payloads for deployments were too big, and the business was frustrated. As on overall company we decided that we would adopt Agile utilizing Scrum practices but there were still many problems that needed to be solved. Those included determining a branching strategy, determining a deployment strategy, identifying how to measure a sprints performance, and many others. By providing the context of why changes need to be made, the team will recognize the significance of their input and by allowing them to come up with solutions they will feel empowered and have ownership in the game. All of this will help you to create a higher performing engineering team.

2) What do I need to do as a leader to establish guardrails in my organization and sustain the change as my company grows?

Once the team has determined how to solve some of the problems, you will want to make sure that you sustain them in your organization. To do this, work with them to establish documented guardrails or principles that be used as the what. For example, the Agile Manifesto essentially identifies principles that you can use. One principle could be “Business people and developers must work together daily throughout the project.” This describes a behavior that’s needed but doesn’t identify how. The team can decide the specifics on how they will achieve the principle. We at AKF believe in establishing architectural and scaling principles that are essentially guardrails as well. (Visit our earlier blog on architectural principles for more details.) For example, Rule Number 39 – Ensure You Can Wire on and Off Functions in our book Scalability Rules identifies the need to have a framework where you can quickly disable and enable features to minimize impact to customers when a failure occurs. You can solve this many ways and your engineers can determine what the solution should be. Once your principles are established you should monitor them to make sure they are being upheld and reward results.

If you ask yourself these two simple questions before diving into a big change your road to success will be much easier. Now that you have empowered your team to determine how to solve the problem and you have documented your principles and guardrails, the team should be able to operate within them and make decisions on their own. In turn, you will find that they feel empowered and you will have helped to create and maintain a higher performing team.

The Art of Scalability

Scalability Rules

I have been involved, in some form or another, in scaling technology systems for nearly two decades, which does not take into account the decade before when I was hacking on Commodore PETs and Apple IIs learning how to program. Over that time period there have been noticeable trends such as the centralization/decentralization cycle within both technology and organizations. With regards to technology, think about the transitions from mainframe (centralized) to client/server (decentralized) to web 1.0 (centralized) to web 2.0/Ajax (decentralized) as an example of the cycle. The trend that has lately attracted my attention is about scaling. I’m proposing that we’re approaching the end of scalability.

As a scalability consultant who travels almost every week to clients in order to help them scale their systems, I don’t make this end of scalability statement lightly. However, before we jump into my reasoning, we first need to define scalability. To some scalability means that a system needs to scale infinitely no matter what the load over any period of time. While certainly ideal, the challenge with this definition is that it doesn’t take into account the underlying business needs. Investing too much in scalability before its necessary isn’t a wise investment for a business when there are other great projects in which to invest such as more customer facing features. A definition that takes this into account defines scalability as “the ability of a system to maintain the satisfaction of its quality goals to levels that are acceptable to its stakeholders when characteristics of the application domain and the system design vary over expected operational ranges.” [2:119]

The most difficult problem with scaling a system is typically the database or persistent data storage. AKF Partners teaches general database and applications scale theory in terms of a three-dimensional cube where the X-axis of the cube represents replication of identical code or data, the Y-axis represents a split by dissimilar functions or services, and the Z-axis represents a split across similar transactions or data.[3] Having taught this scaling technique and seen it implemented in hundreds of systems, we know that by combining all three axes a system can scale infinitely. However, the cost of this scalability is increased complexity for development, deployment, and management of the system. But is this really the only option?

NoSQL
The NoSQL and NewSQL movement has produced a host of new persistent storage solutions that attempt to solve the scalability challenges without increased complexity. Solutions such as MongoDB, a self-proclaimed “scalable, high-performance, open source NoSQL database”, attempt to solve scaling by combining replica data sets (X-axis splits) with sharded clusters (Y & Z-axis splits) to provide high levels of redundancy for large data sets transparently for applications. Undoubtedly, these technologies have advanced many systems scalability and reduced the complexity of requiring developers to address replica sets and sharding.

But the problem is that hosting MongoDB or any other persistent storage solution requires keeping the hardware capacity on hand for any expected increase in traffic. The obvious solution to this is to host it in the cloud, where we can utilize someone else’s hardware capacity to satisfy our demand. Unless you are utilizing a hybrid-cloud with physical hardware you are not getting direct attached storage. The problem with this is that I/O in the cloud is very unpredictable, primarily because it requires traversing the network of the cloud provider. Enter Solid-State Drives (SSD).

SSD
Chris Lalonde, CEO of ObjectRocket a MongoDB cloud provider hosted entirely on SSDs, states that “Developers have been thinking that they need to keep their data set size the same size as memory because of poor I/O in the cloud and prior to 2.2.x MongoDB had a single lock, both making it unfeasible to go to disk in systems that require high performance. With SSDs the I/O performance gains are so large that it effectively negates this and people need to re-examine how their apps/platforms are architected.”

Lots of forward thinking technology organizations are moving towards SSDs. Facebook’s appetite for solid-state storage has made it the largest customer for Fusion-io, putting NAND Flash memory products in its new data centers in Oregon and North Carolina. Lalonde says “When I/O becomes cheap and fast it drastically changes how developers think about architecting their application e.g. a flat file might be just fine for storing some data types vs. the heavy over head of any kind of structured data.” ObjectRocket’s service offering also provides some other nice features such as “instant sharding” where through the click of a button provides an entire 3-node shard on demand.

GAE
Besides the advances being made in leveraging NoSQL and SSDs to allow applications to scale using Infrastructure as a Service (IaaS), there are advances in Platform as a Service (PaaS) offerings such as Google App Engine (GAE) that are also helping systems scale with little to no burden on developers. GAE allows applications to take advantage of scalable technologies like BigTable that Google applications use, allowing them to claim that, “Automatic scaling is built in with App Engine, all you have to do is write your application code and we’ll do the rest. No matter how many users you have or how much data your application stores, App Engine can scale to meet your needs.” While GAE doesn’t have customers as large as Netflix who run exclusively on Amazon’s Web Services (AWS) their customers do include companies like the Khan Academy, which has over 3.8 million monthly unique visitors to their growing collection of over 2,000 videos.

So with solutions like ObjectRocket, GAE, and the myriad of others that make it easier to scale to significant numbers of users and customers without having to worry about data set replication (X-axis splits) or sharding (Y & Z-axis splits), are we at the end of scalability? If we’re not there yet we soon will be. “But hold on” you say, “our systems are producing more and consuming more data…much more.” No doubt the amount of data that we process is rapidly expanding. In fact, according to the EMC sponsored IDC study in 2009, the amount of digital data in the world was almost 500 exabytes and doubling every 18 months. But when we combine the benefits we achieve from such advances as transistor density on circuits (Moore’s Law), NoSQL technologies, cheaper and faster storage (e.g. SSD), IaaS, and PaaS offerings, we are likely to see the end of the need for most applications developers to care about manually scaling their applications themselves. This will at some point in the future all be done for them in “the cloud”.

So What?
Where does this leave us as experts in scalability? Do we close up shop and go home? Fortunately, no, there are still reasons that application developers or technologists need to be concerned with splitting data replica sets and sharding data across nodes. Two of these reasons are 1) reduce risk and 2) improve developer efficiency.

Reducing Risk
As we’ve written about before, risk has several high-level components (probability of an incident, duration, and % of customers impacted). Google “GAE outages” or “AWS outages” or any other IaaS or PaaS provider and the word “outage” and see what you find. All hosting providers that I’m aware of have had outages in their not-so-distant past. GAE had a major outage on October 26, 2012 for 4 hours. GAE proudly states at the bottom of their outage post “Since launching the High Replication Datastore in January 2011, App Engine has not experienced a widespread system outage.” Which sounds impressive until you do the math and realize that this one outage caused their availability to drop to 99.975% for the entire year and a half that the service has been available. Not to mention that they have much more frequent local outages or issues that affect some percentage of their customers. We have been at clients when they’ve experienced incidents caused by GAE.

The point here is not to call out GAE, trust me all other providers have the exact same issue. The point is that when you rely on a 3rd party for 100% of your availability you by definition have their availability as your ceiling. Now add on your availability issues because of maintenance, code releases, bugs in your code, etc. Why is this? Incidents are almost always have multiple root causes that include architecture, people, and process. Everything eventually fails including our people and processes.

Given that you cannot reduce the probability of an incident to 0%, no matter whether you run the datacenter or a 3rd party provider does, you must focus on the other risk factors (reduce the duration and reduce the % of customers impacted). The way you achieve this is by splitting services (Y-axis splits) and by separating customers (Z-axis splits). While leveraging AWS’s RDS or GAE’s HRD provides cross availability zone / datacenter redundancy, in order to have your application take advantage of these you still have to do the work to split it. And if you want even higher redundancy (across vendors) you definitely have to do the work to split applications or customers between IaaS or PaaS providers.

Improving Efficiency
Let’s say you’re happy with GAE’s 99.95% SLA which no doubt is pretty good especially when you don’t have to worry about scaling. But don’t throw away the AKF Scalability Cube just yet. One of the major reasons we get called in to clients is because their BOD or CEO aren’t happy with how the development team is delivering new products. They recall the days when there were 10 developers and features flew out of the door. Now that they have 100 developers, everything seems to take forever. The reason for this loss of efficiency is that with a monolithic code base (no splits for different services) all 100 developers trying to make changes and add functionality, they are stepping all over each other. There needs to be much more coordination, more communication, more integration, etc. By splitting the application in to separate services (Y-axis splits) with separate code bases the developers can split into independent teams or pods that makes them much more efficient.

Conclusion
We are likely to see continued improvement in IaaS and PaaS solutions that auto-scale and perform to such a degree that most applications will not need to worry about scaling because of user traffic. However, this does not obviate the need to consider scaling for greater availability / vendor independence or to improve a development teams efficiency. All great technologist, CTOs, or application developers will continue to care about scalability for these and other reasons.

 
REFERENCES
1. Francis, F., The End of History? The National Interest, 1989. 16(4).
2. Duboc, L., E. Leiter, and D. Rosenblum, Systematic Elaboration of Scalability Requirements through Goal-Obstacle Analysis. 2012.
3. Abbott, M.L. and M.T. Fisher, The Art of Scalability: Scalable Web Architecture, Processes, and Organizations for the Modern Enterprise. 2009: Addison-Wesley Professional.

The Art of Scalability

Scalability Rules

What are some of the signs that you may not be aligned closely enough with the business and its performance and what should you do about it?

1) Not understanding feature impact – New features are introduced into your product without an understanding of the business impact.

You should never introduce new features without understanding the impact it is supposed to have on your business. In other words, establish a business goal for new features. For example, a new feature that allows for one click purchase is expected to improve conversion rates by 15% within 2 months. Remember all goals should be SMART goals (per chapter 1 in The Art of Scalability – Specific, Measurable, Attainable, Realistic, and Time constrained).

2) Celebrating launch and not success – Upon deployment of your product and confirmation that everything is working as expected, fireworks go off, confetti falls from the ceiling, and the gourmet lunch is ordered.

While recognizing your team for their efforts to launch a feature can be important, you really should celebrate when you have reached the business goal that is associated with that feature (see item #1 above). This might be achieving a revenue target, increasing the number of active accounts, or reaching a conversion target. If you deploy a feature or a product, and your business targets are not met as expected, you have more work to do. This should not be a surprise. Agile development’s basic premise is that we do not know what the final state of the feature and thus we must iterate.

3) No business metric monitoring – Your DevOps team is alerted that something is wrong with one of your services but you rely on your customer support or service desk to tell you if customers are impacted.

This is something that many of our clients struggle with. We believe its critical to detect a problem before you customers have to tell you or you have to ask them. You should be able determine if your business is being impacted without asking your customers.

By monitoring real time business metrics and comparing the actual data to a historical curve you can more quickly detect if there is a problem and avoid sifting trough alerting and monitoring white noise that your systems will inevitability produce. For more details on our preferred strategy, visit one of our earlier blogs. You can also read more about the Design To Be Monitored principle in our book Scalability Rules.

As you company grows, make sure that your product, your engineering team, and your technical operations are closely aligned with your business. We firmly believe that we as technologists must stay aligned with our business for success. In fact, we really are the business. Our solutions enable the company to earn revenue.

The Art of Scalability

Scalability Rules

The Art of Scalability

Scalability Rules

At the intersection of cognitive biases (especially attribution bias) and the Dunning-Kruger effect lies this (to my knowledge unnamed) phenomenon that keeps all of us from understanding how our contributions might have resulted in a successful outcome.  Cognitive biases cause us to take far too much credit for successes, and incorrectly attribute the reasons for our success.   The Dunning-Kruger effect causes us overrate our abilities to achieve similar success in the future.   All of these work against us in trying to determine what allowed us to be successful.

Incorrectly attributing the causes for our success can be a huge problem.  Imagine that you decide that the reason your company achieved a particular successful outcome is because of your sales ability as a senior executive.  You might be inclined to believe, based on your personal analysis that your sales ability and sales execution will result in similar future successes.  There are two potential problems here.  The first is that we  incorrectly attributed the reason for success when the real reason was due to, for instance, the efficacy of our product.  The second is that we overrated our contribution and abilities when the real credit should go somewhere else.   Clearly both can lead to future disasters down the road.

You may be thinking that many people are successful time and time again by taking similar actions and by employing similar behaviors.  That is absolutely true.  The issue is not whether we can repeat our past successes – it’s that we can’t accurately identify (for ourselves) which actions or behaviors led to those successes.

Research suggests that we are much more likely, if we apply disciplined process, to learn from failures as compared to successes.  Even greater learning can be gleaned from comparing our successes and failures.   Furthermore, involving others helps us triangulate and either validate or invalidate our beliefs as to the causes of both successes and failures.  The key here is disciplined process coupled with introspection and outsider perspective to guard against cognitive bias and eliminate the Dunning-Kruger effect.

In summary:

1)      We are ill equipped to identify the causes of success without outside help.

2)      We are better equipped to identify the causes of failure.

3)      We are best equipped to compare successes and failures and draw conclusions.

4)      It is always best to involve multiple people in the identification of either success or failure.

The Art of Scalability

Scalability Rules