Tuning is something that you absolutely must do, but we stress that it’s more of an operational endeavor than an architectural endeavor.  In the ideal world, as a typical part of your operation, you would focus some time monthly or quarterly on identifying the slowest portions of your product or the most costly in terms of hardware, engineering time or clock cycles.  These would then be prioritized based on cost to fix and expected return with changes being made to them as engineering allocations allow.  This process is just good housecleaning; throwing out old clothes, and books or getting rid of the motorcycle and lawn mower that no longer work to make room in a garage bay for other “stuff”.

And that’s just what you’re doing with tuning – throwing out old stuff to make room for new stuff.  Just as with the house analogy, you aren’t really making your house “bigger”, you are making the current house capable of holding different and ideally more valuable “stuff”.   If the occupants of your house aren’t growing (i.e. your business isn’t growing quickly), as with a new addition to a family or several new roommates, then this disposition of old less valuable stuff (old code that is seldom executed but costly, often executed code that is very costly) for newer more valuable stuff (new functionality or more efficient code) may be all you ever need to do.

Scaling, however, is an architectural effort that implies significant modifications to your current structure.  In the house analogy, it might mean adding a second floor or expanding the house.  Structurally the house is different and is capable of holding more “stuff” because the total floor space has increased, as compared to the tuning example where you just freed up existing floor space.  Scaling involves architectural effort in redesign of the underlying fundamentals of your system such as adding replicated read databases or splitting databases along data boundaries while tuning is just “spring cleaning” of your platform.

If you are experiencing rapid or hyper growth, you need to both tune your system for the sake of efficiency and performance and scale it for future growth.  Scaling is what allows you to achieve orders of magnitude of improvement (10x, 100x, etc), while tuning should afford linear growth and more efficient operation over time.

The Art of Scalability

In the event that you are reading this via RSS, we updated our site last night with a new look. A couple things are worth mentioning. First is the “All Posts” page. One thing that annoys me about most blogs is the lack of a way to see all the posts on one page. We now have a drop down menu  as well as a link on the right sidebar of the blog for “All Posts”. This page list by topic all of our blog posts. It is a very simple way to find an article or post that you might be interested in.

The second thing is the “Best of Posts” page. I wanted a way for our readers to be able to select what posts they liked the best, as ranked by the number of views, comments, pingbacks (links to the post), etc. The “Best of Posts” page on the right sidebar of the blog is a dynamically generated list of posts that through our readers interaction has ranked them the most popular. If you self-host a WordPress blog and like this idea of a reader selected “best of” list, you can get the plugin here.

As none of us are very good graphic artists we outsourced the design of our new logo. We used 99designs.com to host an auction and thought it was a great experience. You get to establish the price you are willing to pay, provide requirements, and give feedback to designers who submit entries. Our auction ran for 1 week and generated over 200 entries.

If you have any questions or comments about the site feel free to let us know.

The Art of Scalability

• Project Voldemort Blog: Building a terabyte-scale data cycle at LinkedIn
• Latency is Everywhere and it Costs You Sales – How to Crush it
• Improving performance and scalability with DDD
• Large-scale Graph Computing at Google
• PlentyOfFish Architecture
• InfoQ – Scalability Principles
• Graph Databases and the Future of Large-Scale Knowledge Management
• Google Wave Architecture
• HotPads Shows the True Cost of Hosting on Amazon

If you have any favorite articles you’ve read in the last month share them with us in the comments.

The Art of Scalability

The topic of this post is taking continuous integration to an extreme and performing continuous deployment. And it is exactly what it sounds like, all code that is written for an application is immediately deployed into production. If you haven’t heard of this before you’re first thought is probably that this is the ultimate in Cowboy Coding but it is in use by some household technology names like Flickr and IMVU. If you don’t believe this check out code.flickr.com and look at the bottom of the page, last time I checked it said:

Flickr was last deployed 20 hours ago, including 1 change by 1 person.

In the last week there were 34 deploys of 385 changes by 17 people.

Eric Ries, co-founder and former CTO of IMVU, is a huge proponent of continuous deployment as a method of improving software quality due to the  discipline, automation, and rigorous standards that are required in order to accomplish continuous deployment. Other folks at IMVU also seem to be fans of the continuous deployment methodology as well from the post by Timothy Fitz. Eric suggest a 5 step approach for moving to a continuous deployment environment.

1. Continuous Integration – Obviously before moving beyond integration into full deployment, this is a prerequisite that must be in place.
2. Source Code Commit Checks – This feature which is available in almost all modern source code control systems,  allows the process of checking in code to halt if one of the tests fail.
3. Simple Deployment Script – Deployment must be automated and have the ability to rollback, which we wholeheartedly agree with here and here.
4. Real-time altering – Bugs will slip through so you must be monitoring for problems and have the processes in place to react quickly
5. Root Cause Analysis – Eric recommends the Five Why’s approach to find root cause, whatever the approach, finding and fixing the root cause of problems is critical to stop repeating them.

Admittedly, this concept of developers pushing code straight to production scares me quite a bit, since I’ve seen the types of horrific bugs that can make their way into pre-production environments. However, I think Eric and the other continuous deployment proponents are onto something that perhaps the reason so many bugs are found by weeks of testing is a self-fulfilling prophecy. If engineers know their code is moving straight into production upon check in they might be a lot more vigilant about their code, I know I would be. How about you, what do you think about this development model?

The Art of Scalability

We will have one more round of editing before the January, 2010 launch of the book, so it’s not too late for you to sign up for Safari Books Online, review our book and give us your valuable insights!

The Art of Scalability

We were intrigued by a question asked by Tony Bain of Read Write Web, in his recent article “Is the Relational Database Dead?” Since relational databases management systems (RDBMS) are a significant part of most SaaS or Web 2.0 architectures, we spend a lot of time helping people scale their databases. We also often find ourselves part of discussions about the future of relational databases.

Do Relational Databases Scale?
We believe that RDBMS scale very well especially when not on a single server. This is the major principle behind the three axes of the AKF Scale Cube.

The AKF Database Scale Cube consists of an X, Y and Z axes – each addressing a different approach to scale transactions applied to a database. The lowest left point of the cube (coordinates X=0, Y=0 and Z=0) represents the worst case monolithic database – a case where all data is located in a single location and all accesses go to this single database.

The X Axis of the cube represents spreading read requests across multiple instances of the same data set. All writes get executed on a single database that asynchronously propagates to the read replicas. Caching can be implemented to further reduce database load. The Y Axis of the cube represents a split by function or service. The Z Axis represents ways to split transactions by performing a lookup, a modulus or other indiscriminate function (hash for instance).

Key / Value Stores
We don’t believe in the imminent demise of the RDBMS but we do expect more applications and services to use a relational database as a persistent data store and use key/value stores as part of the transactional system. We have seen and expect to see more of the use of key/value stores such as memcached and redis as a shared memory object cache that is the primary data source for the service or application.  Writes work their way back to the relational database asynchronously and reads into the object cache work their way forward either scheduled or on demand.

This trend is possible obviously because of the software being developed to allow this but also the cheaper hardware including virtualization in clouds. As services become more specialized and service level agreements demand faster response times, in memory data stores are very likely to be involved in more architectures as a layer between the application servers and the relational databases.

Specialized Databases
Another trend in RDBMS is the creation of specialized databases such as Hive that is an open source, peta-byte scale date warehousing framework based on Hadoop that was developed by Facebook. Other databases include Amazon’s distributed, web service SimpleDB, Google’s column-oriented BigTable, and the Apache JSON based CouchDB. Each of these with advantages in particular scenarios. It has been shown in research papers like “One Size Fits All? – Part 2: Benchmarking Results” by Michael Stonebraker and Ugur Cetintemel that major RDBMS vendors can be outperformed by 1 – 2 orders of magnitude by specialized database engines.

In “The End of an Architectural Era” they continue their argument  that the current legacy RDBMS code attempting to be a “one size fits all” excels at nothing. In fact they offer data that the completely written from scracth, H-Store database, built in collaboration with MIT, Brown and Yale Universities, can outperform these legacy RDBMS by two orders of magnitude on standard transactional benchmarks.

Conclusion
The demise of the RDBMS has been prophesied almost since it’s inception in E.F. Codd’s 1970 paper “A Relational Model of Data for Large Shared Data Banks“. One of the most promising replacements given the advent of object oriented programming languages was the object-oriented database, which never became mainstream but has resulted in the inclusion of object-relational features in more popular RDBMS.

Even with the improved performance from specialized databases the most popular relational databases including Oracle, MySQL, and PostgreSQL will continue to be an integral part of SaaS and Web 2.0 architectures. Learning how to scale them will continue to be important for many years to come.

The Art of Scalability

Eric Ries, co-founder and CTO of IMVU, posted an article about the difference between vanity metrics and actionable metrics.  The entire article and accompaning video are worth a read and listen but the take away is that most people are using and looking for metrics that are great soundbites but do not offer any definable actions.  One example is the total number of hits to a website.  Eric ask the questions “Now what? Do you really know what actions you took in the past that drove those visitors to you, and do you really know which actions to take next?” This makes total sense to me as we  often see teams misusing monitoring in an attempt to determine what actions to take with their systems.

Back to our discussion of what question your monitoring is attempting to answer. We think there are five evolutionary questions that monitoring should answer:

1. Is there a problem?
2. Where is the problem?
3. What is the problem?
4. Why is there a problem?
5. Will there be a problem?

Where most people fail is using a monitoring tool that is designed to answer “Where” or “What” and try to use it to answer “Is”. For example, if you are monitoring all of your servers vitals such as CPU, memory, and I/O what is the appropriate action for your team to take when the CPU utilization goes to 100%? The reason that might be a tough question is that you are missing the vital piece of information “Is this affecting my customers?”.  The “Is there a problem” is intended to be a proxy for customer impact in order to help determine the degree and speed of escalation of the issue.

If you have monitoring services in place now it is worthwhile to determine what question each one answers. If you are missing a monitor for a particular question, the time to remedy it is before you need that question answered.

The Art of Scalability

David Silverman, PricewaterhouseCoopers Assurance partner, stated “Current economic conditions are clearly challenging … nonetheless, interactive media continues to consume a larger piece of the overall advertising pie.” According to the growth rates of advertising display mediums, internet display ads were growing at 7% year over year from 2007 to 2008 while all other mediums (radio, newspaper, magazine, outdoor, etc) were shrinking, except for television which grew at a modest 2%.  

The Art of Scalability

A product (hardware or software) perhaps starts out as a unique technology that is innovative and sells because of this. Over time this product becomes less unique and eventually commoditized, everyone sells essentially the same product only differentiated by price. High tech companies generally don’t like selling commodity products because the profit margins continue to get squeezed each year. The way to prevent this is to add features to the product, sort of like making highly processed food products. The more “value” the vendor adds the more they can keep the price high. What we as the consumers are left with is highly processed products that are not what they were originally designed for.

What’s the problem you say? What’s the harm in a piece of network gear that does it all?  It routes, load balances, firewalls, manages sessions, and even can have business logic. The problem is that it was more than likely designed to do one of those things and when the vendor sells it to us as a “do everything” panacea we’re setting ourselves up for problems. This is not to say that it’s never okay to dual use hardware or software or that you can’t use added features of a vendor’s product. What we do say is that the more you rely on these the more you are tied to this vendor and the more likely you are to have problems with that product.

We have a term that we use with our customers to explain this, we call it “purpose driven hardware/software”. The general principle is to use the hardware or software for its original purpose, a load balancer as a load balancer and a database as a database. Use an X, Y, or Z axis split to scale your database, don’t rely on a vendor for scale. Following this principle will 1) allow you to remain vendor agnostic, achieving the best pricing 2) rely on yourself for scaling not a vendor who may or may not release the next version/bug fix when you need to scale and 3) simplify your architecture which leads to less human error when everyone understands how the system works.

The Art of Scalability

Sometimes the intersection of what’s possible and what is practical is narrow or non-existent.  In the case where it’s nonexistent we should run, not walk, away from the idea as very often it is destructive to shareholder value.  Where that intersection exists but is small we should tread carefully to build something that can create shareholder value and hope that the foothold we create can expand once technology advances giving us a first mover advantage. 

But what about when there is a larger intersection?  Which is more important, the art or the science?  While there are many people who will take one side or the other, and while I have my own opinion, the best answer is that you need both.  Rarely is it the case that you will find someone with a compelling product vision and incredible execution skills as these two things seem to sit on opposite sides of a spectrum. It is much more likely that you will find someone who sits in the middle of the spectrum.  These people can paint a vision and execute against it, but they aren’t Steve Jobs on the vision side or Jack Welch on the execution side.

We think some of the best teams are built with two people and potentially two organizations embodying the art and the science.  A natural conflict occurs here, but it needn’t be an adversarial conflict.  Assuming the appropriate company culture and a healthy respect for each other’s abilities and value-add to the company, it is possible for these people or organizations to work together and for the healthy tension to accelerate the creation of shareholder value.  The “science” tethers the “art” and keeps it from wondering off aimlessly with no clear implementable path to value.  The art buoys the science and keeps it from sinking and being mired for a lack of product differentiating vision.

As chief executives, it is our responsibility to attract and retain “the best of the best” within both the art and the science and to create a culture where both are valued.  We should reward the folks who think and appear to be lost in the clouds to the same level as those who are attempting to keep them anchored to the practical.  Neither should be seen as less or more important and the debates between them should be cordial and filled with mutual respect.

The Art of Scalability