We often talk to clients about the differences between tuning an application or platform and scaling it.  Often the clients intuitively see them as different things, but oddly still refer to them similarly as in “We scaled the database last year by tuning the top SQL queries and getting 20% better performance”.

Tuning is something that you absolutely must do, but we stress that it’s more of an operational endeavor than an architectural endeavor.  In the ideal world, as a typical part of your operation, you would focus some time monthly or quarterly on identifying the slowest portions of your product or the most costly in terms of hardware, engineering time or clock cycles.  These would then be prioritized based on cost to fix and expected return with changes being made to them as engineering allocations allow.  This process is just good housecleaning; throwing out old clothes, and books or getting rid of the motorcycle and lawn mower that no longer work to make room in a garage bay for other “stuff”.

And that’s just what you’re doing with tuning – throwing out old stuff to make room for new stuff.  Just as with the house analogy, you aren’t really making your house “bigger”, you are making the current house capable of holding different and ideally more valuable “stuff”.   If the occupants of your house aren’t growing (i.e. your business isn’t growing quickly), as with a new addition to a family or several new roommates, then this disposition of old less valuable stuff (old code that is seldom executed but costly, often executed code that is very costly) for newer more valuable stuff (new functionality or more efficient code) may be all you ever need to do.

Scaling, however, is an architectural effort that implies significant modifications to your current structure.  In the house analogy, it might mean adding a second floor or expanding the house.  Structurally the house is different and is capable of holding more “stuff” because the total floor space has increased, as compared to the tuning example where you just freed up existing floor space.  Scaling involves architectural effort in redesign of the underlying fundamentals of your system such as adding replicated read databases or splitting databases along data boundaries while tuning is just “spring cleaning” of your platform.

If you are experiencing rapid or hyper growth, you need to both tune your system for the sake of efficiency and performance and scale it for future growth.  Scaling is what allows you to achieve orders of magnitude of improvement (10x, 100x, etc), while tuning should afford linear growth and more efficient operation over time.

What questions do each of your system monitors answer? You probably think they answer questions such as “Is there a problem?” and if so “Where is the problem?” Most likely this is not the case and instead of telling you “Is there a problem?” it really only tells you “Where” or “What” the problem might be. Before we continue this, first a quick detour to discuss metrics, which while different than monitoring are very similar in many ways.

Eric Ries, co-founder and CTO of IMVU, posted an article about the difference between vanity metrics and actionable metrics.  The entire article and accompaning video are worth a read and listen but the take away is that most people are using and looking for metrics that are great soundbites but do not offer any definable actions.  One example is the total number of hits to a website.  Eric ask the questions “Now what? Do you really know what actions you took in the past that drove those visitors to you, and do you really know which actions to take next?” This makes total sense to me as we  often see teams misusing monitoring in an attempt to determine what actions to take with their systems.

Back to our discussion of what question your monitoring is attempting to answer. We think there are five evolutionary questions that monitoring should answer:

1. Is there a problem?
2. Where is the problem?
3. What is the problem?
4. Why is there a problem?
5. Will there be a problem?

Where most people fail is using a monitoring tool that is designed to answer “Where” or “What” and try to use it to answer “Is”. For example, if you are monitoring all of your servers vitals such as CPU, memory, and I/O what is the appropriate action for your team to take when the CPU utilization goes to 100%? The reason that might be a tough question is that you are missing the vital piece of information “Is this affecting my customers?”.  The “Is there a problem” is intended to be a proxy for customer impact in order to help determine the degree and speed of escalation of the issue.

If you have monitoring services in place now it is worthwhile to determine what question each one answers. If you are missing a monitor for a particular question, the time to remedy it is before you need that question answered.