In the sci-fi cult classic "The Fifth Element" starring Bruce Willis, there is a scene of man vs. unknown power where the military fires all they've got at an advancing alien life form. After an unsuccessful first foray, General Staedert (played by John Neville) asks his technician "what do we have that is bigger than 240 [last foray of missiles launched]?" "Nothing Sir" is the reply and it is implied that the military is defeated by the expanding ball of flame and darkness.
Unfortunately, this scenario is not regaled only to science fiction movies, we have seen many of our clients and former employers utilizing the largest databases built and maxing out compute and storage. When the threshold is reached there aren't other options for larger systems and customer requests are either throttled or fail.
“Scaling up is failing up. Eventually you will get to a point where either the cost becomes uneconomical or there is no bigger hardware made.”
"It is our experience that within hyper-growth environments it is critical that companies plan to scale in a horizontal fashion—what we describe as scaling out. Most often this is done through the segmentation or duplication of workloads across multiple systems." (Abbott, Martin L. "Scalability Rules" Pearson Education. Chapter 3)
Deploying Big is Planning to Fail Miserably
Too often we see within our clients - and read in headlines of news stories - that companies fail to scale out with small and inexpensive hardware. Instead, they buy larger and faster systems until they have hit the upper limits and cannot buy a larger system. The accumulated tech debt of not building systems to scale horizontally comes due and systems collapse under their own weight.
The constraints and problems with scaling up aren’t only physical issues – often they are caused by a logical contention which cannot be solved with bigger and faster hardware. This also applies within developing software - one size does not fit all for long and having all of your code residing in one application to which others are all dependent results in a failure in one area causing failures at all areas.
Scale Out, Not Up
This is why we recommend scaling out, and not scaling up. Small "pizza box" 1U chassis servers are cheap and easy to add as additional compute is required and also allows for redundancy to maximize both availability and scalability. Similarly, sharding databases into smaller, easier and cheaper to maintain, build, and search allows for better horizontal scale while avoiding monoliths.
Beyond hardware, this also goes for sizes of services and teams – all tangible and intangible aspects of your organization. It also includes looking for single points of failure (SPOFs) within the organization. The best system I've seen that utilizes the AKF Scale Cube Z-Axis had sharded databases for every customer and taken every precaution I've witnessed to segregate, isolate, and protect customer data and were spread across multiple availability zones and regions in AWS - but then everything was bottlenecked through one access point.
Conclusions
As you build out systems, teams, etc. consider how an individual component will fail and if it will be graceful, or if it will be like pushing down the first domino which impacts the rest of the dominos.
Take advantage of the slower pace during the COVID isolation and review our questions we ask during technical due diligences to see where the weak areas are in your technology and staffing.
Plan for success and design your systems to scale out. Don't get caught in the trap of expecting to scale up only to find out that you've run out of faster and larger systems to purchase, lease, or rent. Give us a call! We can help.