AKF Partners

Abbott, Keeven & Fisher Partners Partners in Growth

Growth Blog

Advice on how to fix broken $#*!

Scalability Best Practices

April 3, 2017  |  Posted By: AKF

Here are a baker’s dozen of items that we feel are Best Practices for Scalability:

baker's dozen

1. Asynchronous

Use asynchronous communication when possible. Synchronous calls tie the availability of the two services together. If one has a failure or is slow the other one is affected.

2. Swim Lanes

Create fault isolated “swim lanes” of hardware by customer segmentation. This prevents problems with one customer from causing issues across all customers. This also helps with diagnosis of issues and code roll outs.

3. Cache

Make use of cache at multiple layers including object caches in front of databases (such as memcached), page or item caches for content (such as squid) and edge caches (such as Akamai).

4. Monitoring

Understand your application’s performance from a customer’s perspective. Monitor outside of your network and have tests that simulate a real user’s experience. Also monitor the internal working of the application in terms of query and transaction execution count and timing.

5. Replication

Replicate databases for recovery as well as to off load reads to multiple instances.

6. Sharding

Split the application and databases by service and / or by customer using a modulus. While this requires slightly more complicated logic in the application it allows for massive scaling.

7. Use Few RDBMS Features

Use the OLTP database as a persistent storage device as much as possible. The more you rely on the features offered in most RDBMS for your transactions, the greater load you are putting on the hardest item in your system to scale. Remove all business logic from the database such as stored procedures and move it into the application. When significant scaling is required join in the application and not through the SQL.

8. Slow Roll

Roll out new code versions slowly, to a small subset of your servers without bringing the entire site down. This requires that all code be backwards compatible because you will have two versions of code running in production during the roll out. This method allows you to find problems that your quality and L&P testing missed while having minimal impact on customers.

9. Load & Performance Testing

Test the performance of the application version before it goes into production. This will not catch all the issues, which is why you need the ability to rollback, but it is very worthwhile.

10. Capacity Planning / Scalability Summits

Know how much capacity you have on all tiers and services in your system. Use Scalability Summits to plan for the increased capacity demands.

11. Rollback

Always have the ability to rollback a code release.

12. Root Cause Analysis

Ensure you have a learning culture that is evident by utilizing Root Cause Analysis to find and fix the real cause of issues.

13. Quality From The Beginning

Quality can’t be tested into a product, it must be designed in from the beginning.

Next: Splitting Applications or Services for Scale