AKF Partners

Abbott, Keeven & Fisher PartnersPartners In Hyper Growth

Category » Scalability Rules

Paying Down Your Technical Debt

During the course of our client engagements, there are a few common topics or themes that are almost always discussed, and the clients themselves usually introduce them. One such area is technical debt. Every team has it, almost every team believes they have too much of it, and no team has an easy time explaining to the business why it’s important to address. We’re all familiar with the concept of technical debt, and we’ve shared a client horror story in a previous blog post that highlights the dangers of ignoring it: Technical Debt


When you hear the words “technical debt”, it invokes a negative connotation. However, the judicious use of tech debt is a valuable addition to your product development process. Tech debt is analogous to financial debt. Companies can raise capital to grow their business by either issuing equity or issuing debt. Issuing equity means giving up a percentage of ownership in the company and dilutes current shareholder value. Issuing debt requires the payment of interest, but does not give up ownership or dilute shareholder value. Issuing debt is good, until you can’t service it. Once you have too much debt and cannot pay the interest, you are in trouble.

Tech debt operates in the same manner. Companies use tech debt to defer performing work on a product. As we develop our minimum viable product, we build a prototype, gather feedback from the market, and iterate. The parts of the product that didn’t meet the definition of minimum or the decisions/shortcuts made during development represent the tech debt that was taken on to get to the MVP. This is the debt that we must service in later iterations. Taking on tech debt early can pay big dividends by getting your product to market faster. However, like financial debt, you must service the interest. If you don’t, you will begin to see scalability and availability issues. At that point, refactoring the debt becomes more difficult and time critical. It begins to affect your customers’ experience.


Many development teams have a hard time convincing leadership that technical debt is a worthy use of their time. Why spend time refactoring something that already “works” when you could use that time to build new features customers and markets are demanding now? The danger with this philosophy is that by the time technical debt manifests itself into a noticeable customer problem, it’s often too late to address it without a major undertaking. It’s akin to not having a disaster recovery plan when a major availability outage strikes. To get the business on-board, you must make the case using language business leaders understand – again this is often financial in nature. Be clear about the cost of such efforts and quantify the business value they will bring by calculating their ROI. Demonstrate the cost avoidance that is achieved by addressing critical debt sooner rather than later – calculate how much cost would be in the future if the debt is not addressed now. The best practice is to get leadership to agree and commit to a certain percentage of development time that can be allocated to addressing technical debt on an on-going basis. If they do, it’s important not to abuse this responsibility. Do not let engineers alone determine what technical debt should be paid down and at what rate – it must have true business value that is greater than or equal to spending that time on other activities.


Additionally, be clear about how you define technical debt so time spent paying it down is not comingled with other activities. Generally, bugs in your code are not technical debt. Refactoring your code base to make it more scalable, however, would be. A good test is to ask if the path you chose was a conscious or unconscious decision. Meaning, if you decided to go in one direction knowing that you would later need to refactor. This is also analogous to financial debt; technical debt needs to be a conscious choice. You are making a specific decision to do or not to do something knowing that you will need to address it later. Bugs are found in sloppy code, and that is not tech debt, it is just bad code.

So how do you decide what tech debt should be addressed and how do you prioritize? If you have been tracking work with Agile storyboards and product backlogs, you should have an idea where to begin. Also, if you track your problems and incidents like we recommend, then this will show elements of tech debt that have begun to manifest themselves as scalability and availability concerns. We recommend a budget of 12-25% of your development efforts in servicing tech debt. Set a budget and begin paying down the debt. If you are working on less than the lower range, you are not spending enough effort. If you are spending over 25%, you are probably fixing issues that have already manifested themselves, and you are trying to catch up. Setting an appropriate budget and maintaining it over the course of your development efforts will pay down the interest and help prevent issues from arising.

Taking on technical debt to fund your product development efforts is an effective method to get your product to market quicker. But, just like financial debt, you need to take on an appropriate amount of tech debt that you can service by making the necessary interest and principle payments to reduce the outstanding balance. Failing to set an appropriate budget will result in a technical “bankruptcy” that will be much harder to dig yourself out of later.

Comments Off on Paying Down Your Technical Debt

IT Conversations Podcast

We had the chance to speak with Phil Windley on the Technometria series at IT Conversations.

Here is the link to the podcast from the show.


Scalability Rules Videos and Reviews

Here are a few videos that we did to explain Scalability Rules…ignore the scary faces:

You can find more videos by following this link.

Here are a couple reviews of the book:
Code Ranch
LinkedIn Reviews

Comments Off on Scalability Rules Videos and Reviews

Code as Craft: Scalability Rules Video

In case you missed us at Etsy’s Code as Craft Speaker Series, don’t worry you can watch the full video here:

Watch live streaming video from etsy at livestream.com


Scalability Rules Android App

Whether you love Scalability Rules or you haven’t gotten around to purchasing your copy, check out the new android application. The app has the what, when, why, and how for each of the 50 rules. Follow this link, scan this QR code, or search for “scalability” in the android market place.

Comments Off on Scalability Rules Android App

Code as Craft: Scalability Rules

How cool is this?! We’ve been asked to speak at Etsy’s Code as Craft Speaker Series. Our presentation will take place on July 28th at 7pm at the Etsy Labs on the 7th floor at 55 Washington Street in beautiful Brooklyn. If you’re in the neighborhood, sign up here and stop by.

1 comment

Scalability Rules – Released This Week

Our newest book, Scalability Rules, has just been released. Here are a few places you can purchase the book:

You can also help us get the word out about this book by liking and sharing the book’s Facebook page or the book’s official website, where we’ll keep up to date information about reviews and speaking engagements.

Scalability Rules brings together 50 rules that are grounded in experience garnered from over a hundred companies such as eBay, Intuit, PayPal, Etsy, Folica, and Salesforce. Put together and organized to be easily read and referenced for rapid application to nearly any technical environment. The rules are technology agnostic and have been applied to LAMP, .net, and even midrange system architectures.

We are very thankful for everyone’s help in making this project come together and here are just a few of those folks:

    Technical Reviewers – Robert Guild, Geoffrey Weber, and Jeremy Wright
    Pre-reviewers – Chad Dickerson, Chris Lalonde, Jonathan Heiliger, Jerome Labat, and Nanda Kishore.
    Senior Acquisitions Editor – Trina MacDonald
    Development Editor – Songlin Qiu
    Project Editor – Anne Goebel

We dedicated this book to to our friend and partner Tom Keeven who in our mind is the originator of many of these concepts and has helped countless companies in his nearly 30 years in the business.


Scalability Rules TOC

We’ve completed the first draft of our new book “Scalability Rules – 50 Principles For Scaling Web Sites” and wanted to share the table of contents with everyone. We have a terrific team of technical editors who are reviewing every rule in detail but would also like to offer this opportunity to anyone else so inclined. Our publisher, Addison-Wesley Professional, has posted the draft versions of Chapters 1-5 (Rules 1-19) on line at Safari Rough Cuts and should have a couple more chapters available soon. If you’re interested in a sneak preview or would like to provide feedback, sign up and take a look. Below is the book’s table of contents.

Chapter 1 – Reduce the Equation

  • Rule 1 Don’t Over Engineer The Solution
  • Rule 2 Design Scale Into the Solution (D-I-D Process)
  • Rule 3 Simplify the Solution 3 Times Over
  • Rule 4 Reduce DNS Lookups
  • Rule 5 Reduce Objects Where Possible
  • Rule 6 Use Homogenous Networks

Chapter 2 – Distribute Your Work

  • Rule 7 Design to Split Reads and Writes (X axis)
  • Rule 8 Design to Split Different Things (Y axis)
  • Rule 9 Design to Split Similar Things   (Z axis)

Chapter 3 – Design to Scale Out Horizontally

  • Rule 10 Design Your Solution to Scale Out – Not Just Up
  • Rule 11 Use Commodity Systems (Goldfish not Thoroughbreds)
  • Rule 12 Scale Out Your Data Centers
  • Rule 13 Design to Leverage the Cloud

Chapter 4 – Use The Right Tools

  • Rule 14 Use Databases Appropriately
  • Rule 15 Firewalls, Firewalls, Everywhere!
  • Rule 16 Actively Use Log Files

Chapter 5 – Don’t Duplicate Your Work (Nov 30th)

  • Rule 17 Don’t Check Your Work
  • Rule 18 Stop Redirecting Traffic
  • Rule 19 Relax Temporal Constraints

Chapter 6 – Use Caching Aggressively

  • Rule 20 Leverage CDNs
  • Rule 21 Use Expires Headers
  • Rule 22 Cache Ajax Calls
  • Rule 23 Leverage Page Caches
  • Rule 24 Utilize Application Caches
  • Rule 25 Make Use of Object Caches
  • Rule 26 Put Object Caches on Their Own “Tier”

Chapter 7 – Learn From Your Mistakes

  • Rule 27 Learn Aggressively
  • Rule 28 Don’t Rely on QA to Find Mistakes
  • Rule 29 Failing to Design for Rollback Is Designing to Fail
  • Rule 30 Discuss and Learn from Failures

Chapter 8 – Database Rules

  • Rule 31 Be Aware of Costly Relationships
  • Rule 32 Use the Right Type of Database Locks
  • Rule 33 Pass on Using Multi-phase Commits
  • Rule 34 Try Not to Use “Select For Update”
  • Rule 35 Don’t Select Everything

Chapter 9 – Design for Fault Tolerance and Graceful Failure

  • Rule 36 Design Using Fault Isolative “Swim Lanes”
  • Rule 37 Never Trust Single Points of Failure
  • Rule 38 Avoid Putting Systems in Series
  • Rule 39 Ensure You Can Wire On and Off Functions

Chapter 10 – Avoid or Distribute State

  • Rule 40 Strive For Statelessness
  • Rule 41 Maintain Sessions in the Browser When Possible
  • Rule 42 Make Use of a Distributed Cache For States

Chapter 11 – Asynchronous Communication and Message Buses

  • Rule 43 Communicate Asynchronously As Much As Possible
  • Rule 44 Ensure Your Message Bus Can Scale
  • Rule 45 Avoid Overcrowding Your Message Bus

Chapter 12 – Miscellaneous Rules

  • Rule 46 Be Wary of Scaling Through 3rd Parties
  • Rule 47 Purge, Archive, and Cost-justify Storage
  • Rule 48 Remove Business Intelligence from Transaction Processing
  • Rule 49 Design Your Application to Be Monitored

Chapter 13 – Rule Review and Prioritization