Technical Due Diligence and Technical Debt
April 30, 2018 | Posted By: Daniel Hanley
Over the past 11 years AKF has been on a number of technical due diligences where we have seen technology organizations put off a large portion of engineering effort creating technical debt. A technical due diligence as introduced in Optimize Your Investment with Technical Due Diligence, should identify the amount of technical debt and quantify the amount of engineering resources dedicated to service the debt.
What is Technical Debt?
Technical debt is the difference between doing something the desired or best way and doing something quickly. Technical debt is a conscious choice, made knowingly and with commission to take a shortcut in the technology arena - the delta between the desired or intended way and quicker way. The shortcut is usually taken for time to market reasons and is a sound business decision within reason.
Is Tech Debt Bad?
Technical debt is analogous in many ways to financial debt - a complete lack of it probably means missed business opportunities while an excess means disaster around the corner. Just like financial debt, technical debt is not necessarily bad, by accruing some debt it allows the technology organization to release an MVP (minimal viable product) to customers, just as some financial debt allows a company to start new investments earlier than capital growth would allow. Too little debt can result in a product late to market in a competitive environment and too much debt can choke business innovation and cause availability and scalability issues. Tech debt becomes bad when the engineering organization can no longer service that debt. A technical due diligence should discover a proper management of a team’s technical debt.
Tech Debt Maintenance
Similar to financial debt, technical debt must be serviced, and it is serviced by the efforts of the engineering team - the same team developing the software. A failure to service technical debt will result in high interest payments as seen by slowing time to market for new product initiatives post investment. Our experience indicates that most companies should expect to spend 12% to 25% of engineering effort on a mix of servicing technical debt and handling defect correction (two different topics). Whether that resource allocation keeps the debt static, reduces it, or allows it to grow depends upon the amount of technical debt and also influences the level of spend. It is easy to see how a company delinquent in servicing their technical debt will have to increase the resource allocation to deal with it, reducing resources for product innovation and market responsiveness. Just as a financial diligence is verifying the CFO has accountability of their balance sheet, AKF looks to verify the CTO is on top of their technology balance sheet during technology due diligence.
Tech Debt Takeaways:
Delaying attention to address technical issues allows greater resources to be focused on higher priority endeavors
The absence of technical debt probably means missed business opportunities – use technical debt as a tool to best meet the needs of the business
Excessive technical debt will cause availability and scalability issues, and can choke business innovation (too much engineering time dealing with debt rather than focusing on the product)
The interest on tech debt is the difficulty or increased level of effort in modifying something in subsequent releases
The principal of technical debt is the difference between “desired” and “actual”
Technology resources to continually service technical debt should be clearly planned in product road maps – 12 to 25% is suggested
AKF’s technical due diligence will discover a team’s ability to quantify the amount of debt accrued and the engineering effort to service the debt. Leadership should be allocating approximately 20% of resources towards this debt and planning for the debt in the product road map.
Click here to see how AKF Partners can help you assess your company’s technical debt, or assess the debt and issues therein for any investment you plan to make.
Subscribe to the AKF Newsletter
Technical Due Diligence: Did We Get It Right
April 4, 2018 | Posted By: Geoffrey Weber
Technical Due Diligence: Did We get it Right?
AKF Partners have performed 100s of technical due diligence engagements on behalf of strategic investors, private equity, venture capitalists and others. As such, we have amassed a significant repository of data, of patterns and anti-patterns, and the personal characteristics of successful and unsuccessful executive teams. Moreover, our teams have been on the “other side” of the table countless times as executives for companies being acquired or looking to acquire.
It’s not rare, however, when a potential client asks: how accurate is your technical due diligence? How can I trust that 8 hours with a company is enough to evaluate their technology stack? We love this question!
Due diligence, whether financial or technical is about assessing the risk the investor or acquirer is taking on by investing money into a company. It’s less about trying to identify and measure every gap and more about understanding where significant gaps exist relative to the investment thesis, calculating the risk to which those gaps expose the investor, and estimating the cost of closing the most critical gaps.
Due diligence is remarkably similar to playing poker: not any one player at the table has complete information about the cards remaining in the deck and the cards held by each player. Great poker players combine an ability to calculate probability nearly instantly with respect to the possible combination of cards as well as an ability to read the other players; looking for “tells” that inform the player as to whether their opponent is bluffing or playing with a pair of Aces heading into the “flop.” Great poker players seamlessly combine science and art.
At AKF, we’ve developed a formal checklist of questions around which we build our due diligence practice. In the big picture, this includes evaluating People, Technology, and Process. Our experience suggests that companies cannot build the right technology without the right people in charge, with the right individual contributors displaying the right behaviors to maximize success. Further, building reliable performance and predictable scalability (process, in other words) is of little value if they’re built on top of a weak technology stack.
People >> Technology >> Process
Read more about technical due diligence and AKF Partners here.
So how do we know we’re right?
First and foremost we need to understand the backgrounds, responsibilities and behaviors of the team present in the room with us. A dominate CEO that answers even the most technical questions, shutting out the rest of the team, is a red flag. We might ask specific questions to others and ask the CEO to let us listen to what others at the table have to say. If we’re still road-blocked, then our assessment will be very specific about the behavior of the CEO and we might ask for additional time without the CEO present. Another red flag: a CTO answering a technical question while the VP of Engineering’s face turns purple; an engineering manager choking a piece of paper to death while listening to the CTO review architectural principles or a senior leader refusing eye contact. . . the list of cues is nearly endless. We’ve seen all of them.
Seeing red flags early in an engagement is wonderful, and it allows us time to adjust the agenda on the fly. If we’re hearing something that sounds implausible, we will focus on the area in question and fire off repeated questions. Nothing helps like asking the same question three times in three different ways at three different times during an engagement. We can then judge the quality of the answers by the variety of answers.
Everything is Perfect
The biggest red flag of all: when things are too good. No down time; the architecture scales across all three axes of the AKF Scalability cube without human interaction; obscure or bleeding-edge technologies implemented without significant issues, and, most of all, always on time/under budget. Each of these red flags highlights an area that begs for further digging. This is a good time to start asking for real data. Let’s see the logs in real-time; demonstrate that 2-factor works as described and let’s watch a build. This is a good time to get familiar with Benford’s Law which states “in many naturally occurring collections of numbers, the leading significant digit is likely to be small.” Math is useful here just as it was for Bernie Madoff and just as it is for the poker player seeing a 5th Ace being dealt.
Assessments with significantly dysfunctional teams are a little easier and our assessments reflect this by candidly reporting that we could not validate certain facts or statements due to these dysfunctions. Either we come back and dig more deeply or we meet with a different group of people with the investor.
The Truth about Broken Things
While we occasionally see these anti-patterns, we’re much more likely to see positive patterns:
- A leader that is capable of expressing a vision and behavioral standards (a.k.a. culture) and then seeing the echoes of those standards from more junior members of the team. Huge green flag – maybe even a checkered flag for the finish line.
- Finding flaws in the architecture and watching the technical team members finally see the flaw themselves and begin to energetically discuss ways to resolve the flaws.
- A willingness to listen to different ideas and a recognition that the technology team may not have all of the answers.
Example: Dozens of times we’ve heard teams say their monolithic database is fine, peak load is 10%, no worries here. But there is no answer for how to handle a 10x event that will take the database down. There are plenty of examples of surprise 10x events: think of serving ads online when a famous singer or actor suddenly dies.
A willingness to admit that such a possibility exists coupled with a desire to build load generating tools and capacity planning processes is another green flag. Properly building these tools and then designing a credible scalability plan will absolutely reduce future operation risks. Those are critical cues we’re looking for during a due diligence session.
The best green flag of all: “I don’t know.” This is not an admission of failure, this is an admission of the fact that our systems are so complex, it’s impossible for a single person to understand all of the dependencies. “I don’t know” followed by “let me go get someone who does know” is one of the biggest green flags we see. “I don’t know” is a green flag – “I always know” (hubris) is a red flag. We seek teams that seek truth – not stories.
An important dimension to explore is the maturity of the company versus other companies in their industry as well as other companies of their size and age compared against the universe of companies. A company of 10 people that has invested significant time focusing on scale has likely not spent enough time building out a product with the necessary features to capture the market. A private company that has existed for 15 years with > $100M in sales but has no capability to perform capacity planning has underinvested in architecture. We’ve seen companies in all stages of maturity and size and our data helps structure our diligence recommendations: we can compare virtually any company against our universe of experience.
Technical due diligence is not about calculating a risk number to 6 digits of precision. It’s not just science, and it’s not just art: it’s both. The art is in ensuring alignment of people, process and technology - particularly the people component. The science is in applying a data set garnered over 11 years to help identify where the largest issues do (or are likely) to exist. The end result is a report that outlines risks and plausible solutions written by a team of people who’ve had a combined 200 years experience doing the real work.
Subscribe to the AKF Newsletter
Managing Risk with Technical Due Diligence
February 20, 2018 | Posted By: Greg Fennewald
You should not buy a home without an inspection by a licensed home inspector and you should not buy a used car without having a mechanic check it out for you. Diligence - it just makes good sense. Similarly, it is prudent to include technical diligence as part of the evaluation for a potential technology company investment.
Diligence Informs Risk Management
Private equity and venture capital firms typically evaluate many areas preceding a potential investment. The business case, legal structure, competitive analysis, product strategy, financial audits and contractual landscape are all examples of diligence deemed necessary prior to an investment. A company with a great product but three years left on an extremely expensive office lease will probably have a lower value. Breaking the lease or living with it until the term expires means higher costs and thus lower EBITDA. A hot start up with an inexperienced CFO that has run on cash-based accounting from day 1 and is rapidly approaching $6 million in annual revenue needs to move to accrual-based accounting. That takes time and effort and possibly a talent search - this affects the value of the investment.
But what about the technical underpinnings of the product itself? A company with a solitary production database and a marketing analyst with access to directly query that database is likely headed for performance and availability incidents. Single points of failure create a high probability of non-availability. Solutions that don’t allow for seamless and elastic scalability may run into either capacity or cost of operations problems.
Preventing these incidents and altering the conditions that enabled them to exist takes time and effort. All of these assessment areas boil down to risk management. Further, understanding the cost of fixing these solutions helps a company understand their true cost of investment. Your investment includes not just the “PIC” or capital that you put into the company - it also includes all the costs to ensure continuing operations of the product that enables that company. A comprehensive diligence including technical diligence will prepare the investor to make an informed business decision - know the risks and adjust the value proposition accordingly.
Technology Risk Areas
Technology risks can be grouped into four broad areas - Architecture, Process, Organization, and Security. Each area has several subordinate themes.
Architecture - subordinate themes are availability, scalability, cost control.
• Commodity hardware - Corollas, not Carreras
• Horizontal scalability - scale out, not up
• Design for monitoring - see issues before your customers do
• N+1 design - everything fails eventually
• Design for rollback - minimize the impairment
• Asynchronous design - stateless systems
Process - subordinate themes are engineering, operations, and problem management
• Product management - a product owner should be able add, delay, or deprecate features from an upcoming release
• Metrics - development teams should use effort estimation and velocity measurement metrics to monitor progress and performance
• Development practices - developers should conduct code reviews and be held accountable for unit testing
• Incident management - incidents should be logged with sufficient details for further follow up
• Post mortem - a structured process should be in place to review significant problems, assign action items, and track resolution
• PDLC - the Product Development Lifecycle should align with the company’s desires to be customer driven (not desirable in most cases) or market driven (resulting in the highest returns and fastest saturation of any market)
Organization - subordinate themes are PDLC (Product Development Lifecycle) structure, product alignment and team composition
• Product or Service Alignment - cross functional teams should be aligned by product or service and understand how their efforts complement business goals
• Agile or Waterfall - if “discovering” the market or choosing the best possible product for a market then Agile is appropriate - if developing to well defined contracts then waterfall may be necessary.
• Team composition - the engineer to QA tester ratio should ideally exceed 3.5:1. Significant deviations may be a sign or trouble or a harbinger of problems to come
• Goals - measurable goals aligned with business priorities should be visible to all with clear accountability
Security - subordinate themes are framework, prevention, detection and response
• Framework - use NIST, ISO, PCI or other regulatory standards to establish the framework for a security program. The standards do overlap, think it through and avoid duplication of effort.
• Policies in place - a sound security program will have multiple security related policies such as employee acceptable use, access controls, data classification, and an incident response plan.
• Security risk matrix - security risks should be graded by their impact, probability of occurrence, and controlling measures
• Business metrics - analysis of business metrics (revenue per minute, change of address, checkout value anomalies, file saves per minute, etc) can develop thresholds for alerting to a potential security incident. Over time, the analysis can inform prevention techniques.
• Response plan - a plan must be in place and must have regular rehearsals.
Technology Cost Impact on Investment Value
Technology costs can have a significant impact on the overall investment value. Strengths and weaknesses uncovered during a technical diligence effort help the investor make the best overall business decision.
Technology costs are normally captured in 2 areas of the income statement, cost of revenue (production environment and personnel) and operating expenses (software development). Technology costs can also affect depreciation (server capital purchases) and amortization (pre-paid licensing and support). These cost areas should be reviewed for unusual patterns or abnormally high or low spend rates. It is also important to understand the term of equipment purchase, software licensing, and support contracts - spend may be committed for several years.
Cost Cautions - tales from the past
• Support for production equipment purchased from a 3d party because the equipment is old and no longer supported by the OEM. Use equipment as long as possible, but don’t risk a production outage.
• Constant software vendor license audits - they will find revenue, but the technology team that leaves their company vulnerable on a recurring basis is likely to have other significant issues.
• Lack of an RFP or benchmarking process to periodically assess the cost effectiveness of hardware, software, hosting, and support vendors. Making a change in one of these areas is not simple, but the technology team should know how much they should pay before a change is better for the company.
A technical diligence effort should also identify the level of technical debt and quantify the amount of engineering resources dedicated to servicing the technical debt.
Technical debt is a conscious choice to take a shortcut in the technology arena - the delta between the desired or intended way and quicker way. The shortcut is usually taken for time to market reasons and is a sound business decision within reason. Technical debt is analogous in many ways to financial debt - a complete lack of it probably means missed business opportunities while an excess means disaster around the corner.
Just like financial debt, technical debt must be serviced, and it is serviced by the efforts of the engineering team - the same team developing the software. AKF recommends 12% to 25% of engineering effort be spent servicing technical debt. Whether that resource allocation keeps the debt static, reduces it, or allows it to grow depends upon the amount of technical debt. It is easy to see how a company delinquent in servicing their technical debt will have to increase the resource allocation to deal with it, reducing resources for product innovation and market responsiveness.
Put It All Together
The investor has made use of several specialists in an overall diligence effort and is digesting the information to zero in on the choice to invest and at what price. The business side looks good - revenue growth, product strategy, and marketing are solid. The legal side has some risks relating to returning a leased office space to its original condition, but the lease has 5 years to run. Now for technology;
• Tech refresh is overdue, so additional investment is needed or a move to the cloud accelerated - either choice puts pressure on thin margins.
• An expensive RDBMS is in use, but the technology team avoids stored procedures and keeps their SQL as vanilla as possible - moving to open source is doable.
• Technical debt service is constantly derailed by feature requests from sales and marketing. Additional resources, hired or contracted, will be needed and will raise the technology run rate. More margin pressure.
• Conclusion - the investment needed to address tech refresh and technical debt changes the investment value. The investor lowers the offer price.
Interested in learning more about technical due diligence? Here are some due diligence do’s and don’ts.
How AKF can help
AKF has conducted hundreds of technical due diligence studies over the last 10 years. One would want an attorney for a legal diligence effort and one would want a technologist for a technical due diligence. AKF does technology right. Read more about our technical due diligence offerings here.
Subscribe to the AKF Newsletter
Technical Due Diligence Best Practices
January 23, 2018 | Posted By: Marty Abbott
Technical due diligence of products is about more than the solution architecture and the technologies employed. Performing diligence correctly requires that companies evaluate the solution against the investment thesis, and evaluate the performance and relationship of the engineering and product management teams. Here we present the best practices for technology due diligence in the format of things to do, and things not to do:
1. Understand the Investment/Acquisition Thesis
One cannot perform any type of diligence without understanding the investment/acquisition thesis and equally as important, the desired outcomes. Diligence is meant to not only uncover “what is” or “what exists”, but also identify the obstacles to achieve “what may or can be”. The thesis becomes the standard by which the diligence is performed.
2. Evaluate the Team against the Desired Outcomes
The technology product landscape is littered with the carcasses of great ideas run into the ground with the wrong leadership or the wrong team. Disagree? We ask you to consider the Facebook and Friendster battle. We often joke that the robot apocalypse hasn’t happened yet, and technology isn’t building itself. Great teams are the reasons solutions succeed and substandard teams behind those solutions that fail technically. Make sure your diligence is identifying whether you are getting the right team along with the product/company you acquire.
3. Understand the Tech/Product Relationship
Product Management teams are the engines of products, and engineering teams are the transmission. Evaluating these teams in isolation is a mistake – as regardless of the PDLC (product development lifecycle) these teams must have an effective working relationship to build great products. Make sure your diligence encompasses an evaluation of how these teams work together and the lifecycle they use to maximize product value and minimize time to market.
4. Evaluate the Security Posture
Cyber-crime and fraud is going to increase at a rate higher than the adoption of online solutions pursuant to a number of secular forces that we will enumerate in a future post. As such, it is in your best interest as an investor to understand the degree to which the company is focused on increasing the perceived cost of malicious activity and decreasing the perceived value of said malicious activity. Ensure that your diligence includes evaluating the security focus, spending, approach and mindset of the target company. This need not be a separate diligence for small investments – just ensure that you are comfortable with the spend, attention and approach.
1. Don’t Waste Too Much Time (or money) on Code Reviews
The one thing I know from years of running engineering teams is that anytime an engineer reviews code for the first time she is going to say, “This code is crap and needs to be rewritten.” Code reviews are great to find potential defects and to ensure that code conforms to the standards set forth by the company. But you are unlikely to have the time or resources to review everything. The company is also unlikely to give you unfettered access to all of their code (Google “Sybase Microsoft SQLServer” for reasons why). That leaves you at the whims of the company to cherry-pick what you review, which in turn means you aren’t getting a good representative sample.
Further, your standards likely differ from those of the target company. As such, a review of the software is simply going to indicate that you have different standards.
Lastly, we’ve seen great architecture and terrible code succeed whereas terrible architecture and great code rarely is successful. You may find small code reviews enlightening, but we urge you to spend a majority of your time on the architecture, people and process of the acquisition or investment.
2. Don’t Start a Fight
Far too often technology diligence sessions start in discussion and end in a fight. The people performing the diligence start asking questions in a way that may seem judgmental to the target company. Then the investing/acquiring team shifts from questions to absolute statements that can only be taken as judgmental. There’s simply no room for this. Diligence is clinical – not personal. It’s not a place to prove who is smarter than whom. This dynamic is one of the many reasons it is often a good idea to have a third party perform your diligence: The target company is less likely to feel threatened by the acquiring product team, and the third party is oftentimes more experienced with establishing a non-threatening environment.
3. Don’t Be Religious
In a services oriented world, it really doesn’t matter what code or what data persistence platform comprises a service you may be calling. Assuming that you are acquiring a solution and its engineers, you need not worry about supporting the solution with your existing skillsets. Debates around technology implementations too often come from a place of what one knows (“I know Java, Java rocks, and everything else is substandard”) than what one can prove. There are certainly exceptions, like aging and unsupported technology – but stay focused on the architecture of a solution, not the technology that implements that architecture.
4. Don’t Do Diligence Remotely
As we’ve indicated before, diligence is as much about teams as it is the technology itself. Performing diligence remotely without face to face interaction makes it difficult to identify certain cues that might otherwise be indicators that you should dig more deeply into a certain space or set of questions. Examples are a CTO giving an authoritative answer to a certain question while members her team roll their eyes or slightly shake or bow their heads.
You may also want to read about the necessary components of technical due diligence in our article on optimizing technical diligence.
AKF Partners performs diligence on behalf of a number of venture capital and private equity firms as well as on behalf of strategic acquirers. Whether for a third party view, or because your team has too much on their plate, we can help. Read more about our technical due diligence services here.
Subscribe to the AKF Newsletter