April 10, 2018 | Posted By: Marty Abbott
The decomposition of monoliths into services, or alternatively the development of new products in a services-oriented fashion (oftentimes called microservices), is one of the greatest architectural movements of the last decade. The benefits of a services (alternatively microservices or micro-services) approach are clear:
- Independent deployment, decreasing time to market and decreasing time to value realization– especially when continuous delivery is employed.
- Team velocity and ownership (informed by Conway’s Law).
- Increased fault isolation – but only when properly deployed (see below).
- Individual scalability – and the decreasing cost of operations that entails when properly architected.
- Freedom of implementation and technology choices – choosing the best solution for each service rather than subjecting services to the lowest common denominator implementation.
Unfortunately, without proper architectural oversight and planning, improperly architected services can also result in:
- Lower overall availability, especially when those services are deployed in one of a handful of microservice anti-patterns like the mesh, services in depth (aka the Christmas Tree Light String) and the Fuse.
- Higher (longer) response times to end customers.
- Complicated fault isolation and troubleshooting that increases average recovery time for failures.
- Service bloat: Too many services to comprehend (see our service sizing post)
The following are patterns companies should avoid (anti-patterns) when developing services or microservices architectures:
Mesh architectures, where individual services both “fan out” and “share” subsequent services result in the lowest possible availability.
Services that are strung together in long (deep) call trees suffer from low availability and slow page response times as calculated from the product of each service offering availability.
The Fuse is a much smaller anti-pattern than “The Mesh”. In “The Fuse”, 2 distinct services (A and B) rely on service C. Should service C become slow or unavailable, both service A and B suffer.
Architecture Principle: Services – Broad, But Never Deep
These services anti-patterns protect against a lack of fault isolation, where slowness and failures propagate along a synchronous path. One service fails, and the others relying upon that service also suffer.
They also serve to guard against longer latency in call streams. While network calls tend to be minimal relative to total customer response times, many solutions (e.g. payment solutions) need to respond as quickly as possible and service calls slow that down.
Finally, these patterns help protect against difficult to diagnose failures. The Xmas Tree pattern name is chosen because of the difficulty in finding the “failed bulb” in old tree lights wired in series. Similarly, imagine attempting to find the fault in “The Mesh”. The time necessary to find faults negatively effects service restoration time and therefore availability.
As such, we suggest a principal that services should never be deep but instead should be deployed in breadth along product offering boundaries defined by nouns (resources like “customer” or “sales”) or verbs (services like “search” or “add to cart”). We often call this approach “slices instead of layers”.
How then do we accomplish the separation of software for team ownership, and time to market where a single service would otherwise be too large or unwieldy?
Old School – Libraries!
When you need service-like segmentation in a deep call tree but can’t suffer the availability impact and latency associated with multiple calls, look to libraries. Libraries will both eliminate the network associated latency of a service call. In the case of both The Fuse and The Mesh libraries eliminate the shared availability constraints. Unfortunately, we still have the multiplicative effect of failure of the Xmas Tree, but overall it is a faster pattern.
“But My Teams Can’t Release Separately!”
Sure they can – they just have to change how they think about releasing. If you need immediate effect from what you release and don’t want to release the calling services with libraries compiled or linked, consider performing releases with shared objects or dynamically loadable libraries. While these require restarts of the calling service, simple automation will help you keep from having an outage for the purpose of deploying software.
AKF Partners helps companies architecture highly available, highly scalable microservice architecture products. We apply our aggregate experience, proprietary models, patterns, and anti-patterns to help ensure your products can meet your company’s scale and availability goals. Contact us today - we can help!
April 10, 2018 | Posted By: Greg Fennewald
It seems as though a week cannot go by without news reports of yet another data breach at a large, recognizable company. One wonders what has been compromised but not yet detected or announced.
Security issues are perceived far differently than other technology issues. Consider an example of “Dilly Dilly Fidget Spinners has hard coded IP addresses in their code base” – most people would infer little if anything from that fact, while a minority would shake their heads and feel nauseous. On the other hand, “Dilly Dilly Fidget Spinners suffered a data breach affecting thousands of customers” is likely to have a negative perception from everyone who hears about it. The public sensitivity to all things security warrants a thorough investigation of security practices and incidents prior to any investment.
What should a potential investor look for in regard to information security during a due diligence effort? The answer to that question will vary widely based on the market segment of the potential investment, but there are some common considerations for information security
Common Security Considerations
1. Fit the Risk
Security posture should fit the risk a company faces. A company providing financial services or healthcare has a far higher risk to manage than a company involved in consumer product pricing and availability. The security policies, regulatory compliance and certifications, and operational practices should fit the risk. Going beyond the appropriate degree of security adds cost and may not make business sense but is far superior to inadequate security.
A security program that fits the risk profile for the company can be a business enabler. Security programs consume time and cost money – establishing the right fit and balance can conserve resources. Alternatively, a poor fit can add drag to a company and damage the business. Consider industries that have a strong reputation for security and face significant regulatory requirements, industries such as financial services and insurance. An experienced security professional with a banking background moves to a telematics company and is determined to bring bank level security to his new role. The telematics company deals with route optimization and fleet maintenance management. It does not process credit card payments or store PII. Bank level security would be a horrible fit that adds cost without benefit and ultimately damages the culture.
2. Security Minded Culture
Security awareness and accountability should be part of the culture. Well written policies do not accomplish much if they are not internalized and emphasized by leaders. Technology leaders must treat security in the same manner as they treat availability, quality of service, and engineering productivity - by establishing transparent goals and objective metrics by which those goals are measured.
An excellent resource for security awareness training is the OWASP Top 10 Application Security Risks list. The top 10 list is revised periodically as security threat vectors morph. The top three risks from the 2017 list are injections, broken authentication, and sensitive data exposure. More information can be found here.
3. Validation via Recurring Testing
Recurring testing is a hallmark of successful security programs. Areas to test include employee security policy training, 3d party network penetration tests, static code vulnerability testing, and drills to rehearse information security policies such as a security incident response plan. Testing validates the policies and practices are effective and part of the company’s culture.
QA automation is needed for agile product development that seeks rapid iteration and market discovery. 75% code coverage or greater is recommended. Incorporating automated security testing into the overall testing program is a smart move.
4. Cover the Basics
Basic security hygiene items that should be considered table stakes today include role-based access with audit trails, closing server ports by default and opening them by exception, segregating networks, logging production access, and encrypting data at rest. None of these actions are particularly difficult or expensive. Implementing them demonstrates security awareness and commitment. Controlling who can access data in a taciturn server farm, logging that access, and encrypting the data is a pretty good start to effective security.
How AKF Can Help
AKF Partners has performed hundreds of due diligence efforts over the last 10 years and is comprised of technology professionals that have walked the walk at widely recognized companies such as eBay, PayPal, and General Electric. Our security expertise comes from living the reality of technology, not an auditing course.
April 4, 2018 | Posted By: AKF
Technical Due Diligence: Did We get it Right?
AKF Partners have performed 100s of technical due diligence engagements on behalf of strategic investors, private equity, venture capitalists and others. As such, we have amassed a significant repository of data, of patterns and anti-patterns, and the personal characteristics of successful and unsuccessful executive teams. Moreover, our teams have been on the “other side” of the table countless times as executives for companies being acquired or looking to acquire.
It’s not rare, however, when a potential client asks: how accurate is your technical due diligence? How can I trust that 8 hours with a company is enough to evaluate their technology stack? We love this question!
Due diligence, whether financial or technical is about assessing the risk the investor or acquirer is taking on by investing money into a company. It’s less about trying to identify and measure every gap and more about understanding where significant gaps exist relative to the investment thesis, calculating the risk to which those gaps expose the investor, and estimating the cost of closing the most critical gaps.
Due diligence is remarkably similar to playing poker: not any one player at the table has complete information about the cards remaining in the deck and the cards held by each player. Great poker players combine an ability to calculate probability nearly instantly with respect to the possible combination of cards as well as an ability to read the other players; looking for “tells” that inform the player as to whether their opponent is bluffing or playing with a pair of Aces heading into the “flop.” Great poker players seamlessly combine science and art.
At AKF, we’ve developed a formal checklist of questions around which we build our due diligence practice. In the big picture, this includes evaluating People, Technology, and Process. Our experience suggests that companies cannot build the right technology without the right people in charge, with the right individual contributors displaying the right behaviors to maximize success. Further, building reliable performance and predictable scalability (process, in other words) is of little value if they’re built on top of a weak technology stack.
People >> Technology >> Process
Read more about technical due diligence and AKF Partners here.
So how do we know we’re right?
First and foremost we need to understand the backgrounds, responsibilities and behaviors of the team present in the room with us. A dominate CEO that answers even the most technical questions, shutting out the rest of the team, is a red flag. We might ask specific questions to others and ask the CEO to let us listen to what others at the table have to say. If we’re still road-blocked, then our assessment will be very specific about the behavior of the CEO and we might ask for additional time without the CEO present. Another red flag: a CTO answering a technical question while the VP of Engineering’s face turns purple; an engineering manager choking a piece of paper to death while listening to the CTO review architectural principles or a senior leader refusing eye contact. . . the list of cues is nearly endless. We’ve seen all of them.
Seeing red flags early in an engagement is wonderful, and it allows us time to adjust the agenda on the fly. If we’re hearing something that sounds implausible, we will focus on the area in question and fire off repeated questions. Nothing helps like asking the same question three times in three different ways at three different times during an engagement. We can then judge the quality of the answers by the variety of answers.
Everything is Perfect
The biggest red flag of all: when things are too good. No down time; the architecture scales across all three axes of the AKF Scalability cube without human interaction; obscure or bleeding-edge technologies implemented without significant issues, and, most of all, always on time/under budget. Each of these red flags highlights an area that begs for further digging. This is a good time to start asking for real data. Let’s see the logs in real-time; demonstrate that 2-factor works as described and let’s watch a build. This is a good time to get familiar with Benford’s Law which states “in many naturally occurring collections of numbers, the leading significant digit is likely to be small.” Math is useful here just as it was for Bernie Madoff and just as it is for the poker player seeing a 5th Ace being dealt.
Assessments with significantly dysfunctional teams are a little easier and our assessments reflect this by candidly reporting that we could not validate certain facts or statements due to these dysfunctions. Either we come back and dig more deeply or we meet with a different group of people with the investor.
The Truth about Broken Things