GROWTH BLOG: Backend for Frontend (BFF) Pattern: The Dos and Don’ts of the BFF Pattern
AKF Partners Logo Technology ConsultingScalability - We wrote the book on it ℠

Growth Blog

Scalability and Technology Consulting Advice for SaaS and Technology Companies

Microservice Anti-Pattern: Data Fan Out

April 21, 2019  |  Posted By: Marty Abbott

This article is the third in a multi-part series on microservices (micro-services) anti-patterns.  The introduction of the first article, Service Calls In Series, covers the benefits of splitting services, many of the mistakes or failure points teams create in services splits and the first anti pattern.  The second article, Service Fan Out discusses the anti-pattern of a single service acting as a proxy or aggregator of mulitple services.

Data Fan Out, the topic of this microservice anti-pattern, exists when a service relies on two or more persistence engines with categorically unique data, or categorically similar data that is not meant to be processed in parallel.  “Categorically Unique” means that the data is in no way related.  Examples of categorical uniqueness would be a database that stores customer data and a separate database that stores catalog data.  Instances of the same data, such as two separate databases each storing half of product catalog, are not categorically unique.  Splitting of similar data is often known as sharding.  Such “sharded” instances only violate the Data Fan Out pattern if:

1) They are accessed in series (database 1 is accessed and subsequently database 2 is accessed) –or-

2) A failure or slowness in either database, even if accessed in parallel, will result in a very slow or unavailable service.

Persistence engine means anything that stores data as in the case of a relational database, a NoSQL database, a persistent off-system cache, etc. 

Anytime a service relies on more than one persistence engine to perform a task, it is subject to lower availability and a response time equivalent to the slower of the N data stores to which it is connected.  Like the Service Fan Out anti-pattern, the availability of the resulting service (“Service A”) is the product of the availability of the service and its constituent infrastructure multiplied by the availability of each N data store to which it is connected. 

Further, the response of the services may be tied to the slowest of the runtime of Service A added to the slowest of the connected solutions.  If any of the N databases become slow enough, Service A may not respond at all. 

Because overall availability is negatively impacted, we consider Data Fan Out to be a microservice anti-pattern.

Microservice Anti-Pattern - Data Fan Out

One clear exception to the Data Fan Out anti-pattern is the highly parallelized querying done of multiple shards for the purpose of getting near linear response times out of large data sets (similar to one component of the MapReduce algorithm).  In a highly parallelized case such as this, we propose that each of the connections have a time-out set to disregard results from slowly responding data sets.  For this to work, the result set must be impervious to missing data.  As an example of an impervious result set, having most shards return for any internet search query is “good enough”.  A search for “plumber near me” returns 19/20ths of the “complete data”, where one shard out of 20 is either unavailable or very slow.  But having some transactions not present in an account query of transactions for a checking account may be a problem and therefore is not an example of a resilient data set.

Our preferred approach to resolve the Data Fan Out anti-pattern is to dedicate services to each unique data set.  This is possible whenever the two data sets do not need to be merged and when the service is performing two separate and otherwise isolatable functions (e.g. “Customer_Lookup” and “Catalog_Lookup”). 

Microservice Anti-Pattern Data Fan Out Solution - Split Service

When data sets are split for scale reasons, as is the case with data sets that have both an incredibly high volume of requests and a large amount of data, one can attempt to merge the queried data sets in the client.  The browser or mobile client can request each dataset in parallel and merge if successful.  This works when computational complexity of the merge is relatively low.

Microservice Anti-Pattern Data Fan Out Solution Client Side Aggregation

When client-side merging is not possible, we turn to the X Axis  of the Scale Cube for resolution.  Merge the data sets within the data store/persistence engine and rely on a split of reads and writes.  All writes occur to a single merged data store, and read replicas are employed for all reads.  The write and read services should be split accordingly and our infrastructure needs to correctly route writes to the write service and reads to the read service.  This is a valuable approach when we have high read to right ratios – fortunately the case in many solutions.  Note that we prefer to use asynchronous replication and allow the “slave” solutions to be “eventually consistent” - but ideally still within a tolerable time frame of milliseconds or a handful of seconds.

Microservice Anti-Pattern Data Fan Out Solution - Scale Cube X Axis Read Write Split


What about the case where a solution may have a high write to read ratio (exceptionally high writes), and data needs to be aggregated?  This rather unique case may be best solved by the Z axis of the AKF Scale Cube, splitting transactions along customer boundaries but ensuring the unification of the database for each customer (or region, or whatever “shard key” makes sense).  As with all Z axis shards, this not only allows faster response times (smaller data segments) but engenders high scalability and availability while also allowing us to put data “closer to the customer” using the service. 

Microservice Anti-Pattern Data Fan Out Solution - Scale Cube Y Axis Customer Split

AKF Partners helps companies create highly available, highly scalable, easily maintained and easily developed microservice architectures.  Give us a call - we can help!

Subscribe to the AKF Newsletter

Contact Us

Microservice Anti-Pattern: Service Fan Out

April 8, 2019  |  Posted By: Marty Abbott

This article is the second in a multi-part series on microservices (micro-services) anti-patterns.  The introduction of the first article, Service Calls In Series, covers the benefits of splitting services, many of the mistakes or failure points teams create in services splits and the first anti pattern.

Fan Out, the topic of this microservice anti-pattern, exists when one service either serves as a proxy to two or more downstream services, or serves as an integration of two subsequent service calls. Any of the services (the proxy/integration service “A”, or constituent services “B” and “C”) can cause a failure of all services.  When service A fails, service B and C clearly can’t be called.  If either service B or C fails or becomes slow, they can affect service A by tying up communication ports.  Ultimately, under high call volume, service A may become unavailable due to problems with either B or C.

Further, the response of the services may be tied to the slowest responding service.  If A needs both B and C to respond to a request (as in the case of integration), then the speed at which A responds is tied to the slowest response times of B and C.  If service A merely proxies B or C, then extreme slowness in either may cause slowness in A and therefore slowness in all calls.

Because overall availability is negatively impacted, we consider Service Fan Out to be a microservice anti-pattern.

Microservice Anti-Pattern Service Fan Out


One approach to resolve the above anti-pattern is to employ true asynchronous messaging between services.  For this to be successful, the requesting service A must be capable of responding to a request without receiving any constituent service responses.  Unfortunately, this solution only works in some cases such as the case where service B is returning data that adds value to service A.  One such example is a recommendation engine that returns other items a user might like to purchase.  The absence of service B responding to A’s request for recommendations is unfortunate but doesn’t eliminate the value of A’s response completely.

Fix to Service Fan Out Anti-Pattern - Async Calls

As was the case with the Calls In Series Anti-Pattern, we may also be able to solve this anti-pattern with ”Libraries for Depth” pattern.

Fix to Service Fan Out Anti-Pattern - Libraries

Of course, each of the libraries also represents a constituent part that may fail for any call – but the number of moving parts for each constituent part decreases significantly relative to a separately deployed service call.  For instance, no network interface is required, no additional host and virtual VM is employed during the call, etc.  Additionally, call latency goes down without network interfaces.

The most common complaint about this pattern is that development teams cannot release independently.  But, as we all know, this problem has been fixed for quite some time with Unix, Linux and Windows dynamically loadable libraries (dlls, dls) and the like.

Finally, we can remove the proxy/integration service into the browser and make multiple browser requests.  Data returned from service B or service C can either be displayed in separate browser frames/divisions or can be evaluated and integrated using browser scripting (e.g. javascript).  We prefer this method whenever possible.  If A is simply serving as a proxy, the solution is relatively simple.  If A was serving as an integration/aggregation service then Service A’s logic must be moved into the browser/client.  Doing so creates complete fault isolation and allows the services to fail independently without an impact on each other.

Fix to Service Fan Out Anti-Pattern - Browser Fan Out

AKF Partners has helped to architect some of the most scalable, highly available, fault-tolerant and fastest response time solutions on the internet.  Give us a call - we can help.

Subscribe to the AKF Newsletter

Contact Us

Microservice Anti-Pattern: Calls in Series (The Xmas Tree Light Anti-Pattern)

March 25, 2019  |  Posted By: Marty Abbott

This article is the first in a multi-part series on microservices (micro-services) anti-patterns. 

There are several benefits to carving up very large applications into service-oriented architectures.  These benefits can include many of the following:

  • Higher availability through fault isolation
  • Higher organizational scalability through lower coordination
  • Lower cost of development through lower overhead (coordination)
  • Faster time to market achieved again through lower overhead of coordination
  • Higher scalability through the ability to independently scale services
  • Lower cost of operations (cost of goods sold) through independent scalability
  • Lower latency/response time through better cacheability

The above should be considered only a partial list.  See our articles on the AKF Scale Cube, and when you should split services for more information.

In order to achieve any of the above benefits, you must be very careful to avoid common mistakes. 

Most of the failures that we see in microservices stem from a lack of understanding of the multiplicative effect of failure or “MEF”.  Put simply, MEF indicates that the availability of any solution in series is a product of the availability of all components in that series. 

Service A has an availability calculated by the product of its constituent parts.  Those parts include all of the software and infrastructure necessary to run service A.  The server availability, the application availability, associated library and runtime environment availabilities, operating system availability, virtualization software availability, etc.  Let’s say those availabilities somehow achieve a “service” availability of “Five 9s” or 99.999 as measured by duration of outages.  To achieve 99.999 we are assuming that we have made the service “highly available” through multiple copies, each being “stateless” in its operation.

Service B has a similar availability calculated in a similar fashion.  Again, let’s assume 99.999.

If, for a request from any customer to Service A, Service B must also be called, the two availabilities are multiplied together.  The new calculated availability is by definition lower than any service in isolation.  We move our availability from 99.999 to 99.998. 

When calls in series between services become long, availability starts to decline swiftly and by definition is always much smaller than the lowest availability of any service or the constituent part of any service (e.g. hardware, OS, app, etc).

This creates our first anti-pattern.  Just as bulbs in the old serially wired Christmas Tree lights would cause an entire string to fail, so does any service failure cause the entire call stream to fail.  Hence multiple names for this first anti-pattern:  Christmas Tree Light Anti-Pattern, Microservice Calls in Series Anti-Pattern, etc.

 Microservice Anti-Pattern - Calls in Series

The multiplicative effect of failure sometimes is worse with slowly responding solutions than with failures themselves.  We can easily respond from failures through “heartbeat” transactions.  But slow responses are more difficult.  While we can use circuit breaker constructs such as hystrix switches – these assume that we know the threshold under which our call string will break.  Unfortunately, under intense flash load situations (unforeseen high demand), small spikes in demand can cause failure scenarios.

One pattern to resolve the above issue is to employ true asynchronous messaging between services.  To make this effective, the requesting service must not care whether it receives a response.  This service must be capable of responding to a request without receiving any downstream response.  Unfortunately, this solution only works in some cases such as the case where service B is returning data that adds value to service A.  One such example is a recommendation engine that returns other items a user might like to purchase.  The absence of service B responding to A’s request for recommendations is unfortunate, but doesn’t eliminate the value of A’s response completely.

Microservice Calls in Series Anti-Pattern Solution - Async Calls

While the above pattern can resolve some use-cases, it doesn’t resolve most of them.  Most often downstream services are doing more than “modifying” value for the calling service:  they are providing specific necessary functions.  These functions may be mail services, print services, data access services, or even component parts of a value stream such as “add to cart” and “compute tax” during checkout.

In these cases, we believe in employing the Libraries for Depth pattern.

Microservice Calls in Series Anti-Pattern Solution - Use Libraries

Of course, each of the libraries also represents a constituent part that may fail for any call – but the number of moving parts for each constituent part decreases significantly relative to another service call.  For instance, no network interface is required, no additional host and virtual VM is employed during the call, etc.  Additionally, call latency goes down without network interfaces.

The most common complaint about this pattern is that development teams cannot release independently.  But, as we all know, this problem has been fixed for quite some time with Unix, Linux and Windows dynamically loadable libraries (dlls, dls) and the like.

 

Subscribe to the AKF Newsletter

Contact Us

 < 1 2

Categories:

Most Popular: