Scaling a software product involves adopting architecture patterns that can handle increasing demands while maintaining performance, reliability, and flexibility. AKF often sees clients who attempt to push too much data, resulting in excessively large message payloads and performance problems. This problem can be addressed with the Claim Check Pattern, which is particularly valuable in distributed systems with large message sizes. It works hand-in-hand with the Dumb Pipes, Smart Endpoints principle, often used in a distributed architecture. Here’s how these concepts can help scale a software product, along with examples of messaging or event bus technology that complement this approach.
The Claim Check Architecture Pattern: An Overview
Avoid Pushing A Bowling Ball Down A Garden Hose
The Claim Check Pattern is designed to offload large message payloads from the messaging system, allowing only small references or “claim checks” to travel across the message bus. The pattern divides the message into two parts:
- A claim check (a reference or ID to the data)
- The payload (the actual large data, stored externally)
When a sender needs to transmit data to a recipient, the payload is stored in a scalable external storage (like an S3 bucket or a database), and a reference (the claim check) is sent through the message bus. The recipient then uses this reference to fetch the actual payload from the storage system when necessary.
Benefits:
- Efficiency: Reduces the load on the message bus, preventing it from becoming a bottleneck, especially when dealing with large amounts of data.
- Scalability: By storing payloads in an external system, you can scale both the storage and the messaging system independently.
- Asynchronous Processing: Allows services to process messages without needing all the data in real-time, which is critical for high-throughput environments.
- Sensitive Data Protection: Allows offloading of sensitive data so that it’s not transmitted across the messaging system and can be secured by persisting the data elsewhere.
- Cost Efficiency: Increased message size limits are an premium feature and can become costly. Offloading part of the message can reduce the size and allow for lower cost messaging solution.
Dumb Pipes, Smart Endpoints
The Dumb Pipes, Smart Endpoints principle emphasizes that communication channels (pipes) between services should be simple and lightweight, with the endpoints (services) performing the heavy lifting in terms of logic and data processing.
In relation to the Claim Check Pattern, the pipes (message bus) should handle only the claim checks, making the system more efficient. Meanwhile, the smart endpoints (services) are responsible for pulling the data from external storage and processing it. This separation helps to scale the system without overloading the communication infrastructure.
Why Dumb Pipes?
Dumb pipes ensure that the message bus remains fast and efficient. If you pass large payloads through a message bus, such as Kafka or RabbitMQ, it can become a bottleneck, slowing down the entire system as it struggles to handle both message routing and large data transport. By keeping the pipes “dumb,” you allow them to focus purely on routing messages, not handling the actual payload.
Why Smart Endpoints?
The services on either end of the communication (the smart endpoints) handle the complexity of retrieving the data and processing it. These services are better equipped to deal with large payloads asynchronously. For example, a service can use the claim check to fetch the data when needed, and this decoupling of the data retrieval process from the message bus helps in maintaining system performance.
Examples of Messaging or Event Bus Technologies
To implement the Claim Check Pattern, several messaging technologies can be used alongside external storage systems. Here are some examples:
- Apache Kafka: Kafka is a distributed messaging platform known for its high throughput and scalability. However, sending large messages through Kafka can degrade performance. By applying the Claim Check Pattern, Kafka can be used to handle the claim check references, while large payloads are stored externally in systems like Amazon S3.
- RabbitMQ: RabbitMQ is a widely-used message broker that excels in reliable message delivery. Similar to Kafka, RabbitMQ can struggle with large message sizes. Using the Claim Check Pattern, RabbitMQ only needs to handle small claim check references, while external storage systems take care of the large payloads.
- Amazon SQS (Simple Queue Service): SQS is a fully-managed message queuing service that makes it easy to decouple services. However, it has message size limits. By adopting the Claim Check Pattern, you can store large data in Amazon S3 or a database, and send only the references through SQS.
- Azure Service Bus: Azure Service Bus is a cloud-based messaging service that allows scalable communication between services. Like SQS, it has message size limits, making it a perfect candidate for the Claim Check Pattern. Large messages can be stored in Azure Blob Storage, and only the reference is passed through Service Bus.
Real-World Use Case: Media Storage
A common use case for the Claim Check Pattern is in media storage systems where large files (images, videos) need to be transferred between services. For example, a video upload service may store the video in Amazon S3 and send the claim check through Kafka. The processing service would retrieve the claim check from Kafka, fetch the video from S3, and process it.
This approach helps maintain system performance by preventing large media files from overloading the message bus, ensuring that the system remains responsive even under heavy load.
Conclusion
The Claim Check Architecture Pattern and the Dumb Pipes, Smart Endpoints principle work together to scale software systems efficiently. By separating large payloads from message buses, you can reduce load and latency while allowing your services to handle data retrieval and processing asynchronously. Technologies like Kafka, RabbitMQ, and Amazon SQS, combined with scalable storage solutions like Amazon S3 or Azure Blob Storage, make implementing this pattern feasible and effective.
By leveraging these patterns, you can ensure your software architecture is prepared to handle growth, improve performance, and reduce the risk of bottlenecks, allowing your system to scale smoothly.
AKF Partners can help with software product architecture. Contact us if you’d like to learn more.