Distributed Spring Batch Without a Message Broker
Distributed Spring Batch processing typically requires a shared, persistent job repository. This article proposes using that database as the entire coordination plane, eliminating the message broker. It focuses on dynamically sizing work according to the number of active nodes, employing a per-job master without a permanent coordinator, and implementing a compare-and-set claiming method to prevent double execution. Additionally, it outlines a two-phase heartbeat failover mechanism and discusses scenarios where a broker may still be beneficial.
Spring Batch has been a leading framework for batch processing on the JVM for over a decade. It allows jobs to be constructed from steps, including chunk-oriented readers, processors, and writers, all managed by a metadata repository that enables restartability. This setup is particularly effective for tasks such as nightly extract, transform, load (ETL), financial reconciliation, and data validation on a single machine.
However, as workloads grow, a single machine may no longer suffice. When a reconciliation task expands beyond its processing window, the typical response is to distribute the workload across multiple nodes. Traditionally, Spring Batch addresses this through remote partitioning, where a manager step divides the work into partitions and sends them to worker nodes via a message broker like Apache Kafka or RabbitMQ.
While remote partitioning is a proven method, it introduces an additional operational layer that many teams prefer to avoid. Most teams already manage a relational database, which Spring Batch requires for its job repository. Consequently, there is often little desire to establish, secure, and maintain a messaging cluster solely for distributing batch partitions.
This article presents an alternative approach for coordinating a distributed, fault-tolerant Spring Batch job across a small to medium cluster using only the shared database. It does not argue against the use of brokers but rather illustrates how to eliminate them from this process, outlining the limitations of this method and scenarios where brokers remain the preferred option.
Key Considerations for Distributed Batch Processing
To effectively manage distributed batch processing, four key questions must be addressed:
- How is the workload divided among the workers?
- Who oversees the execution of the job?
- How does a worker claim ownership of a partition to prevent data corruption?
- What happens if a node fails during execution?
A message broker typically resolves these issues by ensuring that each message is delivered to a single consumer, decoupling dispatch from execution. Without a broker, these questions must be explicitly answered using SQL against a database designed for durable shared state rather than message dispatch.
Implementing Database-Based Coordination
This approach utilizes the relational database required by Spring Batch as the sole coordination mechanism, eliminating the need for a broker or additional coordination services like Apache ZooKeeper. All cluster state information is stored in standard SQL tables, making the entire cluster inspectable through simple SQL queries.
Three additional tables complement Spring Batch’s existing schema: the `BATCH_NODES` table, which registers nodes with their status and load; the `BATCH_PARTITIONS` table, which tracks the lifecycle of each partition; and the `BATCH_JOB_COORDINATION` table, which links job executions to their respective master nodes.
While a shared, persistent job repository remains essential, this requirement is inherent to distributed partitioning and not a limitation of the proposed method. The execution context for each partition is written by the master node and read by the worker node executing it, ensuring that no partition is lost.
Coordination is decentralized by design, with no permanent master node. Instead, the node that initiates a job assumes the role of master for that execution only. This design minimizes the risk of a single point of failure, as losing a master affects only that specific job execution.
Integration with Spring Batch
For developers familiar with Spring Batch, the integration of this approach requires minimal changes to existing code. Spring Batch already provides a `Partitioner` interface for splitting work and a `PartitionHandler` interface for executing partitions. By implementing a custom `PartitionHandler` that interacts with the database, developers can maintain the existing programming model.
When a job is launched, the master node records the job execution in the `BATCH_JOB_COORDINATION` table and creates pending rows for each partition in the `BATCH_PARTITIONS` table. The master then monitors the execution and reassigns any orphaned partitions if a node fails.
This design allows for a straightforward implementation of distributed batch processing without the complexity of a message broker, making it suitable for scenarios where operational simplicity and cost are paramount.
Source: InfoQ