Spring Batch and Reactive Streams

[spring-projects-issues]

Christian Trutz opened BATCH-2596 and commented

Hello Spring Batch team,

are there any plans to incorporate Reactive Streams (https://projectreactor.io/) in Spring Batch? I think about concepts like backpressure (between ItemReaders and ItemWriters within a Spring Batch step). I talked today with Mark Paluch @JAX and he suggested to open an discussion issue.

Christian

---

1 votes, 4 watchers

[spring-projects-issues]

Michael Minella commented

We have had some conversations about it. No concrete plans yet. We'll keep you posted.

[waqaskamran]

@spring-issuemaster any update on this thread ?

[philipbel]

Any updates? It's been 3+ years since the original request.

[mminella]

@philipbel We're really interested in working with someone with a concrete use case for this. If you have one you can share, let us know!

[philipbel]

@philipbel We're really interested in working with someone with a concrete use case for this. If you have one you can share, let us know!

@mminella,
I am working on a server where upon user signup, a job is triggered to fetch some resources from 3rd-party ReST APIs and populate a MongoDB collection for the user.
A second use case is to periodically (once every few hours) run a batch job, again fetching some external resources, and doing a computation on them, storing the result in MongoDB and in a 3rd-party ReST API.
A third use case I have is running machine learning jobs (tied to AWS) once a day, which, too, access external resources and store their results in MongoDB.

For all jobs, I would like to provide an overview to the user of what the job's status is, preferably with a progress.
In terms of data, not much, < 10 MB of JSON per job per user. Each job has a few steps.

The application uses WebFlux + WebClient and MongoDB Reactive.

[mminella]

@philipbel So what are you looking to get out of a Reactive Streams integration with Spring Batch?

[omar-napoleon]

Hello. Is there a way to integrate rsocket (reactor) into an ItemProcessor? I have the following problem. I have to call a server with tcp but when sending the data I need the response to process the data and be able to send it to the itemwriter, but waiting for the response delays all my processing. I need the communication to be asynchronous with the server but not to queue the next iterations of the itemprocessor. I have already converted the itemProcessor to asynchronous but in the same way when I run out of threads due to the delay of the external server my processing is delayed. I must meet a minimum amount of TPS to send to the server.

[fmbenhassine]

Spring Batch and Reactive Streams

First of all, Reactive Streams are designed for data streams (infinite) while Spring Batch is a batch processing framework designed for fixed datasets (finite). In my opinion, this is already a fundamental mismtatch that can result in a non-natural integration between these two tools.

Now even if we try to introduce "reactive" in some way in Spring Batch, we need to be careful about several design choices. The following excerpt from the Spring Framework FAQ section is key:

For handlers to be fully non-blocking, you need to use reactive libraries throughout the processing chain,
all the way to the persistence layer.

Spring Framework even recommends to keep using the blocking model if the stack is partially reactive:

By all means, keep using Spring MVC if you are developing web apps that don't benefit from a non-blocking
programming model, or that use blocking JPA or JDBC APIs for persistence (typically in combination with
thread-bound transactions).

Based on these statements, for a web application to be fully reactive, the entire stack should be reactive, from the controller all the way down to the persistence layer. This is no different for batch applications, except that we obviously don't have controllers here, but the end-to-end job execution should be reactive. It does not make sense to have a "reactive non-blocking step" that interacts with a blocking job repository.

So to really benefit from this reactive story, the entire framework should be reactive, from batch artefacts (reader, processor, writer, listeners, etc) to infrastructure beans (job repository, transaction manager, etc). And to achieve that, a huge effort is needed:

• A reactive job repository implementation should be provided
• All APIs using PlatformTransactionManager (namely BatchConfigurer, Tasklet, StepBuilderHelper, etc ) should be updated to use the newly introduced org.springframework.transaction.TransactionManager interface. This is to be able to provide either a ReactiveTransactionManager or a classic PlatformTransactionManager as needed.
• All implementations of readers/writers should be updated to use non-blocking APIs (File IO, Database IO, etc)
• And many other changes

Moreover, the current chunk-oriented processing model is actually incompatible with the reactive paradigm. The reason is that ChunkOrientedTasklet waits for the chunkProcessor (processor + writer) to process the whole chunk before reading the next chunk:

Chunk inputs = chunkProvider.provide();
chunkProcessor.process(inputs);

So this implementation should be adapted as well. And all these changes are required without even talking about
the current concurrency model of Spring Batch (that is incompatible with the reactive paradigm) and the optimistic locking
strategy used at the job repository level..

In my opinion, "reactive support" is not a feature we can introduce in Spring Batch, it actually requires a complete re-write of 80% of the framework (if not more). For all these reasons, I believe the cost/benefit of such integration is too high to be considered, but I'm open to be convinced otherwise.

[jjmargon]

Very interesting last comment by benas. It's true that Spring Batch has the "chunk" (i.e., finite data) concept deeply extended in the framework. Also, in Batch applications or jobs, you don't usually have concurrency issues as you do in APIs with an external interface (ex, a REST API). Recall that Reactive applications won't be faster. They're just there to optimize computing resources with IO operations and support more concurrent users with the same resources than a thread-per-request model.
However... Batch applications in the enterprise world are really important. There is a lot of focus nowadays in web APIs and things like that, but many apps have their batch counterpart to do a lot of business needs. Just a lot... So, it's not rare to have an enterprise use case with hundreds or thousands of concurrent job (or even task) executions. Different job definitions, but executing concurrently. Also, by Batch job's nature, these executions have a lot of IO operations.
IMHO, it's very important that Spring could have a solution for Reactive Batch. For example, in Spring Cloud Data Flow, imagine a server with just a couple of threads managing all job executions in the event-loop paradigm. It would be awesome in terms of computing optimization, and in the cloud world, it would mean a huge saving in terms of cost.
In any case, I think that you are right. It's virtually impossible to adapt the current Spring Batch (and Cloud Data Flow) project to support a reactive model. To give this reactive support to batch applications, I think the only solution for this is to create a new project for this goal.

[omerk706]

Are there any updates on this subject?
I do see the point in @benas comment, and I understand there will be a lot of rewriting to do.
But wasn't a similar thing done to allow webflux?

I see a usecase in my workplace where a huge amount of data being read from a Hadoop cluster record by record, collecting data from external services for each record, then processing and submitting the result on a message queue.

The first part of this job is to accumulate all of the Hadoop records to a file on the local batch job file system, then read them in the next section, collecting them to subsets of data to be process by the next tasklet.

I see how different this is from reading one record, then .zipWith( fetchDataFromExternal...)
and in the end submit it to message queue.
However, apart from the huge file download, the rest feels like a reactive is the natural solution, and if reactive existed the way it is now when spring batch was designed, wouldn't reactive be the natural choice for that kind of flow?

[toniocus]

Let me add a little Use Case I'm facing nowadays in an environment moving to cloud services, I'm pretty new to reactive programming, just a few weeks, but a very experienced Java developer, so I might make some big mistakes about things I'm not aware off, sorry for it if it is the case.

We have a few batch-jobs, some using spring-batch, some not, that basically take care of getting some information needed for business, the company is moving to a SF app + cloud services that deal with some services out SF.

So as a consequence some 'BusinessRules' are moving from being replicated in a lot of places to more something like SOA where we need to call REST APIs (in cloud and in SF) to apply those rules, instead of having them replicated in our code.

I guess at some point, but really not sure, this batch jobs might be replaced with some better Architectural Solution but for now, and probably for a couple of years, our batch applications need to replace DB calls or replicated code logic, with calls to our rest APIs.

So I started investigating basically how can I use new reflective WebClient in batch applications, and really facing a lot of wired things (for a new reactive programmer probably), when I discover this thread and I thought may be I'm not the only one facing this kind of problem.

Hope this adds something to this thread, and by the way if anyone can point me to any articles that discuss a similar problem I'll be very grateful.

Thanks in advance,