Add support for parallel processing of batches / bulk sets of messages

[astubbs]

Currently the interface only provides the user a way to process single messages in parallel. There are use cases however where the user may want to processes batches of messages in parallel as well.

For example, say the API they're calling out to actually supports batches of requests, for example Elastic Search batch API that can take several documents at once.

The API might look like:

void pollBatch(int batchLevel, Consumer<List<ConsumerRecord<K, V>>> usersVoidConsumptionFunction);

So the user could request batches of 5 records, which could then be transformed into a single HTTP request. These batches of 5 messages would all be run in parallel, up to the usual concurrency limits. The same could be added to the vertx module.

Further from this, options could be added to specify requirements for the batch - for example, give me 5 messages in a batch that all have the same key, are all in the same partition etc.

Feature comes out of the backlog, but was also prompted immediately from initial user feedback upon announcement: https://twitter.com/derekm00r3/status/1323375584701984768

[astubbs]

POC #19

[raphaelauv]

This feature is absolutely necessary for my use case

And since the official Kafka consumer lib is bulk I think this wrapper should support it.

[ismarslomic]

I can also confirm that this feature is highly wanted. I´m working as we speak with a use case where we need to move records from Kafka to Elasticsearch and doing this in bulks would reduce high volume of REST API calls

[raphaelauv]

this will not be part of release 1.0.0 ?

[astubbs]

@raphaelauv yes I think it should be. I'm currently working on another project atm, but hope to get back to PC 1.0 release soon.