Skip to content

Commit bdc2932

Browse files
astubbsastubbs
committed
docs: Update README with features, links, demo, support status (#173)
1 parent 20b494b commit bdc2932

File tree

3 files changed

+77
-36
lines changed

3 files changed

+77
-36
lines changed

‎.idea/runConfigurations/README.xml‎

Lines changed: 27 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎README.adoc‎

Lines changed: 25 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
//
1111

1212

13-
= parallel-consumer (beta)
13+
= parallel-consumer
1414
:icons:
1515
:toc: macro
1616
:toclevels: 3
@@ -52,10 +52,12 @@ Parallel Apache Kafka client wrapper with client side queueing, a simpler consum
5252

5353
Confluent's https://www.confluent.io/confluent-accelerators/#parallel-consumer[product page for the project].
5454

55-
WARNING: This is not a Confluent supported product.
56-
It is an experimental alpha stage accelerator.
55+
CAUTION: This is not a Confluent supported product.
5756
See the <<Support and Issues>> section for more information.
5857

58+
IMPORTANT: This project has been stable and reached its initial target feature set in Q1 2021.
59+
It is actively maintained by the CSID team at Confluent.
60+
5961
[[intro]]
6062
This library lets you process messages in parallel via a single Kafka Consumer meaning you can increase consumer parallelism without increasing the number of partitions in the topic you intend to process.
6163
For many use cases this improves both throughput and latency by reducing load on your brokers.
@@ -71,6 +73,15 @@ It also opens up new use cases like extreme parallelism, external data enrichmen
7173

7274
An overview article to the library can also be found on Confluent's https://www.confluent.io/blog/[blog]: https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/[Introducing the Confluent Parallel Consumer].
7375

76+
[[demo]]
77+
.Relative speed demonstration
78+
--
79+
.Click on the animated SVG image to open the https://asciinema.org/a/404299[Asciinema.org player].
80+
image::https://gist.githubusercontent.com/astubbs/26cccaf8b624a53ae26a52dbc00148b1/raw/0e2a444a9ade392aab92918c48bc0bc3b7e5599c/demo.svg[link="https://asciinema.org/a/404299"]
81+
--
82+
83+
'''
84+
7485
toc::[]
7586

7687
== Motivation
@@ -197,7 +208,7 @@ The end-to-end latency of the responses to these answers needs to be as low as t
197208
We can break out as described below into the tool for processing that step, then return to the Kafka Streams context.
198209
* Provisioning extra machines (either virtual machines or real machines) to run multiple clients has a cost, using this library instead avoids the need for extra instances to be deployed in any respect.
199210

200-
== Feature List
211+
== Features List
201212

202213
* Have massively parallel consumption processing without running hundreds or thousands of:
203214
** Kafka consumer clients,
@@ -209,7 +220,10 @@ without operational burden or harming the cluster's performance
209220
* Solution for the https://en.wikipedia.org/wiki/Head-of-line_blocking["head of line"] blocking problem where continued failure of a single message, prevents progress for messages behind it in the queue
210221
* Per `key` concurrent processing, per partition and unordered message processing
211222
* Offsets committed correctly, in order, of only processed messages, regardless of concurrency level or retries
212-
* Vert.x non-blocking library integration (HTTP and any Vert.x future support)
223+
* Vert.x and Reactor.io non-blocking library integration
224+
** Non-blocking I/O work management
225+
** Vert.x's WebClient and general Vert.x Future support
226+
** Reactor.io Publisher (Mono/Flux) and Java's CompletableFuture (through `Mono#fromFuture`)
213227
* Reactor non-blocking library integration
214228
* Fair partition traversal
215229
* Zero~ dependencies (`Slf4j` and `Lombok`) for the core module
@@ -689,28 +703,21 @@ image::https://lucid.app/publicSegments/view/43f2740c-2a7f-4b7f-909e-434a5bbe3fb
689703
== Roadmap
690704

691705
For released changes, see the link:CHANGELOG.adoc[CHANGELOG].
692-
693-
=== Short Term - What we're working on nowish ⏰
694-
695-
* Depth~ first or breadth first partition traversal
696-
* JavaRX and other streaming modules
706+
For features in development, have a look at the https://github.com/confluentinc/parallel-consumer/issues[GitHub issues].
697707

698708
=== Medium Term - What's up next ⏲
699709

700-
* https://github.com/confluentinc/parallel-consumer/issues/21[Automatic fanout] (automatic selection of concurrency level based on downstream back pressure) (https://github.com/confluentinc/parallel-consumer/pull/22[draft PR])
701-
* Support for general Vert.x Verticles (non-blocking libraries)
702-
* Dead Letter Queue (DLQ) handling
703-
* Non-blocking I/O work management
704-
** More customisable handling of HTTP interactions
705-
** Chance to batch multiple consumer records into a single or multiple http request objects
706710
* https://github.com/confluentinc/parallel-consumer/issues/28[Distributed tracing integration]
707711
* https://github.com/confluentinc/parallel-consumer/issues/24[Distributed rate limiting]
708712
* https://github.com/confluentinc/parallel-consumer/issues/27[Metrics]
713+
* More customisable handling[https://github.com/confluentinc/parallel-consumer/issues/65] of HTTP interactions
714+
* Chance to https://github.com/confluentinc/parallel-consumer/issues/18[batch multiple consumer records] into a single or multiple http request objects
709715

710716
=== Long Term - The future ☁️
711717

712-
* Apache Kafka KIP?
713-
* Call backs only offset has been committed
718+
* https://github.com/confluentinc/parallel-consumer/issues/21[Automatic fanout] (automatic selection of concurrency level based on downstream back pressure) (https://github.com/confluentinc/parallel-consumer/pull/22[draft PR])
719+
* Dead Letter Queue (DLQ) handling
720+
* Call backs only once offset has been committed
714721

715722
== Usage Requirements
716723

‎src/docs/README.adoc‎

Lines changed: 25 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
//
1111

1212

13-
= parallel-consumer (beta)
13+
= parallel-consumer
1414
:icons:
1515
:toc: macro
1616
:toclevels: 3
@@ -52,10 +52,12 @@ Parallel Apache Kafka client wrapper with client side queueing, a simpler consum
5252

5353
Confluent's https://www.confluent.io/confluent-accelerators/#parallel-consumer[product page for the project].
5454

55-
WARNING: This is not a Confluent supported product.
56-
It is an experimental alpha stage accelerator.
55+
CAUTION: This is not a Confluent supported product.
5756
See the <<Support and Issues>> section for more information.
5857

58+
IMPORTANT: This project has been stable and reached its initial target feature set in Q1 2021.
59+
It is actively maintained by the CSID team at Confluent.
60+
5961
[[intro]]
6062
This library lets you process messages in parallel via a single Kafka Consumer meaning you can increase consumer parallelism without increasing the number of partitions in the topic you intend to process.
6163
For many use cases this improves both throughput and latency by reducing load on your brokers.
@@ -69,6 +71,15 @@ include::{project_root}/parallel-consumer-examples/parallel-consumer-example-cor
6971

7072
An overview article to the library can also be found on Confluent's https://www.confluent.io/blog/[blog]: https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/[Introducing the Confluent Parallel Consumer].
7173

74+
[[demo]]
75+
.Relative speed demonstration
76+
--
77+
.Click on the animated SVG image to open the https://asciinema.org/a/404299[Asciinema.org player].
78+
image::https://gist.githubusercontent.com/astubbs/26cccaf8b624a53ae26a52dbc00148b1/raw/cbf558b38b0aa624bd7637406579d2a8f00f51db/demo.svg[link="https://asciinema.org/a/404299"]
79+
--
80+
81+
'''
82+
7283
toc::[]
7384

7485
== Motivation
@@ -195,7 +206,7 @@ The end-to-end latency of the responses to these answers needs to be as low as t
195206
We can break out as described below into the tool for processing that step, then return to the Kafka Streams context.
196207
* Provisioning extra machines (either virtual machines or real machines) to run multiple clients has a cost, using this library instead avoids the need for extra instances to be deployed in any respect.
197208

198-
== Feature List
209+
== Features List
199210

200211
* Have massively parallel consumption processing without running hundreds or thousands of:
201212
** Kafka consumer clients,
@@ -207,7 +218,10 @@ without operational burden or harming the cluster's performance
207218
* Solution for the https://en.wikipedia.org/wiki/Head-of-line_blocking["head of line"] blocking problem where continued failure of a single message, prevents progress for messages behind it in the queue
208219
* Per `key` concurrent processing, per partition and unordered message processing
209220
* Offsets committed correctly, in order, of only processed messages, regardless of concurrency level or retries
210-
* Vert.x non-blocking library integration (HTTP and any Vert.x future support)
221+
* Vert.x and Reactor.io non-blocking library integration
222+
** Non-blocking I/O work management
223+
** Vert.x's WebClient and general Vert.x Future support
224+
** Reactor.io Publisher (Mono/Flux) and Java's CompletableFuture (through `Mono#fromFuture`)
211225
* Reactor non-blocking library integration
212226
* Fair partition traversal
213227
* Zero~ dependencies (`Slf4j` and `Lombok`) for the core module
@@ -610,28 +624,21 @@ image::https://lucid.app/publicSegments/view/43f2740c-2a7f-4b7f-909e-434a5bbe3fb
610624
== Roadmap
611625

612626
For released changes, see the link:CHANGELOG.adoc[CHANGELOG].
613-
614-
=== Short Term - What we're working on nowish ⏰
615-
616-
* Depth~ first or breadth first partition traversal
617-
* JavaRX and other streaming modules
627+
For features in development, have a look at the https://github.com/confluentinc/parallel-consumer/issues[GitHub issues].
618628

619629
=== Medium Term - What's up next ⏲
620630

621-
* https://github.com/confluentinc/parallel-consumer/issues/21[Automatic fanout] (automatic selection of concurrency level based on downstream back pressure) (https://github.com/confluentinc/parallel-consumer/pull/22[draft PR])
622-
* Support for general Vert.x Verticles (non-blocking libraries)
623-
* Dead Letter Queue (DLQ) handling
624-
* Non-blocking I/O work management
625-
** More customisable handling of HTTP interactions
626-
** Chance to batch multiple consumer records into a single or multiple http request objects
627631
* https://github.com/confluentinc/parallel-consumer/issues/28[Distributed tracing integration]
628632
* https://github.com/confluentinc/parallel-consumer/issues/24[Distributed rate limiting]
629633
* https://github.com/confluentinc/parallel-consumer/issues/27[Metrics]
634+
* More customisable handling[https://github.com/confluentinc/parallel-consumer/issues/65] of HTTP interactions
635+
* Chance to https://github.com/confluentinc/parallel-consumer/issues/18[batch multiple consumer records] into a single or multiple http request objects
630636

631637
=== Long Term - The future ☁️
632638

633-
* Apache Kafka KIP?
634-
* Call backs only offset has been committed
639+
* https://github.com/confluentinc/parallel-consumer/issues/21[Automatic fanout] (automatic selection of concurrency level based on downstream back pressure) (https://github.com/confluentinc/parallel-consumer/pull/22[draft PR])
640+
* Dead Letter Queue (DLQ) handling
641+
* Call backs only once offset has been committed
635642

636643
== Usage Requirements
637644

0 commit comments

Comments
 (0)