wip! feature: Continuous encoding with per partition realtime offset space allocation back pressure

CHANGELOG.adoc

@@ -1,302 +1,41 @@
-:toc: macro
-:toclevels: 1
-
 = Change Log
 
-A high level summary of noteworthy changes in each version.
-
-NOTE:: Dependency version bumps are not listed here.
-
-// git log --pretty="* %s" 0.3.0.2..HEAD
-
-// only show TOC if this is the root document (not in the README)
-ifndef::github_name[]
-toc::[]
-endif::[]
-
-== 0.5.2.4
-
-=== Improvements
-
-* feature: Simple PCRetriableException to remove error spam from logs (#444)
-
-=== Fixes
-
-* fixes #409: Adds support for compacted topics and commit offset resetting (#425)
-** Truncate the offset state when bootstrap polled offset higher or lower than committed
-** Prune missing records from the tracked incomplete offset state, when they're missing from polled batches
-
-== 0.5.2.3
-
-=== Improvements
-
-* Transactional commit mode system improvements and docs (#355)
-** Clarifies transaction system with much better documentation.
-** Fixes a potential race condition which could cause offset leaks between transactions boundaries.
-** Introduces lock acquisition timeouts.
-** Fixes a potential issue with removing records from the retry queue incorrectly, by having an inconsistency between compareTo and equals in the retry TreeMap.
-* Adds a very simple Dependency Injection system modeled on Dagger (#398)
-* Various refactorings e.g. new ProducerWrap
-
-* Dependencies
-** build(deps): prod: zstd, reactor, dev: podam, progressbar, postgresql maven-plugins: versions, help (#420)
-** build(deps-dev): bump postgresql from 42.4.1 to 42.5.0
-** bump podam, progressbar, zstd, reactor
-** build(deps): bump versions-maven-plugin from 2.11.0 to 2.12.0
-** build(deps): bump maven-help-plugin from 3.2.0 to 3.3.0
-** build(deps-dev): bump Confluent Platform Kafka Broker to 7.2.2 (#421)
-** build(deps): Upgrade to AK 3.3.0 (#309)
-
-
-=== Fixes
-
-* fixes #419: NoSuchElementException during race condition in PartitionState (#422)
-* Fixes #412: ClassCastException with retryDelayProvider (#417)
-* fixes ShardManager retryQueue ordering and set issues due to poor Comparator implementation (#423)
-
-
-== v0.5.2.2
-
-=== Fixes
-
-- Fixes dependency scope for Mockito from compile to test (#376)
-
-== v0.5.2.1
-
-=== Fixes
-
-- Fixes regression issue with order of state truncation vs commit (#362)
-
-== v0.5.2.0
-
-=== Fixes and Improvements
-
-- fixes #184: Fix multi topic subscription with KEY order by adding topic to shard key (#315)
-- fixes #329: Committing around transaction markers causes encoder to crash (#328)
-- build: Upgrade Truth-Generator to 0.1.1 for user Subject discovery (#332)
-
-=== Build
-
-- build: Allow snapshots locally, fail in CI (#331)
-- build: OSS Index scan change to warn only and exclude Guava CVE-2020-8908 as it's WONT_FIX (#330)
-
-=== Dependencies
-
-- build(deps): bump reactor-core from 3.4.19 to 3.4.21 (#344)
-- build(deps): dependabot bump Mockito, Surefire, Reactor, AssertJ, Release (#342) (#342)
-- build(deps): dependabot bump TestContainers, Vert.x, Enforcer, Versions, JUnit, Postgress (#336)
-
-=== Linked issues
-
-- Message with null key lead to continuous failure when using KEY ordering #318
-- Subscribing to two or more topics with KEY ordering, results in messages of the same Key never being processed #184
-- Cannot have negative length BitSet error - committing transaction adjacent offsets #329
-
-== v0.5.1.0
-
-=== Features
-
-* #193: Pause / Resume PC (circuit breaker) without unsubscribing from topics
-
-=== Fixes and Improvements
-
-* #225: Build and runtime support for Java 16+ (#289)
-* #306: Change Truth-Generator dependency from compile to test
-* #298: Improve PollAndProduce performance by first producing all records, and then waiting for the produce results.Previously, this was done for each ProduceRecord individually.
-
-== v0.5.0.0
-
-=== Features
-
-* feature: Poll Context object for API (#223)
-** PollContext API - provides central access to result set with various convenience methods as well as metadata about records, such as failure count
-* major: Batching feature and Event system improvements
-** Batching - all API methods now support batching.
-See the Options class set batch size for more information.
-
-=== Fixes and Improvements
-
-* Event system - better CPU usage in control thread
-* Concurrency stability improvements
-* Update dependencies
-* #247: Adopt Truth-Generator (#249)
-** Adopt https://github.com/astubbs/truth-generator[Truth Generator] for automatic generation of https://truth.dev/[Google Truth] Subjects
-* Large rewrite of internal architecture for improved maintence and simplicity which fixed some corner case issues
-** refactor: Rename PartitionMonitor to PartitionStateManager (#269)
-** refactor: Queue unification (#219)
-** refactor: Partition state tracking instead of search (#218)
-** refactor: Processing Shard object
-* fix: Concurrency and State improvements (#190)
+== next
 
-=== Build
+* Features:
+** Queueing and pressure system now self tuning, performance over default old tuning values (`softMaxNumberMessagesBeyondBaseCommitOffset` and `maxMessagesToQueue`) has doubled.
+*** These options have been removed from the system.
 
-* build: Lock TruthGenerator to 0.1 (#272)
-* build: Deploy SNAPSHOTS to maven central snaphots repo (#265)
-* build: Update Kafka to 3.1.0 (#229)
-* build: Crank up Enforcer rules and turn on ossindex audit
-* build: Fix logback dependency back to stable
-* build: Upgrade TestContainer and CP
-
-== v0.4.0.1
-
-=== Improvements
-
-- Add option to specify timeout for how long to wait offset commits in periodic-consumer-sync commit-mode
-- Add option to specify timeout for how long to wait for blocking Producer#send
-
-=== Docs
-
-- docs: Confluent Cloud configuration links
-- docs: Add Confluent's product page for PC to README
-- docs: Add head of line blocking to README
-
-== v0.4.0.0
-// https://github.com/confluentinc/parallel-consumer/releases/tag/0.4.0.0
-
-=== Features
-
-* https://projectreactor.io/[Project Reactor] non-blocking threading adapter module
-* Generic Vert.x Future support - i.e. FileSystem, db etc...
-
-=== Fixes and Improvements
-
-* Vert.x concurrency control via WebClient host limits fixed - see #maxCurrency
-* Vert.x API cleanup of invalid usage
-* Out of bounds for empty collections
-* Use ConcurrentSkipListMap instead of TreeMap to prevent concurrency issues under high pressure
-* log: Show record topic in slow-work warning message
-
-== v0.3.2.0
-
-=== Fixes and Improvements
-
-* Major: Upgrade to Apache Kafka 2.8 (still compatible with 2.6 and 2.7 though)
-* Adds support for managed executor service (Java EE Compatibility feature)
-* #65 support for custom retry delay providers
-
-== v0.3.1.0
-
-=== Fixes and Improvements
-
-* Major refactor to code base - primarily the two large God classes
-** Partition state now tracked separately
-** Code moved into packages
-* Busy spin in some cases fixed (lower CPU usage)
-* Reduce use of static data for test assertions - remaining identified for later removal
-* Various fixes for parallel testing stability
-
-== v0.3.0.3
-
-=== Fixes and Improvements
-
-==== Overview
-
-* Tests now run in parallel
-* License fixing / updating and code formatting
-* License format runs properly now when local, check on CI
-* Fix running on Windows and Linux
-* Fix JAVA_HOME issues
-
-==== Details:
-
-* tests: Enable the fail fast feature now that it's merged upstream
-* tests: Turn on parallel test runs
-* format: Format license, fix placement
-* format: Apply Idea formatting (fix license layout)
-* format: Update mycila license-plugin
-* test: Disable redundant vert.x test - too complicated to fix for little gain
-* test: Fix thread counting test by closing PC @After
-* test: Test bug due to static state overrides when run as a suite
-* format: Apply license format and run every All Idea build
-* format: Organise imports
-* fix: Apply license format when in dev laptops - CI only checks
-* fix: javadoc command for various OS and envs when JAVA_HOME missing
-* fix: By default, correctly run time JVM as jvm.location
-
-== v0.3.0.2
-
-=== Fixes and Improvements
-
-* ci: Add CODEOWNER
-* fix: #101 Validate GroupId is configured on managed consumer
-* Use 8B1DA6120C2BF624 GPG Key For Signing
-* ci: Bump jdk8 version path
-* fix: #97 Vert.x thread and connection pools setup incorrect
-* Disable Travis and Codecov
-* ci: Apache Kafka and JDK build matrix
-* fix: Set Serdes for MockProducer for AK 2.7 partition fix KAFKA-10503 to fix new NPE
-* Only log slow message warnings periodically, once per sweep
-* Upgrade Kafka container version to 6.0.2
-* Clean up stalled message warning logs
-* Reduce log-level if no results are returned from user-function (warn -> debug)
-* Enable java 8 Github
-* Fixes #87 - Upgrade UniJ version for UnsupportedClassVersion error
-* Bump TestContainers to stable release to specifically fix #3574
-* Clarify offset management capabilities
-
-== v0.3.0.1
-
-* fixes #62: Off by one error when restoring offsets when no offsets are encoded in metadata
-* fix: Actually skip work that is found as stale
-
-== v0.3.0.0
-
-=== Features
-
-* Queueing and pressure system now self tuning, performance over default old tuning values (`softMaxNumberMessagesBeyondBaseCommitOffset` and `maxMessagesToQueue`) has doubled.
-** These options have been removed from the system.
-* Offset payload encoding back pressure system
-** If the payload begins to take more than a certain threshold amount of the maximum available, no more messages will be brought in for processing, until the space need beings to reduce back below the threshold.
-This is to try to prevent the situation where the payload is too large to fit at all, and must be dropped entirely.
-** See Proper offset encoding back pressure system so that offset payloads can't ever be too large https://github.com/confluentinc/parallel-consumer/issues/47[#47]
-** Messages that have failed to process, will always be allowed to retry, in order to reduce this pressure.
-
-=== Improvements
-
-* Default ordering mode is now `KEY` ordering (was `UNORDERED`).
-** This is a better default as it's the safest mode yet high performing mode.
-It maintains the partition ordering characteristic that all keys are processed in log order, yet for most use cases will be close to as fast as `UNORDERED` when the key space is large enough.
-* https://github.com/confluentinc/parallel-consumer/issues/37[Support BitSet encoding lengths longer than Short.MAX_VALUE #37] - adds new serialisation formats that supports wider range of offsets - (32,767 vs 2,147,483,647) for both BitSet and run-length encoding.
-* Commit modes have been renamed to make it clearer that they are periodic, not per message.
-* Minor performance improvement, switching away from concurrent collections.
-
-=== Fixes
-
-* Maximum offset payload space increased to correctly not be inversely proportional to assigned partition quantity.
-* Run-length encoding now supports compacted topics, plus other bug fixes as well as fixes to Bitset encoding.
+* Improvements
+** https://github.com/confluentinc/parallel-consumer/issues/37[Support BitSet encoding lengths longer than Short.MAX_VALUE #37] - adds new serialisation formats that supports wider range of offsets - (32,767 vs  2,147,483,647) for both BitSet and run-length encoding
 
 == v0.2.0.3
 
-=== Fixes
-
+* Fixes
 ** https://github.com/confluentinc/parallel-consumer/issues/35[Bitset overflow check (#35)] - gracefully drop BitSet or Runlength encoding as an option if offset difference too large (short overflow)
 *** A new serialisation format will be added in next version - see https://github.com/confluentinc/parallel-consumer/issues/37[Support BitSet encoding lengths longer than Short.MAX_VALUE #37]
 ** Gracefully drops encoding attempts if they can't be run
 ** Fixes a bug in the offset drop if it can't fit in the offset metadata payload
 
 == v0.2.0.2
 
-=== Fixes
-
+* Fixes
 ** Turns back on the https://github.com/confluentinc/parallel-consumer/issues/35[Bitset overflow check (#35)]
 
 == v0.2.0.1 DO NOT USE - has critical bug
 
-=== Fixes
-
+* Fixes
 ** Incorrectly turns off an over-flow check in https://github.com/confluentinc/parallel-consumer/issues/35[offset serialisation system (#35)]
 
 == v0.2.0.0
 
-=== Features
-
+* Features:
 ** Choice of commit modes: Consumer Asynchronous, Synchronous and Producer Transactions
 ** Producer instance is now optional
 ** Using a _transactional_ Producer is now optional
 ** Use the Kafka Consumer to commit `offsets` Synchronously or Asynchronously
 
-=== Improvements
-
+* Improvements:
 ** Memory performance - garbage collect empty shards when in KEY ordering mode
 ** Select tests adapted to non transactional (multiple commit modes) as well
 ** Adds supervision to broker poller
@@ -305,8 +44,7 @@ It maintains the partition ordering characteristic that all keys are processed i
 ** Have onPartitionsRevoked be responsible for committing on close, instead of an explicit call to commit by controller
 ** Make sure Broker Poller now drains properly, committing any waiting work
 
-=== Fixes
-
+* Fixes:
 ** Fixes bug in commit linger, remove genesis offset (0) from testing (avoid races), add ability to request commit
 ** Fixes #25 https://github.com/confluentinc/parallel-consumer/issues/25:
 *** Sometimes a transaction error occurs - Cannot call send in state COMMITTING_TRANSACTION #25
@@ -319,8 +57,7 @@ It maintains the partition ordering characteristic that all keys are processed i
 
 == v0.1
 
-=== Features:
-
+* Features:
 ** Have massively parallel consumption processing without running hundreds or thousands of
 *** Kafka consumer clients
 *** topic partitions