fixes #184: Fix multi topic subscription with KEY order by adding topic to shard key #315

astubbs · 2022-05-25T13:13:29Z

Documentation (if applicable)
Changelog

Subscribing to two or more topics with KEY ordering, results in messages of the same Key never being processed #184
Message with null key lead to continuous failure when using KEY ordering #318

Blocked by:

Discover user Subjects classes either compiler or as source code astubbs/truth-generator#117
build: Upgrade Truth-Generator to 0.1.1 for user Subject discovery #332

Message with null key lead to continuous failure when using KEY ordering #318
With KEY ordering, option to combine queues from different partitions or topics #314

Bumps `testcontainers.version` from 1.17.1 to 1.17.2. Updates `testcontainers` from 1.17.1 to 1.17.2 - [Release notes](https://github.com/testcontainers/testcontainers-java/releases) - [Changelog](https://github.com/testcontainers/testcontainers-java/blob/master/CHANGELOG.md) - [Commits](testcontainers/testcontainers-java@1.17.1...1.17.2) Updates `kafka` from 1.17.1 to 1.17.2 - [Release notes](https://github.com/testcontainers/testcontainers-java/releases) - [Changelog](https://github.com/testcontainers/testcontainers-java/blob/master/CHANGELOG.md) - [Commits](testcontainers/testcontainers-java@1.17.1...1.17.2) Updates `junit-jupiter` from 1.17.1 to 1.17.2 - [Release notes](https://github.com/testcontainers/testcontainers-java/releases) - [Changelog](https://github.com/testcontainers/testcontainers-java/blob/master/CHANGELOG.md) - [Commits](testcontainers/testcontainers-java@1.17.1...1.17.2) Updates `postgresql` from 1.17.1 to 1.17.2 - [Release notes](https://github.com/testcontainers/testcontainers-java/releases) - [Changelog](https://github.com/testcontainers/testcontainers-java/blob/master/CHANGELOG.md) - [Commits](testcontainers/testcontainers-java@1.17.1...1.17.2) --- updated-dependencies: - dependency-name: org.testcontainers:testcontainers dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.testcontainers:kafka dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.testcontainers:junit-jupiter dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.testcontainers:postgresql dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [versions-maven-plugin](https://github.com/mojohaus/versions-maven-plugin) from 2.10.0 to 2.11.0. - [Release notes](https://github.com/mojohaus/versions-maven-plugin/releases) - [Changelog](https://github.com/mojohaus/versions-maven-plugin/blob/master/ReleaseNotes.md) - [Commits](mojohaus/versions@versions-maven-plugin-2.10.0...versions-maven-plugin-2.11.0) --- updated-dependencies: - dependency-name: org.codehaus.mojo:versions-maven-plugin dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]>

… topics

https://ossindex.sonatype.org/component/pkg:maven/com.google.guava/[email protected]?utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1

rkolesnev

LGTM.

Bumps `vertx.version` from 4.3.0 to 4.3.1. Updates `vertx-web-client` from 4.3.0 to 4.3.1 Updates `vertx-junit5` from 4.3.0 to 4.3.1 --- updated-dependencies: - dependency-name: io.vertx:vertx-web-client dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: io.vertx:vertx-junit5 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [postgresql](https://github.com/pgjdbc/pgjdbc) from 42.3.5 to 42.4.0. - [Release notes](https://github.com/pgjdbc/pgjdbc/releases) - [Changelog](https://github.com/pgjdbc/pgjdbc/blob/master/CHANGELOG.md) - [Commits](pgjdbc/pgjdbc@REL42.3.5...REL42.4.0) --- updated-dependencies: - dependency-name: org.postgresql:postgresql dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]>

- CVE-2020-8908: Files::createTempDir local information disclosure vulnerability #4011 - Only used transitively from tests, and is a deprecated function - google/guava#4011

…e-tg # Conflicts: # pom.xml

parallel-consumer-core/src/test/java/io/confluent/parallelconsumer/truth/ConsumerSubject.java

…oduce # Conflicts: # parallel-consumer-vertx/pom.xml # pom.xml

…ignment

…ing assignment

Runs all tests in all modules using pattern search. https://stackoverflow.com/questions/11469122/run-unit-tests-in-intellij-idea-from-multiple-modules-together

…eproduce # Conflicts: # pom.xml

astubbs

lgtm

…eproduce

…bugs/issue-184-reproduce

astubbs

lgtm

…eproduce # Conflicts: # parallel-consumer-core/src/test-integration/java/io/confluent/parallelconsumer/integrationTests/utils/KafkaClientUtils.java

dependabot bot and others added 12 commits May 20, 2022 16:33

START: Try to reproduce issue confluentinc#184 - run against multiple…

ec21ad0

… topics

test compiles

1b9d2ad

failling test

51ef075

fixes test - introduce ShardKey

7e32d4d

step

0a558f6

step

9628bee

step

b1fd89e

turn off ossindex for guava 31.1

d237158

https://ossindex.sonatype.org/component/pkg:maven/com.google.guava/[email protected]?utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1

step

d8ea983

step - extract

b883ba0

astubbs requested review from ifeeneyconfluent and rkolesnev May 25, 2022 13:13

astubbs linked an issue May 25, 2022 that may be closed by this pull request

Subscribing to two or more topics with KEY ordering, results in messages of the same Key never being processed #184

Closed

astubbs mentioned this pull request May 25, 2022

fixes #314: Combine queues from different partitions or topics with KEY ordering #316

Closed

2 tasks

astubbs marked this pull request as draft May 25, 2022 13:27

rkolesnev approved these changes May 25, 2022

View reviewed changes

dependabot bot and others added 2 commits May 25, 2022 17:21

step - tg demo, keep consumer open

ae21fb7

astubbs mentioned this pull request Jun 1, 2022

Message with null key lead to continuous failure when using KEY ordering #318

Closed

astubbs and others added 6 commits June 1, 2022 12:12

test for null keys

ad6c89e

allow truth snapshot

23e5389

add missing annotations

7fdb06a

step

03ff349

build: Exclude Guava 31.1 from OSSIndex sec scan

72ac264

- CVE-2020-8908: Files::createTempDir local information disclosure vulnerability #4011 - Only used transitively from tests, and is a deprecated function - google/guava#4011

astubbs mentioned this pull request Jul 4, 2022

Error in onPartitionsAssigned in parallel consumer #326

Open

astubbs added 2 commits July 9, 2022 09:10

...

f905cdb

build: Update to Truth-Generator 0.1.1 for Subject discovery

12dc7cf

astubbs added 3 commits July 11, 2022 11:24

test snapshot first

5e0db16

test snapshot first

ac65300

Merge remote-tracking branch 'origin/master' into improvements/upgrad…

dce76c2

…e-tg # Conflicts: # pom.xml

astubbs mentioned this pull request Jul 11, 2022

build: Upgrade Truth-Generator to 0.1.1 for user Subject discovery #332

Merged

4 tasks

astubbs commented Jul 11, 2022

View reviewed changes

parallel-consumer-core/src/test/java/io/confluent/parallelconsumer/truth/ConsumerSubject.java Outdated Show resolved Hide resolved

astubbs added 5 commits July 11, 2022 17:46

Merge remote-tracking branch 'origin/master' into bugs/issue-184-repr…

647675d

…oduce # Conflicts: # parallel-consumer-vertx/pom.xml # pom.xml

test mutli topic with simply creating a new consumer and checking ass…

ab5a528

…ignment

update after rename

186144d

Merge branch 'improvements/upgrade-tg' into bugs/issue-184-reproduce

828c1b2

fixup! test mutli topic with simply creating a new consumer and check…

38bf42e

…ing assignment

astubbs force-pushed the bugs/issue-184-reproduce branch from 0b88b3a to 38bf42e Compare July 11, 2022 17:01

astubbs added 5 commits July 11, 2022 18:40

fixup! test mutli topic with simply creating a new consumer and check…

bb61774

…ing assignment

switch from NewTopic to TopicPartitions

914dc59

ide: Add All test run target

75287fa

Runs all tests in all modules using pattern search. https://stackoverflow.com/questions/11469122/run-unit-tests-in-intellij-idea-from-multiple-modules-together

Merge remote-tracking branch 'confluent/master' into bugs/issue-184-r…

906d3fe

…eproduce # Conflicts: # pom.xml

review

a999242

astubbs marked this pull request as ready for review July 11, 2022 22:40

review

f88494a

astubbs commented Jul 11, 2022

View reviewed changes

astubbs changed the title ~~#184: Fix multi topic subscription with KEY order by adding topic to shard key~~ fixes #184: Fix multi topic subscription with KEY order by adding topic to shard key Jul 12, 2022

astubbs linked an issue Jul 12, 2022 that may be closed by this pull request

Message with null key lead to continuous failure when using KEY ordering #318

Closed

review

9010da5

astubbs changed the title ~~fixes #184: Fix multi topic subscription with KEY order by adding topic to shard key~~ fixes #184: Fix multi topic subscription with KEY order by adding topic to shard key (#315) Jul 12, 2022

astubbs changed the title ~~fixes #184: Fix multi topic subscription with KEY order by adding topic to shard key (#315)~~ fixes #184: Fix multi topic subscription with KEY order by adding topic to shard key Jul 12, 2022

astubbs added 2 commits July 12, 2022 13:16

Merge remote-tracking branch 'confluent/master' into bugs/issue-184-r…

3f6f4a7

…eproduce

Merge remote-tracking branch 'astubbs/bugs/issue-184-reproduce' into …

0070dd5

…bugs/issue-184-reproduce

astubbs commented Jul 12, 2022

View reviewed changes

Merge remote-tracking branch 'confluent/master' into bugs/issue-184-r…

9b12f45

…eproduce # Conflicts: # parallel-consumer-core/src/test-integration/java/io/confluent/parallelconsumer/integrationTests/utils/KafkaClientUtils.java

astubbs merged commit 9be14fe into confluentinc:master Jul 12, 2022

astubbs deleted the bugs/issue-184-reproduce branch July 12, 2022 13:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fixes #184: Fix multi topic subscription with KEY order by adding topic to shard key #315

fixes #184: Fix multi topic subscription with KEY order by adding topic to shard key #315

Uh oh!

astubbs commented May 25, 2022 •

edited

Loading

Uh oh!

rkolesnev left a comment

Uh oh!

Uh oh!

astubbs left a comment

Uh oh!

astubbs left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fixes #184: Fix multi topic subscription with KEY order by adding topic to shard key #315

fixes #184: Fix multi topic subscription with KEY order by adding topic to shard key #315

Uh oh!

Conversation

astubbs commented May 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rkolesnev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

astubbs left a comment

Choose a reason for hiding this comment

Uh oh!

astubbs left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

astubbs commented May 25, 2022 •

edited

Loading