[SPARK-25288][Tests]Fix flaky Kafka transaction tests #22293

zsxwing · 2018-08-30T21:47:17Z

What changes were proposed in this pull request?

Here are the failures:

http://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.sql.kafka010.KafkaRelationSuite&test_name=read+Kafka+transactional+messages%3A+read_committed
http://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.sql.kafka010.KafkaMicroBatchV1SourceSuite&test_name=read+Kafka+transactional+messages%3A+read_committed
http://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.sql.kafka010.KafkaMicroBatchV2SourceSuite&test_name=read+Kafka+transactional+messages%3A+read_committed

I found the Kafka consumer may not see the committed messages for a short time. This PR just adds a new method waitUntilOffsetAppears and uses it to make sure the consumer can see a specified offset before checking the result.

How was this patch tested?

Jenkins

SparkQA · 2018-08-30T22:27:53Z

Test build #95497 has finished for PR 22293 at commit 6439367.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-30T22:35:34Z

Test build #4314 has finished for PR 22293 at commit 6439367.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-30T22:36:23Z

Test build #4318 has finished for PR 22293 at commit 6439367.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-30T22:36:31Z

Test build #4311 has finished for PR 22293 at commit 6439367.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-30T22:37:49Z

Test build #4316 has finished for PR 22293 at commit 6439367.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-30T22:38:32Z

Test build #4312 has finished for PR 22293 at commit 6439367.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-30T22:39:13Z

Test build #4317 has finished for PR 22293 at commit 6439367.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-30T22:41:21Z

Test build #4315 has finished for PR 22293 at commit 6439367.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-30T22:42:03Z

Test build #4319 has finished for PR 22293 at commit 6439367.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-30T22:43:21Z

Test build #4310 has finished for PR 22293 at commit 6439367.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-30T22:43:46Z

Test build #4313 has finished for PR 22293 at commit 6439367.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2018-08-30T22:51:44Z

now this passed 11 times on Jenkins

tdas · 2018-08-30T23:19:43Z

...kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala

        waitUntilBatchProcessed,
        CheckAnswer(),
-        WithOffsetSync(topic) { () =>
+        WithOffsetSync(topicPartition) { () =>


Its weird to have a hanging "5" in the thunk. Rather take the expected offset as part of the with. That is, WithOffsetSync(topicPartition, expectedOffset = 5) {

Good suggestion!

tdas · 2018-08-30T23:20:21Z

I was afraid that this would be flaky. Glad you found a solution quickly. just one comment to improve code readability. Otherwise LGTM. Feel free to merge when you have addressed that change.

SparkQA · 2018-08-31T00:10:43Z

Test build #4326 has finished for PR 22293 at commit e17409a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-31T00:11:25Z

Test build #4321 has finished for PR 22293 at commit e17409a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-31T00:11:47Z

Test build #4323 has finished for PR 22293 at commit e17409a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-31T00:15:28Z

Test build #4322 has finished for PR 22293 at commit e17409a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-31T00:16:25Z

Test build #4328 has finished for PR 22293 at commit e17409a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-31T00:17:06Z

Test build #4325 has finished for PR 22293 at commit e17409a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-31T00:17:39Z

Test build #4327 has finished for PR 22293 at commit e17409a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-31T00:21:09Z

Test build #4320 has finished for PR 22293 at commit e17409a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-31T00:21:12Z

Test build #4324 has finished for PR 22293 at commit e17409a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-31T00:21:15Z

Test build #95505 has finished for PR 22293 at commit e17409a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2018-08-31T06:22:38Z

Thanks! Merging to master.

## What changes were proposed in this pull request? Here are the failures: http://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.sql.kafka010.KafkaRelationSuite&test_name=read+Kafka+transactional+messages%3A+read_committed http://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.sql.kafka010.KafkaMicroBatchV1SourceSuite&test_name=read+Kafka+transactional+messages%3A+read_committed http://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.sql.kafka010.KafkaMicroBatchV2SourceSuite&test_name=read+Kafka+transactional+messages%3A+read_committed I found the Kafka consumer may not see the committed messages for a short time. This PR just adds a new method `waitUntilOffsetAppears` and uses it to make sure the consumer can see a specified offset before checking the result. ## How was this patch tested? Jenkins Closes apache#22293 from zsxwing/SPARK-25288. Authored-by: Shixiong Zhu <[email protected]> Signed-off-by: Shixiong Zhu <[email protected]>

Fix flaky Kafka transaction tests

6439367

tdas reviewed Aug 30, 2018

View reviewed changes

address TD's comment

e17409a

HeartSaVioR mentioned this pull request Aug 31, 2018

[SPARK-25151][SS] Apply Apache Commons Pool to KafkaDataConsumer #22138

Closed

asfgit closed this in aa70a0a Aug 31, 2018

zsxwing deleted the SPARK-25288 branch September 4, 2018 22:07

[SPARK-25288][Tests]Fix flaky Kafka transaction tests #22293

[SPARK-25288][Tests]Fix flaky Kafka transaction tests #22293

Uh oh!

Conversation

zsxwing commented Aug 30, 2018

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Aug 30, 2018

Uh oh!

SparkQA commented Aug 30, 2018

Uh oh!

SparkQA commented Aug 30, 2018

Uh oh!

SparkQA commented Aug 30, 2018

Uh oh!

SparkQA commented Aug 30, 2018

Uh oh!

SparkQA commented Aug 30, 2018

Uh oh!

SparkQA commented Aug 30, 2018

Uh oh!

SparkQA commented Aug 30, 2018

Uh oh!

SparkQA commented Aug 30, 2018

Uh oh!

SparkQA commented Aug 30, 2018

Uh oh!

SparkQA commented Aug 30, 2018

Uh oh!

zsxwing commented Aug 30, 2018

Uh oh!

tdas Aug 30, 2018

Choose a reason for hiding this comment

Uh oh!

zsxwing Aug 30, 2018

Choose a reason for hiding this comment

Uh oh!

tdas commented Aug 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Aug 31, 2018

Uh oh!

SparkQA commented Aug 31, 2018

Uh oh!

SparkQA commented Aug 31, 2018

Uh oh!

SparkQA commented Aug 31, 2018

Uh oh!

SparkQA commented Aug 31, 2018

Uh oh!

SparkQA commented Aug 31, 2018

Uh oh!

SparkQA commented Aug 31, 2018

Uh oh!

SparkQA commented Aug 31, 2018

Uh oh!

SparkQA commented Aug 31, 2018

Uh oh!

SparkQA commented Aug 31, 2018

Uh oh!

zsxwing commented Aug 31, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tdas commented Aug 30, 2018 •

edited

Loading