[SPARK-29957][TEST] Reset MiniKDC's default enctypes to fit jdk8/jdk11 #26594

AngersZhuuuu · 2019-11-19T07:26:17Z

What changes were proposed in this pull request?

Hadoop jira: https://issues.apache.org/jira/browse/HADOOP-12911
In this jira, the author said to replace origin Apache Directory project which is not maintained (but not said it won't work well in jdk11) to Apache Kerby which is java binding(fit java version).

And in Flink: apache/flink#9622
Author show the reason why hadoop-2.7.2's MminiKdc failed with jdk11.
Because new encryption types of es128-cts-hmac-sha256-128 and aes256-cts-hmac-sha384-192 (for Kerberos 5) enabled by default were added in Java 11.
Spark with hadoop-2.7's MiniKdcdoes not support these encryption types and does not work well when these encryption types are enabled, which results in the authentication failure.

And when I test hadoop-2.7.2's minikdc in local, the kerberos 's debug error message is read message stream failed, message can't match.

Why are the changes needed?

Support jdk11 with hadoop-2.7

Does this PR introduce any user-facing change?

NO

How was this patch tested?

Existed UT

AngersZhuuuu · 2019-11-19T07:28:37Z

cc @wangyum @dongjoon-hyun
This way can fix error happened in #26533 (comment)

wangyum · 2019-11-19T07:31:23Z

Thank you @AngersZhuuuu

wangyum · 2019-11-19T07:37:36Z

ok to test

SparkQA · 2019-11-19T08:05:02Z

Test build #114072 has finished for PR 26594 at commit e9ef3ba.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-11-19T08:55:27Z

retest this please

SparkQA · 2019-11-19T10:52:43Z

Test build #114083 has finished for PR 26594 at commit e9ef3ba.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2019-11-19T11:27:35Z

strange..I can run these failed UT in local with jdk8/jdk11 success. But failed in Jenkines...

dongjoon-hyun · 2019-11-24T00:39:07Z

Retest this please.

AngersZhuuuu · 2019-11-24T00:53:44Z

Retest this please.

For this. I can pass failed UT by run it alone, but will failed when run all UT together.
Seems unit tests affect each other in terms of environment variables. I am checking on this.

dongjoon-hyun · 2019-11-24T00:56:12Z

@AngersZhuuuu . I also tested your PR locally.
That's the reason I re-trigger this. If this fails on my local environment, I didn't trigger an useless run. 😄

BTW, the first failure was KafkaDelegationTokenSuite at last Jenkins run.

dongjoon-hyun · 2019-11-24T00:59:46Z

BTW, if you can find a corresponding Apache Hadoop JIRA issue, that will be more persuasive
for this PR.

dongjoon-hyun · 2019-11-24T01:02:35Z

pom.xml

    <slf4j.version>1.7.16</slf4j.version>
    <log4j.version>1.2.17</log4j.version>
    <hadoop.version>2.7.4</hadoop.version>
+    <miniKdc.version>3.2.0</miniKdc.version>


BTW, miniKdc -> minikdc. As you see, Spark-introduced properties are not camelcase like codahale and htmlunit. If needed, we may use -, but I prefer minikdc here in this case.

BTW, miniKdc -> minikdc. As you see, Spark-introduced properties are are camelcase like codahale and htmlunit. If needed, we may use -, but I prefer minikdc here in this case.

Thank you for your careful explanation. Updated.

AngersZhuuuu · 2019-11-24T01:10:59Z

BTW, if you can find a corresponding Apache Hadoop JIRA issue, that will be more persuasive
for this PR.

Hadoop jira: https://issues.apache.org/jira/browse/HADOOP-12911
In this jira, the author said to replace origin Apache Directory project which is not maintained (but not said it won't work well in jdk11) to Apache kerby which is java binding(fit all java version).

And in Flink: apache/flink#9622
Author show the reason why hadoop-2.7.2's MminiKdcfailed with jdk11. Because new encryption types ofes128-cts-hmac-sha256-128andaes256-cts-hmac-sha384-192` (for Kerberos 5) enabled by default were added in Java 11.

And when I test hadoop-2.7.2's minikdc in local, the kerberos 's debug error message is read message stream failed, message can't match.

dongjoon-hyun · 2019-11-24T01:15:35Z

Please summarize them into the PR description~

AngersZhuuuu · 2019-11-24T01:19:16Z

Please summarize them into the PR description~

Done

dongjoon-hyun · 2019-11-24T02:18:47Z

Hmm. It seems that KafkaDelegationTokenSuite fails again in Jenkins.

[info] org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite *** ABORTED *** (189 milliseconds)
[info]   java.io.IOException: Could not configure server because SASL configuration did not allow the  ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: kerberos.example.com: Name or service not known

I tested this only at Mac with AdoptOpenJDK8 and 11. How about your environment? I'm wondering if we need to validate this in the linux or not because of the following error message.

javax.security.auth.login.LoginException:
kerberos.example.com: Name or service not known

SparkQA · 2019-11-24T02:18:51Z

Test build #114327 has finished for PR 26594 at commit e9ef3ba.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2019-11-24T02:29:30Z

I tested this only at Mac with AdoptOpenJDK8 and 11. How about your environment? I'm wondering if we need to validate this in the linux or not because of the following error message.

I test this in Mac with Oracle Jdk8 and jdk11. I can pass this UT when run it alone.
I can test this in Debian.

My guess is that there were other tests that affected the environment variables

dongjoon-hyun · 2019-11-24T02:31:23Z

Ya. Let me know if you find the root cause. Thanks.

SparkQA · 2019-11-24T03:16:36Z

Test build #114329 has finished for PR 26594 at commit 3ec2891.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-24T10:54:15Z

Test build #114339 has finished for PR 26594 at commit 2e282f2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-24T11:08:56Z

Test build #114340 has finished for PR 26594 at commit a224fad.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-30T13:31:51Z

Test build #114658 has finished for PR 26594 at commit 476d60e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-12-01T07:45:53Z

retest this please

SparkQA · 2019-12-01T08:28:54Z

Test build #114677 has finished for PR 26594 at commit 476d60e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2019-12-03T12:03:25Z

I think the test comment is not yet resolved: https://github.com/apache/spark/pull/26594/files#r350680194

AngersZhuuuu · 2019-12-04T09:34:13Z

@gaborgsomogyi @wangyum How about current way ?
If don't have a libdefaults section , we add one at krb5.conf's header

SparkQA · 2019-12-04T09:41:51Z

Test build #114848 has finished for PR 26594 at commit b782e99.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-12-04T09:50:04Z

Test build #114850 has finished for PR 26594 at commit 10a0006.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-12-04T10:19:23Z

Test build #114851 has finished for PR 26594 at commit 03f7fb1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2019-12-04T10:51:05Z

The approach looks good, there are still no tests...

AngersZhuuuu · 2019-12-04T10:56:47Z

The approach looks good, there are still no tests...

Here is in UT ..., I really don't know how to add UT for this, so handle it in if condition when there are no libdefaults in krb5.conf

external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala

SparkQA · 2019-12-05T02:17:35Z

Test build #114884 has finished for PR 26594 at commit e252dd0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala

dongjoon-hyun · 2019-12-05T16:09:01Z

external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala

+    // scalastyle:off println
+    writer.println(krb5confStr)
+    // scalastyle:on println
+    writer.close()


Shall we use Files.write(content, file, StandardCharsets.UTF_8) instead of println?

- val writer = new PrintWriter(kdc.getKrb5conf) - // scalastyle:off println - writer.println(krb5confStr) - // scalastyle:on println - writer.close() + Files.write(krb5confStr, kdc.getKrb5conf, StandardCharsets.UTF_8)

Shall we use Files.write(content, file, StandardCharsets.UTF_8) instead of println?

Done thanks.

SparkQA · 2019-12-06T05:13:12Z

Test build #114926 has finished for PR 26594 at commit 223533f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

+1, LGTM. Merged to master. Thank you, @AngersZhuuuu and all.

gatorsmile · 2019-12-06T07:13:23Z

Thanks, everyone! This is a great fix!

### What changes were proposed in this pull request? Hadoop jira: https://issues.apache.org/jira/browse/HADOOP-12911 In this jira, the author said to replace origin Apache Directory project which is not maintained (but not said it won't work well in jdk11) to Apache Kerby which is java binding(fit java version). And in Flink: apache/flink#9622 Author show the reason why hadoop-2.7.2's `MminiKdc` failed with jdk11. Because new encryption types of `es128-cts-hmac-sha256-128` and `aes256-cts-hmac-sha384-192` (for Kerberos 5) enabled by default were added in Java 11. Spark with `hadoop-2.7's MiniKdc`does not support these encryption types and does not work well when these encryption types are enabled, which results in the authentication failure. And when I test hadoop-2.7.2's minikdc in local, the kerberos 's debug error message is read message stream failed, message can't match. ### Why are the changes needed? Support jdk11 with hadoop-2.7 ### Does this PR introduce any user-facing change? NO ### How was this patch tested? Existed UT Closes apache#26594 from AngersZhuuuu/minikdc-3.2.0. Lead-authored-by: angerszhu <[email protected]> Co-authored-by: AngersZhuuuu <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

Update pom.xml

e9ef3ba

AngersZhuuuu changed the title ~~[test-hadoop3.2][test-java11] Bump MiniKdc to 3.2.0~~ [SPARK-29957][Build] Bump MiniKdc to 3.2.0 Nov 19, 2019

wangyum changed the title ~~[SPARK-29957][Build] Bump MiniKdc to 3.2.0~~ [SPARK-29957][BUILD][TEST] Bump MiniKdc to 3.2.0 Nov 19, 2019

dongjoon-hyun added the TESTS label Nov 20, 2019

dongjoon-hyun changed the title ~~[SPARK-29957][BUILD][TEST] Bump MiniKdc to 3.2.0~~ [SPARK-29957][TEST] Bump MiniKdc to 3.2.0 Nov 24, 2019

dongjoon-hyun reviewed Nov 24, 2019

View reviewed changes

miniKdc -> minikdc

3ec2891

AngersZhuuuu added 2 commits November 24, 2019 16:29

Update KafkaTestUtils.scala

2e282f2

Update KafkaTestUtils.scala

a224fad

wangyum changed the title ~~[SPARK-29957][TEST][test-java11][test-hadoop3.2][test-maven] Reset MiniKDC's default enctypes to fit jdk8/jdk11~~ [SPARK-29957][TEST][test-java11] Reset MiniKDC's default enctypes to fit jdk8/jdk11 Dec 1, 2019

AngersZhuuuu added 3 commits December 4, 2019 16:59

Update KafkaTestUtils.scala

b782e99

Update KafkaTestUtils.scala

10a0006

Update KafkaTestUtils.scala

03f7fb1

dongjoon-hyun reviewed Dec 4, 2019

View reviewed changes

external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Dec 4, 2019

View reviewed changes

external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala Outdated Show resolved Hide resolved

Update KafkaTestUtils.scala

e252dd0

wangyum mentioned this pull request Dec 5, 2019

[WIP][test-java11] Test Hadoop 2.7 with JDK 11 #26533

Closed

dongjoon-hyun reviewed Dec 5, 2019

View reviewed changes

external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala Show resolved Hide resolved

dongjoon-hyun reviewed Dec 5, 2019

View reviewed changes

AngersZhuuuu added 2 commits December 6, 2019 12:26

Update KafkaTestUtils.scala

4facbee

Update KafkaTestUtils.scala

223533f

dongjoon-hyun changed the title ~~[SPARK-29957][TEST][test-java11] Reset MiniKDC's default enctypes to fit jdk8/jdk11~~ [SPARK-29957][TEST] Reset MiniKDC's default enctypes to fit jdk8/jdk11 Dec 6, 2019

dongjoon-hyun approved these changes Dec 6, 2019

View reviewed changes

dongjoon-hyun closed this in da27f91 Dec 6, 2019

[SPARK-29957][TEST] Reset MiniKDC's default enctypes to fit jdk8/jdk11 #26594

[SPARK-29957][TEST] Reset MiniKDC's default enctypes to fit jdk8/jdk11 #26594

Uh oh!

Conversation

AngersZhuuuu commented Nov 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

AngersZhuuuu commented Nov 19, 2019

Uh oh!

wangyum commented Nov 19, 2019

Uh oh!

wangyum commented Nov 19, 2019

Uh oh!

SparkQA commented Nov 19, 2019

Uh oh!

wangyum commented Nov 19, 2019

Uh oh!

SparkQA commented Nov 19, 2019

Uh oh!

AngersZhuuuu commented Nov 19, 2019

Uh oh!

dongjoon-hyun commented Nov 24, 2019

Uh oh!

AngersZhuuuu commented Nov 24, 2019

Uh oh!

dongjoon-hyun commented Nov 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Nov 24, 2019

Uh oh!

dongjoon-hyun Nov 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu Nov 24, 2019

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu commented Nov 24, 2019

Uh oh!

dongjoon-hyun commented Nov 24, 2019

Uh oh!

AngersZhuuuu commented Nov 24, 2019

Uh oh!

dongjoon-hyun commented Nov 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Nov 24, 2019

Uh oh!

AngersZhuuuu commented Nov 24, 2019

Uh oh!

dongjoon-hyun commented Nov 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Nov 24, 2019

Uh oh!

SparkQA commented Nov 24, 2019

Uh oh!

SparkQA commented Nov 24, 2019

Uh oh!

SparkQA commented Nov 30, 2019

Uh oh!

wangyum commented Dec 1, 2019

Uh oh!

SparkQA commented Dec 1, 2019

Uh oh!

gaborgsomogyi commented Dec 3, 2019

Uh oh!

AngersZhuuuu commented Dec 4, 2019

Uh oh!

SparkQA commented Dec 4, 2019

Uh oh!

SparkQA commented Dec 4, 2019

Uh oh!

SparkQA commented Dec 4, 2019

Uh oh!

gaborgsomogyi commented Dec 4, 2019

AngersZhuuuu commented Nov 19, 2019 •

edited

Loading

dongjoon-hyun commented Nov 24, 2019 •

edited

Loading

dongjoon-hyun Nov 24, 2019 •

edited

Loading

dongjoon-hyun commented Nov 24, 2019 •

edited

Loading

dongjoon-hyun commented Nov 24, 2019 •

edited

Loading