Mixed Cluster Tests failing after joda migration

[jbaiera]

Tests are failing on 7.x, 7.2, and 7.1 due to unexpected warnings returned from 6.8 nodes pertaining to date pattern usages.

Seems to be related to #41956, Looks like the tests need to be updated to expect warnings.

Muting the tests, will report back here with the commits of the mutes.

Failures:

7.x:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+periodic/618/console
https://gradle.com/s/r6caor7lose3i

:qa:mixed-cluster:v6.8.1#mixedClusterTestRunner FAILED
--
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
 
 
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=search/140_pre_filter_search_shards/pre_filter_shard_size with shards that have no hit} FAILED
java.lang.AssertionError: Failure at [search/140_pre_filter_search_shards:2]: got unexpected warning header [
299 Elasticsearch-6.8.1-SNAPSHOT-6853a61 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier."
]
 
Caused by:
java.lang.AssertionError: got unexpected warning header [
299 Elasticsearch-6.8.1-SNAPSHOT-6853a61 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier."
]
REPRODUCE WITH: ./gradlew :qa:mixed-cluster:v6.8.1#mixedClusterTestRunner --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search/140_pre_filter_search_shards/pre_filter_shard_size with shards that have no hit}" -Dtests.seed=49C104CFAF9D0BDB -Dtests.security.manager=true -Dtests.locale=en-US -Dtests.timezone=America/Buenos_Aires -Dcompiler.java=12 -Druntime.java=8
 
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=search/140_pre_filter_search_shards/pre_filter_shard_size with invalid parameter} FAILED
java.lang.AssertionError: Failure at [search/140_pre_filter_search_shards:2]: got unexpected warning header [
299 Elasticsearch-6.8.1-SNAPSHOT-6853a61 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier."
]
 
Caused by:
java.lang.AssertionError: got unexpected warning header [
299 Elasticsearch-6.8.1-SNAPSHOT-6853a61 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier."
]

7.2:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.2+intake/51/console
https://scans.gradle.com/s/g62fd6k2pxd5m

:qa:mixed-cluster:v6.8.1#mixedClusterTestRunner FAILED
--
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
 
 
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=search/140_pre_filter_search_shards/pre_filter_shard_size with shards that have no hit} FAILED
java.lang.AssertionError: Failure at [search/140_pre_filter_search_shards:2]: got unexpected warning header [
299 Elasticsearch-6.8.1-SNAPSHOT-6853a61 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier."
]
 
Caused by:
java.lang.AssertionError: got unexpected warning header [
299 Elasticsearch-6.8.1-SNAPSHOT-6853a61 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier."
]
REPRODUCE WITH: ./gradlew :qa:mixed-cluster:v6.8.1#mixedClusterTestRunner --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search/140_pre_filter_search_shards/pre_filter_shard_size with shards that have no hit}" -Dtests.seed=282C405AE3A5C1C7 -Dtests.security.manager=true -Dtests.locale=ar-IQ -Dtests.timezone=America/Shiprock -Dcompiler.java=12 -Druntime.java=8
 
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=search/140_pre_filter_search_shards/pre_filter_shard_size with invalid parameter} FAILED
java.lang.AssertionError: Failure at [search/140_pre_filter_search_shards:2]: got unexpected warning header [
299 Elasticsearch-6.8.1-SNAPSHOT-6853a61 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier."
]
 
Caused by:
java.lang.AssertionError: got unexpected warning header [
299 Elasticsearch-6.8.1-SNAPSHOT-6853a61 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier."
]

7.1:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.1+default-distro/84/console
https://gradle.com/s/ysje3e4pgzj4e

:qa:mixed-cluster:v6.8.1#mixedClusterTestRunner FAILED
--
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
 
 
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=search/140_pre_filter_search_shards/pre_filter_shard_size with invalid parameter} FAILED
java.lang.AssertionError: Failure at [search/140_pre_filter_search_shards:2]: got unexpected warning header [
299 Elasticsearch-6.8.1-SNAPSHOT-6853a61 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier."
]
 
Caused by:
java.lang.AssertionError: got unexpected warning header [
299 Elasticsearch-6.8.1-SNAPSHOT-6853a61 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier."
]
REPRODUCE WITH: ./gradlew :qa:mixed-cluster:v6.8.1#mixedClusterTestRunner --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search/140_pre_filter_search_shards/pre_filter_shard_size with invalid parameter}" -Dtests.seed=20C7A190F4DD968A -Dtests.security.manager=true -Dtests.locale=uk -Dtests.timezone=Europe/Brussels -Dtests.distribution=default -Dcompiler.java=12 -Druntime.java=8
 
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=search/140_pre_filter_search_shards/pre_filter_shard_size with shards that have no hit} FAILED
java.lang.AssertionError: Failure at [search/140_pre_filter_search_shards:2]: got unexpected warning header [
299 Elasticsearch-6.8.1-SNAPSHOT-6853a61 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier."
]
 
Caused by:
java.lang.AssertionError: got unexpected warning header [
299 Elasticsearch-6.8.1-SNAPSHOT-6853a61 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier."
]
REPRODUCE WITH: ./gradlew :qa:mixed-cluster:v6.8.1#mixedClusterTestRunner --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search/140_pre_filter_search_shards/pre_filter_shard_size with shards that have no hit}" -Dtests.seed=20C7A190F4DD968A -Dtests.security.manager=true -Dtests.locale=uk -Dtests.timezone=Europe/Brussels -Dtests.distribution=default -Dcompiler.java=12 -Druntime.java=8
 
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=search.aggregation/230_composite/Composite aggregation with format} FAILED
java.lang.AssertionError: Failure at [search.aggregation/230_composite:248]: got unexpected warning header [
299 Elasticsearch-6.8.1-SNAPSHOT-6853a61 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier."
]
 
Caused by:
java.lang.AssertionError: got unexpected warning header [
299 Elasticsearch-6.8.1-SNAPSHOT-6853a61 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier."
]
REPRODUCE WITH: ./gradlew :qa:mixed-cluster:v6.8.1#mixedClusterTestRunner --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search.aggregation/230_composite/Composite aggregation with format}" -Dtests.seed=20C7A190F4DD968A -Dtests.security.manager=true -Dtests.locale=uk -Dtests.timezone=Europe/Brussels -Dtests.distribution=default -Dcompiler.java=12 -Druntime.java=8

[elasticmachine]

Pinging @elastic/es-core-infra

[jbaiera]

Muted on 7.x, 7.2, and 7.1

[pgomulka]

I wanted to update the pattern that was generating the failure yyyy-MM-dd to the intermediate 8yyyy-MM-dd which should work in both 7x and 6x but I was getting this stacktrace as below. I decided to use strict_date which uses the same pattern.

[2019-05-30T09:48:46,632][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-3] fatal error in thread [elasticsearch[node-3][clusterApplierService#updateTask][T#1]], exiting
java.lang.AssertionError: {_doc=org.elasticsearch.index.mapper.DocumentMapper@5b18875d}
        at org.elasticsearch.index.mapper.MapperService.assertMappingVersion(MapperService.java:268) ~[elasticsearch-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
        at org.elasticsearch.index.mapper.MapperService.updateMapping(MapperService.java:222) ~[elasticsearch-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
        at org.elasticsearch.index.IndexService.updateMapping(IndexService.java:570) ~[elasticsearch-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.updateIndices(IndicesClusterStateService.java:538) ~[elasticsearch-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:263) ~[elasticsearch-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$5(ClusterApplierService.java:495) ~[elasticsearch-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
        at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_181]
        at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:493) ~[elasticsearch-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:464) ~[elasticsearch-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:418) ~[elasticsearch-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]