[CI] TextRollingUpgradeIT testIndexing {upgradedNodes=3} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #29118 / bc-upgrade-tests-part2
• elasticsearch-intake #29114 / bc-upgrade-tests-part2
• elasticsearch-intake #29112 / bc-upgrade-tests-part2
• elasticsearch-intake #29113 / bc-upgrade-tests-part2
• elasticsearch-intake #29111 / bc-upgrade-tests-part2
• elasticsearch-intake #29100 / bc-upgrade-tests-part2
• elasticsearch-intake #29101 / bc-upgrade-tests-part2
• elasticsearch-intake #29099 / bc-upgrade-tests-part2
• elasticsearch-intake #29095 / bc-upgrade-tests-part2
• elasticsearch-intake #29097 / bc-upgrade-tests-part2

Reproduction Line:

./gradlew ":x-pack:plugin:logsdb:qa:rolling-upgrade:bcUpgradeTest" -Dtests.class="org.elasticsearch.upgrades.TextRollingUpgradeIT" -Dtests.method="testIndexing {upgradedNodes=3}" -Dtests.seed=9D2167F5A6C69837 -Dtests.bwc.main.version=9.2.0-SNAPSHOT -Dtests.bwc.refspec.main=1d17ce6c97775e896bc45920ced866ab08b77880 -Dtests.locale=en-JE -Dtests.timezone=Europe/Guernsey -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.RuntimeException: An error occurred while checking cluster 'test-cluster' status.

Issue Reasons:

• [main] 14 consecutive failures in step bc-upgrade-tests-part2
• [main] 30 failures in test testIndexing {upgradedNodes=3} (3.4% fail rate in 886 executions)
• [main] 23 failures in step bc-upgrade-tests-part2 (41.8% fail rate in 55 executions)
• [main] 3 failures in step pr-upgrade-part-2 (2.3% fail rate in 129 executions)
• [main] 23 failures in pipeline elasticsearch-intake (39.7% fail rate in 58 executions)
• [main] 3 failures in pipeline elasticsearch-pull-request (2.2% fail rate in 135 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch 9.1

Mute Reasons:

• [9.1] 3 failures in test testIndexing {upgradedNodes=3} (0.6% fail rate in 524 executions)
• [9.1] 2 failures in pipeline elasticsearch-pull-request (4.3% fail rate in 46 executions)

Build Scans:

• elasticsearch-periodic #10772 / 8.19.5_bwc
• elasticsearch-pull-request #92719 / 8.19.4_part_2_bwc-snapshots
• elasticsearch-pull-request #92362 / 9.0.7_part_2_bwc-snapshots

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 consecutive failures in step 8.19.4_part_2_bwc-snapshots
• [main] 2 consecutive failures in step 9.1.4_part_2_bwc-snapshots
• [main] 2 consecutive failures in step 9.0.7_part_2_bwc-snapshots
• [main] 13 failures in test testIndexing {upgradedNodes=3} (1.4% fail rate in 912 executions)
• [main] 2 failures in step 8.19.4_part_2_bwc-snapshots (100.0% fail rate in 2 executions)
• [main] 2 failures in step 9.1.4_part_2_bwc-snapshots (100.0% fail rate in 2 executions)
• [main] 2 failures in step 9.0.7_part_2_bwc-snapshots (100.0% fail rate in 2 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (1.3% fail rate in 151 executions)

Build Scans:

• elasticsearch-intake #28936 / bc-upgrade-tests-part2
• elasticsearch-periodic-java-ea #31 / 9.1.5_bwc
• elasticsearch-periodic-java-ea #31 / 9.0.8_bwc
• elasticsearch-periodic-java-ea #31 / 8.19.5_bwc
• elasticsearch-periodic-java-ea #31 / 9.0.8_matrix-bwc
• elasticsearch-periodic-java-ea #31 / 8.19.5_matrix-bwc
• elasticsearch-periodic-java-ea #31 / 9.1.5_matrix-bwc
• elasticsearch-pull-request #94114 / 8.19.4_part_2_bwc-snapshots
• elasticsearch-pull-request #94114 / 9.1.4_part_2_bwc-snapshots
• elasticsearch-pull-request #94114 / 9.0.7_part_2_bwc-snapshots