[CI] LogsUsageRollingUpgradeIT testUsage {upgradedNodes=3} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #29099 / bc-upgrade-tests-part2
• elasticsearch-intake #29097 / bc-upgrade-tests-part2
• elasticsearch-intake #29093 / bc-upgrade-tests-part2
• elasticsearch-intake #29092 / bc-upgrade-tests-part2
• elasticsearch-intake #29091 / bc-upgrade-tests-part2
• elasticsearch-intake #29090 / bc-upgrade-tests-part2
• elasticsearch-intake #29088 / bc-upgrade-tests-part2
• elasticsearch-pull-request #94818 / 9.2.0_part_2_bwc-snapshots
• elasticsearch-pull-request #94818 / 9.1.4_part_2_bwc-snapshots
• elasticsearch-pull-request #94818 / 8.19.4_part_2_bwc-snapshots

Reproduction Line:

./gradlew ":x-pack:plugin:logsdb:qa:rolling-upgrade:bcUpgradeTest" -Dtests.class="org.elasticsearch.upgrades.LogsUsageRollingUpgradeIT" -Dtests.method="testUsage {upgradedNodes=3}" -Dtests.seed=EA207FF7C624349B -Dtests.bwc.main.version=9.2.0-SNAPSHOT -Dtests.bwc.refspec.main=1d17ce6c97775e896bc45920ced866ab08b77880 -Dtests.locale=nb-SJ -Dtests.timezone=America/Port_of_Spain -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.RuntimeException: An error occurred while checking cluster 'test-cluster' status.

Issue Reasons:

• [main] 24 consecutive failures in step bc-upgrade-tests-part2
• [main] 22 consecutive failures in step 9.1.4_part_2_bwc-snapshots
• [main] 20 consecutive failures in step 8.19.4_part_2_bwc-snapshots
• [main] 7 consecutive failures in step 9.0.7_part_2_bwc-snapshots
• [main] 4 consecutive failures in step pr-upgrade-part-2
• [main] 5 consecutive failures in step 9.1.5_bwc
• [main] 5 consecutive failures in step 9.0.8_bwc
• [main] 5 consecutive failures in step 9.0.8_matrix-bwc
• [main] 5 consecutive failures in step 8.19.5_matrix-bwc
• [main] 5 consecutive failures in step 9.1.5_matrix-bwc
• [main] 2 consecutive failures in step 9.0.8_part_2_bwc-snapshots
• [main] 7 consecutive failures in pipeline elasticsearch-intake
• [main] 5 consecutive failures in pipeline elasticsearch-periodic-java-ea
• [main] 113 failures in test testUsage {upgradedNodes=3} (12.7% fail rate in 888 executions)
• [main] 24 failures in step bc-upgrade-tests-part2 (100.0% fail rate in 24 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 5 consecutive failures in step 9.1.5_bwc
• [main] 5 consecutive failures in step 9.0.8_bwc
• [main] 5 consecutive failures in step 9.0.8_matrix-bwc
• [main] 5 consecutive failures in step 8.19.5_matrix-bwc
• [main] 5 consecutive failures in step 9.1.5_matrix-bwc
• [main] 19 consecutive failures in step 8.19.4_part_2_bwc-snapshots
• [main] 40 consecutive failures in step 9.1.4_part_2_bwc-snapshots
• [main] 16 consecutive failures in step 9.0.7_part_2_bwc-snapshots
• [main] 5 consecutive failures in step pr-upgrade-part-2
• [main] 2 consecutive failures in step 9.0.8_part_2_bwc-snapshots
• [main] 5 consecutive failures in pipeline elasticsearch-periodic-java-ea
• [main] 137 failures in test testUsage {upgradedNodes=3} (14.1% fail rate in 972 executions)
• [main] 2 failures in step 8.19.5_part_2_bwc-snapshots (0.5% fail rate in 428 executions)
• [main] 5 failures in step 8.19.5_bwc (25.0% fail rate in 20 executions)
• [main] 5 failures in step 9.1.5_bwc (100.0% fail rate in 5 executions)

Build Scans:

• elasticsearch-intake #28931 / bc-upgrade-tests-part2
• elasticsearch-periodic-java-ea #31 / 9.1.5_bwc
• elasticsearch-periodic-java-ea #31 / 9.0.8_bwc
• elasticsearch-periodic-java-ea #31 / 8.19.5_bwc
• elasticsearch-periodic-java-ea #31 / 9.0.8_matrix-bwc
• elasticsearch-periodic-java-ea #31 / 8.19.5_matrix-bwc
• elasticsearch-periodic-java-ea #31 / 9.1.5_matrix-bwc
• elasticsearch-pull-request #94114 / 8.19.4_part_2_bwc-snapshots
• elasticsearch-pull-request #94114 / 9.1.4_part_2_bwc-snapshots
• elasticsearch-pull-request #94114 / 9.0.7_part_2_bwc-snapshots

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)