[CI] StandardToLogsDbIndexModeRollingUpgradeIT testLogsIndexing {upgradedNodes=3} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #29120 / bc-upgrade-tests-part2
• elasticsearch-intake #29119 / bc-upgrade-tests-part2
• elasticsearch-intake #29112 / bc-upgrade-tests-part2
• elasticsearch-intake #29111 / bc-upgrade-tests-part2
• elasticsearch-intake #29100 / bc-upgrade-tests-part2
• elasticsearch-intake #29101 / bc-upgrade-tests-part2
• elasticsearch-intake #29095 / bc-upgrade-tests-part2
• elasticsearch-intake #29093 / bc-upgrade-tests-part2
• elasticsearch-intake #29092 / bc-upgrade-tests-part2
• elasticsearch-intake #29091 / bc-upgrade-tests-part2

Reproduction Line:

./gradlew ":x-pack:plugin:logsdb:qa:rolling-upgrade:bcUpgradeTest" -Dtests.class="org.elasticsearch.upgrades.StandardToLogsDbIndexModeRollingUpgradeIT" -Dtests.method="testLogsIndexing {upgradedNodes=3}" -Dtests.seed=73DA72119193C464 -Dtests.bwc.main.version=9.2.0-SNAPSHOT -Dtests.bwc.refspec.main=1d17ce6c97775e896bc45920ced866ab08b77880 -Dtests.locale=raj-Deva-IN -Dtests.timezone=America/Hermosillo -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.RuntimeException: An error occurred while checking cluster 'test-cluster' status.

Issue Reasons:

• [main] 11 consecutive failures in step bc-upgrade-tests-part2
• [main] 2 consecutive failures in step 8.19.4_part_2_bwc-snapshots
• [main] 32 failures in test testLogsIndexing {upgradedNodes=3} (3.3% fail rate in 960 executions)
• [main] 24 failures in step bc-upgrade-tests-part2 (47.1% fail rate in 51 executions)
• [main] 3 failures in step pr-upgrade-part-2 (2.1% fail rate in 141 executions)
• [main] 2 failures in step 8.19.4_part_2_bwc-snapshots (100.0% fail rate in 2 executions)
• [main] 24 failures in pipeline elasticsearch-intake (40.0% fail rate in 60 executions)
• [main] 4 failures in pipeline elasticsearch-pull-request (2.7% fail rate in 149 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 4 consecutive failures in step 8.19.4_part_2_bwc-snapshots
• [main] 4 consecutive failures in step 9.1.4_part_2_bwc-snapshots
• [main] 18 failures in test testLogsIndexing {upgradedNodes=3} (1.9% fail rate in 945 executions)
• [main] 3 failures in step 9.0.7_part_2_bwc-snapshots (60.0% fail rate in 5 executions)
• [main] 4 failures in step 8.19.4_part_2_bwc-snapshots (100.0% fail rate in 4 executions)
• [main] 4 failures in step 9.1.4_part_2_bwc-snapshots (100.0% fail rate in 4 executions)
• [main] 4 failures in pipeline elasticsearch-pull-request (2.5% fail rate in 159 executions)

Build Scans:

• elasticsearch-intake #28931 / bc-upgrade-tests-part2
• elasticsearch-periodic-java-ea #31 / 9.1.5_bwc
• elasticsearch-periodic-java-ea #31 / 9.0.8_bwc
• elasticsearch-periodic-java-ea #31 / 8.19.5_bwc
• elasticsearch-periodic-java-ea #31 / 9.0.8_matrix-bwc
• elasticsearch-periodic-java-ea #31 / 8.19.5_matrix-bwc
• elasticsearch-periodic-java-ea #31 / 9.1.5_matrix-bwc
• elasticsearch-pull-request #94114 / 8.19.4_part_2_bwc-snapshots
• elasticsearch-pull-request #94114 / 9.1.4_part_2_bwc-snapshots
• elasticsearch-pull-request #94114 / 9.0.7_part_2_bwc-snapshots

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 18 failures in test testLogsIndexing {upgradedNodes=3} (2.1% fail rate in 872 executions)
• [main] 16 failures in step bc-upgrade-tests-part2 (37.2% fail rate in 43 executions)
• [main] 16 failures in pipeline elasticsearch-intake (37.2% fail rate in 43 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (1.6% fail rate in 127 executions)

Build Scans:

• elasticsearch-intake #29015 / bc-upgrade-tests-part2
• elasticsearch-intake #29006 / bc-upgrade-tests-part2
• elasticsearch-intake #28998 / bc-upgrade-tests-part2
• elasticsearch-intake #28991 / bc-upgrade-tests-part2
• elasticsearch-intake #28989 / bc-upgrade-tests-part2
• elasticsearch-intake #28987 / bc-upgrade-tests-part2
• elasticsearch-intake #28983 / bc-upgrade-tests-part2
• elasticsearch-intake #28984 / bc-upgrade-tests-part2
• elasticsearch-intake #28979 / bc-upgrade-tests-part2
• elasticsearch-pull-request #94592 / pr-upgrade-part-2