Flaky Test: TestCheckRetryHandles429And5xx has a race condition

[MadhavJivrajani]

Which jobs are flaking?

https://testgrid.k8s.io/sig-release-master-blocking#ci-kubernetes-unit

Which tests are flaking?

TestCheckRetryHandles429And5xx

Since when has it been flaking?

9:20am IST / 3:50am UTC

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#ci-kubernetes-unit

Reason for failure (if possible)

Race condition
Test logs: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-unit/1546340488976535552

Seems like this is again due to golang/go#51907? (race condition in upstream Go)

Anything else we need to know?

No response

Relevant SIG(s)

/sig api-machinery

[MadhavJivrajani]

@aojea @liggitt - FYI since we ran into this last release as well.

[leilajal]

/triage accepted

[muyangren2]

/assign

[aojea]

@muyangren2 we have to repro it first, I can't repro it with go 1.18.3

$ /home/aojea/sdk/go1.18.3/bin/go test -timeout 120s -run ^TestCheckRetryHandles429And5xx$ k8s.io/client-go/rest -v -race -c
$ stress ./rest.test -test.run ^TestCheckRetryHandles429And5xx$ -test.v
5s: 994 runs so far, 0 failures
10s: 2013 runs so far, 0 failures
15s: 3027 runs so far, 0 failures
20s: 4042 runs so far, 0 failures
25s: 5054 runs so far, 0 failures
30s: 6070 runs so far, 0 failures
35s: 7076 runs so far, 0 failures
40s: 8083 runs so far, 0 failures
45s: 9076 runs so far, 0 failures
50s: 10079 runs so far, 0 failures
55s: 11077 runs so far, 0 failures
1m0s: 12083 runs so far, 0 failures
1m5s: 13082 runs so far, 0 failures
1m10s: 14096 runs so far, 0 failures
1m15s: 15107 runs so far, 0 failures
1m20s: 16122 runs so far, 0 failures
1m25s: 17133 runs so far, 0 failures
1m30s: 18152 runs so far, 0 failures
1m35s: 19157 runs so far, 0 failures
1m40s: 20165 runs so far, 0 failures
1m45s: 21164 runs so far, 0 failures
1m50s: 22168 runs so far, 0 failures
1m55s: 23162 runs so far, 0 failures
2m0s: 24170 runs so far, 0 failures
2m5s: 25164 runs so far, 0 failures
2m10s: 26175 runs so far, 0 failures
2m15s: 27182 runs so far, 0 failures
2m20s: 28199 runs so far, 0 failures
2m25s: 29210 runs so far, 0 failures
2m30s: 30222 runs so far, 0 failures
2m35s: 31242 runs so far, 0 failures
^C

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.