Race in shutting down event broadcasters (manifests as test flake)

[DirectXMan12]

Full description of the issue is at kubernetes/kubernetes#94906.

The TL;DR is that it's unherently unsafe (racey) to stop an event broadcaster due to the way it does non-blocking event emission. We might have to fork that code entirely to fix it in a timely manner.

Symptoms look like

relevant race detector output/stack traces

==================
WARNING: DATA RACE
Write at 0x00c0001762b0 by goroutine 171:
  runtime.closechan()
      /usr/local/go/src/runtime/chan.go:334 +0x0
  k8s.io/client-go/tools/record.(*eventBroadcasterImpl).Shutdown()
      /home/prow/go/pkg/mod/k8s.io/[email protected]/pkg/watch/mux.go:199 +0x68
  sigs.k8s.io/controller-runtime/pkg/internal/recorder.(*Provider).Stop.func1()
      /home/prow/go/src/sigs.k8s.io/controller-runtime/pkg/internal/recorder/recorder.go:73 +0xa0
Previous read at 0x00c0001762b0 by goroutine 168:
  runtime.chansend()
      /usr/local/go/src/runtime/chan.go:142 +0x0
  k8s.io/client-go/tools/record.(*recorderImpl).generateEvent.func1()
      /home/prow/go/pkg/mod/k8s.io/[email protected]/pkg/watch/mux.go:189 +0x11b
Goroutine 171 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/internal/recorder.(*Provider).Stop()
      /home/prow/go/src/sigs.k8s.io/controller-runtime/pkg/internal/recorder/recorder.go:66 +0x8c
  sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure()
      /home/prow/go/src/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:563 +0x38d
  sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).Start.func1()
      /home/prow/go/src/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:468 +0x49
  sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).Start()
      /home/prow/go/src/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:518 +0x339
  sigs.k8s.io/controller-runtime/pkg/manager.glob..func3.3.8.3.6()
      /home/prow/go/src/sigs.k8s.io/controller-runtime/pkg/manager/manager_test.go:273 +0x22b
Goroutine 168 (finished) created at:
  k8s.io/client-go/tools/record.(*recorderImpl).generateEvent()
      /home/prow/go/pkg/mod/k8s.io/[email protected]/tools/record/event.go:341 +0x42a
  k8s.io/client-go/tools/record.(*recorderImpl).Event()
      /home/prow/go/pkg/mod/k8s.io/[email protected]/tools/record/event.go:349 +0xcf
  k8s.io/client-go/tools/record.(*recorderImpl).Eventf()
      /home/prow/go/pkg/mod/k8s.io/[email protected]/tools/record/event.go:353 +0xd7
  sigs.k8s.io/controller-runtime/pkg/internal/recorder.(*lazyRecorder).Eventf()
      /home/prow/go/src/sigs.k8s.io/controller-runtime/pkg/internal/recorder/recorder.go:151 +0xf3
  k8s.io/client-go/tools/leaderelection/resourcelock.(*LeaseLock).RecordEvent()
      /home/prow/go/pkg/mod/k8s.io/[email protected]/tools/leaderelection/resourcelock/leaselock.go:85 +0x30b
  k8s.io/client-go/tools/leaderelection/resourcelock.(*MultiLock).RecordEvent()
      /home/prow/go/pkg/mod/k8s.io/[email protected]/tools/leaderelection/resourcelock/multilock.go:88 +0xa5
  k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1()
      /home/prow/go/pkg/mod/k8s.io/[email protected]/tools/leaderelection/leaderelection.go:251 +0x213
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      /home/prow/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x6f
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      /home/prow/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xb3
  k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      /home/prow/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x10d
  k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire()
      /home/prow/go/pkg/mod/k8s.io/[email protected]/tools/leaderelection/leaderelection.go:244 +0x2b2
  k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run()
      /home/prow/go/pkg/mod/k8s.io/[email protected]/tools/leaderelection/leaderelection.go:203 +0xee
==================

[DirectXMan12]

/kind bug
/kind flake
/priority critical-urgent

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[camilamacedo86]

@vincepri since it is priority bug, could we not add it to a milestone 0.9 or 0.10?

[vincepri]

I was waiting for the bug to be cherry picked, but yeah we can add it

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.