ReflectorRunnable returns the same objects again after watch-timeout and reconnect

[synox]

Describe the bug
When using the Reconciler classes, I noticed that the same events are added to the queue again after 30-45 minutes.
I expected that the watcher would not add the the same events again to the queue.

** Client Version **
11.0.0

** Kubernetes Version **
v1.19.2 (minikube)

** Java Version **
Java 11

To Reproduce
Steps to reproduce the behavior:
Create a reconciler with a custom resource.

Expected behavior
Events should not be added to the queue twice.

Additional context
In order to keep the load low on the reconciler, we currently use a resyncPeriodInMillis of 4h. Now with the described behavior, we get a bigger load than expected.

In https://github.com/kubernetes-client/java/blob/master/util/src/main/java/io/kubernetes/client/informer/cache/ReflectorRunnable.java#L117 lastSyncResourceVersion is used to track which events have been processed. However in https://github.com/kubernetes-client/java/blob/master/util/src/main/java/io/kubernetes/client/informer/SharedInformerFactory.java#L259 the parameter is ignored and it always returns new events with the current state. The docs in the code should mention why.

It seems that in the watch api you can only reference resourceVersion that are very recent (like 5min). So once the timeout of the watch request happens after 35min, it can not start another watch with the same resourceVersion. Instead it loads the current state and returns those in the queue.

[brendandburns]

etcd only keeps a short window of recent resourceVersions. If you go outside of that window and try to re-initiate a watch, it can not be established, without first re-listing all of the objects. When that happens, they are all sent through the queue.

We could in theory detect that the resourceVersion of a particular object is unchanged and not send it through.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[shoopdedoop]

I have also noticed this issue in 12.0.1. Its seems to add a lot of extra processing as we receive objects that have already been processed.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.