Cache doesn't start if it's config refers to types that are not installed

[tiraboschi]

controller-runtime v0.16.0 fails starting the manager when the operator is using cache.Options with kinds identified ByObject that are not present on the specific cluster.

This could happen for instance trying to finetune upfront the cache also for monitoring kinds when the prometheous stack is not deployed or OKD/OCP specific kinds when on vanilla k8s.

Now the operator dies with something like:

failed to determine if *v1.ConsoleQuickStart is namespaced: failed to get restmapping: failed to find API group "console.openshift.io"","stacktrace":"github.com/kubevirt/hyperconverged-cluster-operator/cmd/cmdcommon.HcCmdHelper.ExitOnError
	/go/src/github.com/kubevirt/hyperconverged-cluster-operator/cmd/cmdcommon/cmdcommon.go:97
main.main
	/go/src/github.com/kubevirt/hyperconverged-cluster-operator/cmd/hyperconverged-cluster-operator/main.go:104
runtime.main
	/usr/local/go/src/runtime/proc.go:250

the same code was working with controller-runtime <= v0.15.0.

This is probably got probably changed with: #2421
The error comes from

controller-runtime/pkg/cache/cache.go

Lines 399 to 405 in c20ea14

	isNamespaced, err := apiutil.IsObjectNamespaced(obj, opts.Scheme, opts.Mapper)
	if err != nil {
	return opts, fmt.Errorf("failed to determine if %T is namespaced: %w", obj, err)
	}
	if !isNamespaced && byObject.Namespaces != nil {
	return opts, fmt.Errorf("type %T is not namespaced, but its ByObject.Namespaces setting is not nil", obj)
	}

that is supposed to use isNamespaced during cache initialization to ensure that byObject.Namespaces is not set for cluster-scope kind.

A simple fix is just about skipping that additional check for non existing kinds instead of breaking the cache initialization and so the manager one.

[tiraboschi]

@alvaroaleman FYI

[alvaroaleman]

So you have kinds in your cache config that you don't watch, since you say the config worked in 0.15? That doesn't seem correct

[troy0820]

/kind bug support

[alvaroaleman]

/retitle Cache doesn't start if it's config refers to types that are not installed

[tiraboschi]

So you have kinds in your cache config that you don't watch, since you say the config worked in 0.15? That doesn't seem correct

No, not really.
During its initialization our operator is using the APIReader client (whis is not cache based) to try getting specific "optional" (like OKD/OCP console kinds since we have a dynamic plugin for the UI or the monitoring stack that can be skipped on low footprint environments) resources and according to the eventual error we detect a missing kind.

Based on those results, we add the "optional" kinds to the list of watched ones:
https://github.com/kubevirt/hyperconverged-cluster-operator/blob/ae66846d229c1500aadcf48cb24b2e95b2a0547d/controllers/hyperconverged/hyperconverged_controller.go#L190-L207

This was working in v0.15 and earlier, but now it's not working with v0.16 since we fail the initialization of the manager due to the missing kinds in cache sanity checks.

A possible workaround on our side is bootstrapping a first manager with no custom cache (or using another kind of client just for that), detect the presence of the optional kinds earlier and use its results to bootstrap the real manager we are going to use with a cache configured only for existing kinds but this sounds overkill to me.

[alvaroaleman]

Well if you already have code to detect presence and create watches, you can use that same code to conditionally add cache config for those types depending on presence

[tiraboschi]

Well if you already have code to detect presence and create watches, you can use that same code to conditionally add cache config for those types depending on presence

But we have another issue here:
currently our detect code is using the non-caching client we get from mgr.GetAPIReader(), but with v0.16 we are not able to reach that stage single we fail earlier in mgr, err := manager.New( when we try to instantiate the manager with a cache config referring missing kinds.

Can we instantiate the manager with the default cache and configure a custom one only in a later stage?

[alvaroaleman]

You don't need a manager to get a client. You can use client.New to get a client

[tiraboschi]

Thanks, it correctly worked for our case.

[troy0820]

Can this be closed?
/remove-kind bug