OCPBUGS-64719: Add APIServer to HybridOverlay config #3586

jrvaldes · 2025-11-24T17:07:28Z

This commit ensures the hybridOverlay service receives explicitly the address of the API Server Endpoint along with the cacert to enable certificate rotation.

The --bootstrap-kubeconfig flag should be enough but there is a bug
in OVN-K HybridOverlay [1] where the apiserver and cacert are not extracted
from the bootstrap information, hence introducing the --k8s-apiserver and
--k8s-cacert flags as part of the command.

[1] https://issues.redhat.com/browse/OCPBUGS-65856

hybrid-overlay.log before:

I1124 15:56:01.961205    1104 config.go:2474] Kubernetes config: {BootstrapKubeconfig:C:\k\kubeconfig CertDir:C:\k\cni\config CertDuration:10m0s Kubeconfig: CACert: CAData:[] APIServer:  Token: TokenFile: CompatServiceCIDR: RawServiceCIDRs:172.16.1.0/24 ServiceCIDRs:[172.16.1.0/24] OVNConfigNamespace:ovn-kubernetes OVNEmptyLbEvents:false PodIP: RawNoHostSubnetNodes: NoHostSubnetNodes:<nil> HostNetworkNamespace: DisableRequestedChassis:false PlatformType: HealthzBindAddress: CompatMetricsBindAddress: CompatOVNMetricsBindAddress: CompatMetricsEnablePprof:false DNSServiceNamespace:kube-system DNSServiceName:kube-dns}

hybrid-overlay.log with proposed implementation, see the APIServer and CACert fields populated

I1124 15:56:01.961205    1104 config.go:2474] Kubernetes config: {BootstrapKubeconfig:C:\k\kubeconfig CertDir:C:\k\cni\config CertDuration:10m0s Kubeconfig: CACert:C:\k\bootstrap-ca.crt CAData:[] APIServer:https://api-int.jvaldes.vmc.devcluster.openshift.com:6443 Token: TokenFile: CompatServiceCIDR: RawServiceCIDRs:172.16.1.0/24 ServiceCIDRs:[172.16.1.0/24] OVNConfigNamespace:ovn-kubernetes OVNEmptyLbEvents:false PodIP: RawNoHostSubnetNodes: NoHostSubnetNodes:<nil> HostNetworkNamespace: DisableRequestedChassis:false PlatformType: HealthzBindAddress: CompatMetricsBindAddress: CompatOVNMetricsBindAddress: CompatMetricsEnablePprof:false DNSServiceNamespace:kube-system DNSServiceName:kube-dns}

openshift-ci · 2025-11-24T17:08:35Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

jrvaldes · 2025-11-24T17:15:22Z

hybrid-overlay.log showing successful rotation with 10min expiration:

I1124 16:01:50.993123    1104 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client"  
I1124 16:01:51.000616    1104 reflector.go:357] "Starting reflector" logger="kubernetes.io/kube-apiserver-client" type="*v1.CertificateSigningRequest" resyncPeriod="0s" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"  
I1124 16:01:51.000616    1104 reflector.go:403] "Listing and watching" logger="kubernetes.io/kube-apiserver-client" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"  
I1124 16:01:51.001563    1104 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"  
I1124 16:01:51.005853    1104 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client" csr="csr-n7l5w"  
I1124 16:01:51.013189    1104 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client" csr="csr-n7l5w"  
I1124 16:01:51.013189    1104 reflector.go:363] "Stopping reflector" logger="kubernetes.io/kube-apiserver-client" type="*v1.CertificateSigningRequest" resyncPeriod="0s" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"  
I1124 16:01:52.023255    1104 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client" expiration="2025-11-24 16:11:51 +0000 UTC" deadline="2025-11-24 16:09:59.674187978 +0000 UTC"  
I1124 16:01:52.023360    1104 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client" sleep="8m7.650827778s"  
I1124 16:01:52.970199    1104 cert_rotation.go:92] "Certificate rotation detected, shutting down client connections to start using new credentials" logger="tls-transport-cache"  
I1124 16:01:52.970199    1104 streamwatcher.go:123] "Unable to decode an event from the watch stream" err="read tcp 192.168.222.128:49691->192.168.222.31:6443: use of closed network connection"  
I1124 16:01:52.970199    1104 reflector.go:946] "Watch close" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" totalItems=49  
I1124 16:01:52.971824    1104 streamwatcher.go:123] "Unable to decode an event from the watch stream" err="read tcp 192.168.222.128:49691->192.168.222.31:6443: use of closed network connection"  
I1124 16:01:52.971824    1104 reflector.go:946] "Watch close" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Pod" totalItems=71  
I1124 16:02:07.184385    1104 informer.go:314] Successfully synced 'win-webserver/win-webserver-85d5d4469b-gfb7p'  
I1124 16:02:07.189861    1104 informer.go:314] Successfully synced 'win-webserver/win-webserver-85d5d4469b-hmcpf'

Flow:

CSR Created and Approved (lines 2-4): The reflector successfully lists and watches the CertificateSigningRequest, and the cache is populated immediately.
CSR Approved and Issued (lines 5-6):

"Certificate signing request is approved" (16:01:51.005853)
"Certificate signing request is issued" (16:01:51.013189)

The CSR csr-n7l5w was approved and issued successfully within ~7ms.

Certificate Rotation Deadline Set (line 8):

Certificate rotation deadline determined" expiration="2025-11-24 16:11:51" deadline="2025-11-24 16:09:59

The system calculated the next rotation will occur at 16:09:59 (about 1.9 minutes before the 10-minute expiration).
Graceful Connection Handling (lines 10-13): The transport cache detected the new certificate and closed existing connections gracefully. The "use of closed network connection" errors are expected and indicate the system is intentionally closing old connections to use the new certificate.
Successful Resync (lines 14-15): The informers reconnected and successfully synced, processing queued items.

openshift-ci-robot · 2025-11-24T17:16:13Z

@jrvaldes: This pull request references Jira Issue OCPBUGS-64719, which is invalid:

expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This commit ensures the hybridOverlay service receives explicitly the address of the API Server Endpoint along with the cacert to enable certificate rotation.

The --bootstrap-kubeconfig flag should be enough but there is a bug
in OVN-K HybridOverlay [1] where the apiserver and cacert are not extracted
from the bootstrap information, hence introducing the --k8s-apiserver and
--k8s-cacert flags as part of the command.

[1] https://issues.redhat.com/browse/OCPBUGS-65856

hybrid-overlay.log before:
I1124 15:56:01.961205    1104 config.go:2474] Kubernetes config: {BootstrapKubeconfig:C:\k\kubeconfig CertDir:C:\k\cni\config CertDuration:10m0s Kubeconfig: CACert: CAData:[] APIServer:  Token: TokenFile: CompatServiceCIDR: RawServiceCIDRs:172.16.1.0/24 ServiceCIDRs:[172.16.1.0/24] OVNConfigNamespace:ovn-kubernetes OVNEmptyLbEvents:false PodIP: RawNoHostSubnetNodes: NoHostSubnetNodes:<nil> HostNetworkNamespace: DisableRequestedChassis:false PlatformType: HealthzBindAddress: CompatMetricsBindAddress: CompatOVNMetricsBindAddress: CompatMetricsEnablePprof:false DNSServiceNamespace:kube-system DNSServiceName:kube-dns}
hybrid-overlay.log with proposed implementation, see the APIServer and CACert fields populated
I1124 15:56:01.961205    1104 config.go:2474] Kubernetes config: {BootstrapKubeconfig:C:\k\kubeconfig CertDir:C:\k\cni\config CertDuration:10m0s Kubeconfig: CACert:C:\k\bootstrap-ca.crt CAData:[] APIServer:https://api-int.jvaldes.vmc.devcluster.openshift.com:6443 Token: TokenFile: CompatServiceCIDR: RawServiceCIDRs:172.16.1.0/24 ServiceCIDRs:[172.16.1.0/24] OVNConfigNamespace:ovn-kubernetes OVNEmptyLbEvents:false PodIP: RawNoHostSubnetNodes: NoHostSubnetNodes:<nil> HostNetworkNamespace: DisableRequestedChassis:false PlatformType: HealthzBindAddress: CompatMetricsBindAddress: CompatOVNMetricsBindAddress: CompatMetricsEnablePprof:false DNSServiceNamespace:kube-system DNSServiceName:kube-dns}

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

jrvaldes · 2025-11-24T19:00:07Z

/test ?

openshift-ci · 2025-11-24T19:00:11Z

@jrvaldes: The following commands are available to trigger required jobs:

/test aws-e2e-operator

/test azure-e2e-operator

/test azure-e2e-upgrade

/test ci-bundle-wmco-bundle

/test gcp-e2e-operator

/test images

/test lint

/test nutanix-e2e-operator

/test platform-none-vsphere-e2e-operator

/test security

/test unit

/test vsphere-disconnected-e2e-operator

/test vsphere-e2e-operator

/test vsphere-proxy-e2e-operator

/test wicd-unit-vsphere

Use /test all to run all jobs.

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jrvaldes · 2025-11-24T19:00:15Z

/remove-approve

jrvaldes · 2025-11-24T19:00:24Z

/test images

jrvaldes · 2025-11-24T19:00:26Z

/test lint

jrvaldes · 2025-11-24T19:00:31Z

/test unit

jrvaldes · 2025-11-24T19:53:08Z

/test vsphere-e2e-operator

sebsoto

Thanks jose!
Mainly LGTM.

sebsoto · 2025-11-25T20:43:07Z

pkg/nodeconfig/init.go

+// GetAPIServerEndpoint returns the cached Kubernetes API server endpoint
+func GetAPIServerEndpoint() string {
+	return nodeConfigCache.apiServerEndpoint
+}


Exposing this through nodeConfig is weird. Its an internal cache for nodeconfig,

Can we just get the infrastructure object in this function:

windows-machine-config-operator/controllers/configmap_controller.go

Line 716 in 87af75d

func generateServicesManifest(ctx context.Context, client client.Client, port string, platform oconfig.PlatformType) (*servicescm.Data, error) {

And pass the value where we need it?

Addressed in e8bc82b

jrvaldes · 2025-11-26T14:40:16Z

/test vsphere-e2e-operator

This commit ensures the hybridOverlay service receives explicitly the address of the API Server Endpoint along with the cacert to enable certificate rotation. The --bootstrap-kubeconfig flag should be enough but there is a bug in OVN-K HybridOverlay [1] where the apiserver and cacert are not extracted from the bootstrap information, hence introducing the --k8s-apiserver and --k8s-cacert flags as part of the command. [1] https://issues.redhat.com/browse/OCPBUGS-65856

jrvaldes · 2025-11-26T19:45:29Z

/test vsphere-e2e-operator

openshift-ci · 2025-11-26T21:43:55Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sebsoto

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sebsoto]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jrvaldes · 2025-11-26T22:38:54Z

test passed

https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_windows-machine-config-operator/3586/pull-ci-openshift-windows-machine-config-operator-master-vsphere-e2e-operator/1993768150138621952

openshift-ci · 2025-11-27T01:44:33Z

@jrvaldes: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/vsphere-disconnected-e2e-operator	`d89e96e`	link	true	`/test vsphere-disconnected-e2e-operator`
ci/prow/aws-e2e-operator	`d89e96e`	link	true	`/test aws-e2e-operator`
ci/prow/vsphere-proxy-e2e-operator	`d89e96e`	link	true	`/test vsphere-proxy-e2e-operator`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 24, 2025

jrvaldes changed the title ~~[services] Add APIServer to HybridOverlay config~~ OCPBUGS-64719: Add APIServer to HybridOverlay config Nov 24, 2025

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Nov 24, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 24, 2025

jrvaldes force-pushed the 00-hybrid-overlay-add-apiserver branch from b35d84b to f6c28f7 Compare November 24, 2025 18:34

openshift-ci bot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 24, 2025

sebsoto requested changes Nov 25, 2025

View reviewed changes

jrvaldes force-pushed the 00-hybrid-overlay-add-apiserver branch from f6c28f7 to e8bc82b Compare November 26, 2025 14:37

jrvaldes force-pushed the 00-hybrid-overlay-add-apiserver branch from e8bc82b to d89e96e Compare November 26, 2025 19:45

sebsoto approved these changes Nov 26, 2025

View reviewed changes

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 26, 2025

jrvaldes marked this pull request as ready for review November 26, 2025 22:38

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 26, 2025

openshift-ci bot requested a review from mansikulkarni96 November 26, 2025 22:39

openshift-ci bot requested a review from sebsoto November 26, 2025 22:39

OCPBUGS-64719: Add APIServer to HybridOverlay config #3586

Are you sure you want to change the base?

OCPBUGS-64719: Add APIServer to HybridOverlay config #3586

Uh oh!

Conversation

jrvaldes commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Nov 24, 2025

Uh oh!

jrvaldes commented Nov 24, 2025

Uh oh!

openshift-ci-robot commented Nov 24, 2025

Uh oh!

jrvaldes commented Nov 24, 2025

Uh oh!

openshift-ci bot commented Nov 24, 2025

Uh oh!

jrvaldes commented Nov 24, 2025

Uh oh!

jrvaldes commented Nov 24, 2025

Uh oh!

jrvaldes commented Nov 24, 2025

Uh oh!

jrvaldes commented Nov 24, 2025

Uh oh!

jrvaldes commented Nov 24, 2025

Uh oh!

sebsoto left a comment

Choose a reason for hiding this comment

Uh oh!

sebsoto Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

jrvaldes Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

jrvaldes commented Nov 26, 2025

Uh oh!

jrvaldes commented Nov 26, 2025

Uh oh!

openshift-ci bot commented Nov 26, 2025

Uh oh!

jrvaldes commented Nov 26, 2025

Uh oh!

openshift-ci bot commented Nov 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jrvaldes commented Nov 24, 2025 •

edited

Loading