CKS: Service creation of type "LoadBalancer" fails when cluster name contains uppercase

[zap51]

ISSUE TYPE

• Bug Report

COMPONENT NAME

CKS

CLOUDSTACK VERSION

4.16.1.0 onwards

CONFIGURATION

Advanced Networking

OS / ENVIRONMENT

Ubuntu 20.04 and later

SUMMARY

A Kubernetes Cluster managed by CKS fails to acquire external IP for the service of type "LoadBalancer" when the cluster / node names have upper case letters.

STEPS TO REPRODUCE

1. Create a cluster with some name which contains upper case characters like for example "Kubernetescl34t"
2. Nodes come online with lower case letters
# kubectl get no
NAME                                  STATUS   ROLES           AGE   VERSION
kubernetescl34t-control-18dee810e29   Ready    control-plane   88s   v1.28.4
kubernetescl34t-control-18dee813b32   Ready    control-plane   75s   v1.28.4
kubernetescl34t-control-18dee81618f   Ready    control-plane   74s   v1.28.4
kubernetescl34t-node-18dee818726      Ready    <none>          76s   v1.28.4
3. Run a sample pod and create a service of type "LoadBalancer"
# kubectl run --image=nginx nginx
pod/nginx created
root@Kubernetescl34t-control-18dee810e29:~# kubectl expose pod/nginx --type=LoadBalancer --port=80
service/nginx exposed
4. The external IP doesn't get acquired and the service-controller reports the below
# kubectl describe svc/nginx
Name:                     nginx
Namespace:                default
Labels:                   run=nginx
Annotations:              <none>
Selector:                 run=nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.80.24
IPs:                      10.96.80.24
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31692/TCP
Endpoints:                10.44.0.4:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                  Age               From                Message
  ----     ------                  ----              ----                -------
  Normal   EnsuringLoadBalancer    6s (x4 over 43s)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  6s (x4 over 42s)  service-controller  Error syncing load balancer: failed to ensure load balancer: could not find network

Mailing list reference: https://lists.apache.org/thread/bm1g6lmx6pv2tg8v331rtjn0ytxm4xpj

EXPECTED RESULTS

The external IP should be acquired. CKS should enforce only lowercase names for the cluster and the nodes.

ACTUAL RESULTS

Events:
  Type     Reason                  Age               From                Message
  ----     ------                  ----              ----                -------
  Normal   EnsuringLoadBalancer    6s (x4 over 43s)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  6s (x4 over 42s)  service-controller  Error syncing load balancer: failed to ensure load balancer: could not find network

[weizhouapache]

@zap51
can you share the logs of pod cloud-controller-manager-xxx ?

kubectl get pods -n kube-system |grep cloud-controller-manager
kubectl logs cloud-controller-manager-xxxxx-yyyy -n kube-system

and related cloudstack logs if possible ?

[kiranchavala]

@weizhouapache

Able to reproduce the issue

Please find the logs

I0228 08:03:49.899226       1 serving.go:313] Generated self-signed cert in-memory
W0228 08:04:01.913700       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0228 08:04:02.108445       1 controllermanager.go:120] Version: v0.0.0-master+$Format:%h$
I0228 08:04:02.200283       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0228 08:04:02.201011       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0228 08:04:02.202335       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0228 08:04:02.202980       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0228 08:04:02.206901       1 secure_serving.go:178] Serving securely on [::]:10258
I0228 08:04:02.207265       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/cloud-controller-manager...
I0228 08:04:02.295040       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0228 08:04:02.302868       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0228 08:04:02.303632       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
E0228 08:04:02.803733       1 event.go:316] Could not construct reference to: '&v1.Lease{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cloud-controller-manager", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"d3f7ca89-c323-4836-b011-9f844c819b2b", ResourceVersion:"711", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63844704242, loc:(*time.Location)(0x30c2040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"cloudstack-ccm", Operation:"Update", APIVersion:"coordination.k8s.io/v1", Time:(*v1.Time)(0xc0002f62e0), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0002f6300)}}}, Spec:v1.LeaseSpec{HolderIdentity:(*string)(nil), LeaseDurationSeconds:(*int32)(nil), AcquireTime:(*v1.MicroTime)(nil), RenewTime:(*v1.MicroTime)(nil), LeaseTransitions:(*int32)(nil)}}' due to: 'no kind is registered for the type v1.Lease in scheme "pkg/runtime/scheme.go:101"'. Will not report event: 'Normal' 'LeaderElection' 'cloud-controller-manager-574bcb86c-r8s2x_56ae4280-6570-40a3-8fa9-9809a65713ab became leader'
I0228 08:04:02.803902       1 leaderelection.go:252] successfully acquired lease kube-system/cloud-controller-manager
I0228 08:04:02.809440       1 event.go:278] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"cloud-controller-manager", UID:"82955a91-1dda-4ab4-bb57-08f2907bf454", APIVersion:"v1", ResourceVersion:"710", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cloud-controller-manager-574bcb86c-r8s2x_56ae4280-6570-40a3-8fa9-9809a65713ab became leader
I0228 08:04:02.897011       1 controllermanager.go:247] Started "service"
I0228 08:04:02.897052       1 core.go:101] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
W0228 08:04:02.897063       1 controllermanager.go:244] Skipping "route"
I0228 08:04:02.995838       1 controller.go:208] Starting service controller
I0228 08:04:02.995889       1 shared_informer.go:223] Waiting for caches to sync for service
I0228 08:04:02.996018       1 node_controller.go:110] Sending events to api server.
I0228 08:04:02.996085       1 controllermanager.go:247] Started "cloud-node"
I0228 08:04:02.997904       1 node_lifecycle_controller.go:78] Sending events to api server
I0228 08:04:02.997970       1 controllermanager.go:247] Started "cloud-node-lifecycle"
I0228 08:04:03.395633       1 shared_informer.go:230] Caches are synced for service
I0228 08:05:23.902172       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"hello-node", UID:"b67403a9-786a-4952-8a93-d37ed4f599c2", APIVersion:"v1", ResourceVersion:"930", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0228 08:05:24.111288       1 controller.go:244] error processing service default/hello-node (will retry): failed to ensure load balancer: could not find network
I0228 08:05:24.112015       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"hello-node", UID:"b67403a9-786a-4952-8a93-d37ed4f599c2", APIVersion:"v1", ResourceVersion:"930", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: could not find network
I0228 08:05:29.112042       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"hello-node", UID:"b67403a9-786a-4952-8a93-d37ed4f599c2", APIVersion:"v1", ResourceVersion:"934", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0228 08:05:29.177449       1 controller.go:244] error processing service default/hello-node (will retry): failed to ensure load balancer: could not find network
I0228 08:05:29.178621       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"hello-node", UID:"b67403a9-786a-4952-8a93-d37ed4f599c2", APIVersion:"v1", ResourceVersion:"934", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: could not find network
I0228 08:05:39.178564       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"hello-node", UID:"b67403a9-786a-4952-8a93-d37ed4f599c2", APIVersion:"v1", ResourceVersion:"934", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0228 08:05:39.242294       1 controller.go:244] error processing service default/hello-node (will retry): failed to ensure load balancer: could not find network
I0228 08:05:39.242639       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"hello-node", UID:"b67403a9-786a-4952-8a93-d37ed4f599c2", APIVersion:"v1", ResourceVersion:"934", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: could not find network
I0228 08:05:59.243350       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"hello-node", UID:"b67403a9-786a-4952-8a93-d37ed4f599c2", APIVersion:"v1", ResourceVersion:"934", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0228 08:05:59.305626       1 controller.go:244] error processing service default/hello-node (will retry): failed to ensure load balancer: could not find network
I0228 08:05:59.306713       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"hello-node", UID:"b67403a9-786a-4952-8a93-d37ed4f599c2", APIVersion:"v1", ResourceVersion:"934", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: could not find network
I0228 08:06:39.308317       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"hello-node", UID:"b67403a9-786a-4952-8a93-d37ed4f599c2", APIVersion:"v1", ResourceVersion:"934", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0228 08:06:39.394721       1 controller.go:244] error processing service default/hello-node (will retry): failed to ensure load balancer: could not find network
I0228 08:06:39.395185       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"hello-node", UID:"b67403a9-786a-4952-8a93-d37ed4f599c2", APIVersion:"v1", ResourceVersion:"934", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: could not find network

[kiranchavala]

Cloudstack management logs

2024-02-28 08:10:26,157 DEBUG [c.c.a.ApiServlet] (qtp1278254413-13:ctx-5b86be17) (logid:497d02b5) ===START===  10.0.57.163 -- GET  apiKey=NizA0n2i5qSkepq72B4hyj_tVRZDXSRNbSyg4-N6qzFQxFdj8w7PLxs456so47b7Ya4umNvON8nWaa_Ud_pYTw&command=listNetworks&id=&response=json&signature=gVtQmPIE%2BcpdE8CB8sbSWQa8TAQ%3D
2024-02-28 08:10:26,160 DEBUG [c.c.a.ApiServer] (qtp1278254413-13:ctx-5b86be17 ctx-2c32ab42) (logid:497d02b5) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"23728330-cfde-11ee-b10f-1e00e3000295"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-02-28 08:10:26,162 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278254413-13:ctx-5b86be17 ctx-2c32ab42) (logid:497d02b5) Account [Account [{"accountName":"admin","id":2,"uuid":"23728330-cfde-11ee-b10f-1e00e3000295"}]] is Root Admin or Domain Admin, all APIs are allowed.
2024-02-28 08:10:26,163 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278254413-13:ctx-5b86be17 ctx-2c32ab42) (logid:497d02b5) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [listNetworks] for user [User {"username":"admin-kubeadmin","uuid":"acfe855d-c107-4f1f-ac6a-1a2263ff96fe"}.] as allowed.
2024-02-28 08:10:26,163 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1278254413-13:ctx-5b86be17 ctx-2c32ab42) (logid:497d02b5) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-02-28 08:10:26,164 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1278254413-13:ctx-5b86be17 ctx-2c32ab42) (logid:497d02b5) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-02-28 08:10:26,165 DEBUG [c.c.a.d.ParamProcessWorker] (qtp1278254413-13:ctx-5b86be17 ctx-2c32ab42 ctx-a5dd14fa) (logid:497d02b5) Object entity uuid =  does not exist in the database.
2024-02-28 08:10:26,165 INFO  [c.c.a.ApiServer] (qtp1278254413-13:ctx-5b86be17 ctx-2c32ab42 ctx-a5dd14fa) (logid:497d02b5) Unable to execute API command listnetworks due to invalid value. Invalid parameter id value= due to incorrect long value format, or entity does not exist or due to incorrect parameter annotation for the field in api cmd class.
2024-02-28 08:10:26,166 DEBUG [c.c.a.ApiServlet] (qtp1278254413-13:ctx-5b86be17 ctx-2c32ab42 ctx-a5dd14fa) (logid:497d02b5) ===END===  10.0.57.163 -- GET  apiKey=NizA0n2i5qSkepq72B4hyj_tVRZDXSRNbSyg4-N6qzFQxFdj8w7PLxs456so47b7Ya4umNvON8nWaa_Ud_pYTw&command=listNetworks&id=&response=json&signature=gVtQmPIE%2BcpdE8CB8sbSWQa8TAQ%3

[weizhouapache]

could not find network

thanks @kiranchavala
I have seen this issue recently, I thought it was due to some of my configurations.
it looks like a valid bug

[kiranchavala]

@weizhouapache i think this could be related to the PR on cloudstack kubernetes provider

apache/cloudstack-kubernetes-provider#41

[weizhouapache]

@weizhouapache i think this could be related to the PR on cloudstack kubernetes provider

apache/cloudstack-kubernetes-provider#41

yeah @kiranchavala
it has been fixed, but not released yet ...