ExecProvider broken for Nebius federated identity

[peterroelants]

We're running a kubernetes cluster on Nebius and I'm using the Nebius CLI tool with a federated identity. However it seems like the ExecProvider of the Python kubernetes client is not working with this setup.

I've setup a kubeconfig file using nebius using:

nebius mk8s cluster get-credentials --id <id-redacted> --external

Which resulted in a kubeconfig that looks like:

❯ kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: <url-redacted>
  name: <name-redacted>
contexts:
- context:
    cluster: <cluster-redacted>
    user: <user-redacted>
  name: <name-redacted>
current-context: <context-redacted>
kind: Config
preferences: {}
users:
- name: <name-redacted>
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - mk8s
      - v1
      - cluster
      - get-token
      - --profile
      - <user-redacted>
      - --format
      - json
      command: /home/peter/.nebius/bin/nebius
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false

Note the exec provider section to get a key from nebius which works on it's own:

❯ nebius mk8s v1 cluster get-token --profile <user-redacted> --format json
{"kind":"ExecCredential","apiVersion":"client.authentication.k8s.io/v1beta1","spec":{"interactive":false},"status":{"expirationTimestamp":"2025-01-29T17:38:10Z","token":"<redacted>"}}

And kubectl works:

❯ kubectl get namespaces
NAME                  STATUS        AGE
...

What happened (please include outputs or screenshots):

However the following fails:

from kubernetes import client, config

config.load_kube_config()
k8s_api = client.CoreV1Api()
k8s_api.api_client.configuration.debug = True
print(k8s_api.list_namespace())

With error ERROR:root:exec: failed to decode process output: Expecting value: line 1 column 1 (char 0) and full output:

ERROR:root:exec: failed to decode process output: Expecting value: line 1 column 1 (char 0)
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): <url-redacted>
DEBUG:urllib3.connectionpool:<url-redacted> "GET /api/v1/namespaces HTTP/1.1" 403 0
DEBUG:kubernetes.client.rest:response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces is forbidden: User \"system:anonymous\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"namespaces"},"code":403}

send: b'GET /api/v1/namespaces HTTP/1.1\r\nHost: <url-redacted>\r\nAccept-Encoding: identity\r\nAccept: application/json\r\nUser-Agent: OpenAPI-Generator/32.0.0/python\r\nContent-Type: application/json\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Audit-Id: <id-redacted>
header: Cache-Control: no-cache, private
header: Content-Type: application/json
header: X-Content-Type-Options: nosniff
header: X-Kubernetes-Pf-Flowschema-Uid: <redacted>
header: X-Kubernetes-Pf-Prioritylevel-Uid: <redacted>
header: Date: Wed, 29 Jan 2025 08:47:02 GMT
header: Content-Length: 271

And this traceback (click to expand).

---------------------------------------------------------------------------
ApiException                              Traceback (most recent call last)
Cell In[2], line 6
      4 k8s_api = client.CoreV1Api()
      5 k8s_api.api_client.configuration.debug = True
----> 6 print(k8s_api.list_namespace())

File ~/miniforge3/envs/anam-audio-to-motion/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py:14962, in CoreV1Api.list_namespace(self, **kwargs)
  14930 """list_namespace  # noqa: E501
  14931 
  14932 list or watch objects of kind Namespace  # noqa: E501
   (...)
  14959          returns the request thread.
  14960 """
  14961 kwargs['_return_http_data_only'] = True
> 14962 return self.list_namespace_with_http_info(**kwargs)

File lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py:15073, in CoreV1Api.list_namespace_with_http_info(self, **kwargs)
  15070 # Authentication setting
  15071 auth_settings = ['BearerToken']  # noqa: E501
> 15073 return self.api_client.call_api(
  15074     '/api/v1/namespaces', 'GET',
  15075     path_params,
  15076     query_params,
  15077     header_params,
  15078     body=body_params,
  15079     post_params=form_params,
  15080     files=local_var_files,
  15081     response_type='V1NamespaceList',  # noqa: E501
  15082     auth_settings=auth_settings,
  15083     async_req=local_var_params.get('async_req'),
  15084     _return_http_data_only=local_var_params.get('_return_http_data_only'),  # noqa: E501
  15085     _preload_content=local_var_params.get('_preload_content', True),
  15086     _request_timeout=local_var_params.get('_request_timeout'),
  15087     collection_formats=collection_formats)

File lib/python3.11/site-packages/kubernetes/client/api_client.py:348, in ApiClient.call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, async_req, _return_http_data_only, collection_formats, _preload_content, _request_timeout, _host)
    311 """Makes the HTTP request (synchronous) and returns deserialized data.
    312 
    313 To make an async_req request, set the async_req parameter.
   (...)
    345     then the method will return the response directly.
    346 """
    347 if not async_req:
--> 348     return self.__call_api(resource_path, method,
    349                            path_params, query_params, header_params,
    350                            body, post_params, files,
    351                            response_type, auth_settings,
    352                            _return_http_data_only, collection_formats,
    353                            _preload_content, _request_timeout, _host)
    355 return self.pool.apply_async(self.__call_api, (resource_path,
    356                                                method, path_params,
    357                                                query_params,
   (...)
    365                                                _request_timeout,
    366                                                _host))

File lib/python3.11/site-packages/kubernetes/client/api_client.py:180, in ApiClient.__call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, _return_http_data_only, collection_formats, _preload_content, _request_timeout, _host)
    177     url = _host + resource_path
    179 # perform request and return response
--> 180 response_data = self.request(
    181     method, url, query_params=query_params, headers=header_params,
    182     post_params=post_params, body=body,
    183     _preload_content=_preload_content,
    184     _request_timeout=_request_timeout)
    186 self.last_response = response_data
    188 return_data = response_data

File lib/python3.11/site-packages/kubernetes/client/api_client.py:373, in ApiClient.request(self, method, url, query_params, headers, post_params, body, _preload_content, _request_timeout)
    371 """Makes the HTTP request using RESTClient."""
    372 if method == "GET":
--> 373     return self.rest_client.GET(url,
    374                                 query_params=query_params,
    375                                 _preload_content=_preload_content,
    376                                 _request_timeout=_request_timeout,
    377                                 headers=headers)
    378 elif method == "HEAD":
    379     return self.rest_client.HEAD(url,
    380                                  query_params=query_params,
    381                                  _preload_content=_preload_content,
    382                                  _request_timeout=_request_timeout,
    383                                  headers=headers)

File lib/python3.11/site-packages/kubernetes/client/rest.py:244, in RESTClientObject.GET(self, url, headers, query_params, _preload_content, _request_timeout)
    242 def GET(self, url, headers=None, query_params=None, _preload_content=True,
    243         _request_timeout=None):
--> 244     return self.request("GET", url,
    245                         headers=headers,
    246                         _preload_content=_preload_content,
    247                         _request_timeout=_request_timeout,
    248                         query_params=query_params)

File lib/python3.11/site-packages/kubernetes/client/rest.py:238, in RESTClientObject.request(self, method, url, query_params, headers, body, post_params, _preload_content, _request_timeout)
    235     logger.debug("response body: %s", r.data)
    237 if not 200 <= r.status <= 299:
--> 238     raise ApiException(http_resp=r)
    240 return r

ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '<redacted>', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': ''<redacted>', 'X-Kubernetes-Pf-Prioritylevel-Uid': ''<redacted>', 'Date': 'Wed, 29 Jan 2025 08:47:02 GMT', 'Content-Length': '271'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces is forbidden: User \"system:anonymous\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"namespaces"},"code":403}

Environment:

❯ kubectl version
Client Version: v1.32.1
Kustomize Version: v5.5.0
Server Version: v1.30.7
WARNING: version difference between client (1.32) and server (1.30) exceeds the supported minor version skew of +/-1

❯ python --version
Python 3.11.11

❯ pip list | grep kubernetes
kubernetes                  32.0.0

❯ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.1 LTS
Release:        24.04
Codename:       noble

[peterroelants]

It works if I remove the shell=True from the subprocess.Popen call in ExecProvider

[peterroelants]

It works if I remove the shell=True from the subprocess.Popen call in ExecProvider

And I just noticed there is an open PR from a few hours ago that would fix this: #2338

[Velyks]

This also affects using the awscli and the oidc-login plugin.

[roycaihw]

#2338 has been merged. I think this can be related to #2334

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten