minio web console auth issues when port-forwarding a local minikube k8s minio instance

[jinyius]

Steps to Reproduce (for bugs)

0. install minikube and minio inside of minikube
1. kubectl port-forward pod/myminio-blah-blah 9001
2. use browser to hit localhost:9001
3. try to authenticate and use the web console

Expected Behavior

web console loads fine and is usable

Current Behavior

web console loads, but acts oddly due to port-forwarding issues

• when the initial auth screen loads, the cli w/ the port-forward shows these errors:

Forwarding from 127.0.0.1:9001 -> 9001                                                                                                                                                                                                                                                                                         
Forwarding from [::1]:9001 -> 9001                                                                                                                                                                                                                                                                                               
Handling connection for 9001                                                                                                                                                                                                                                                                                                     
# ...                                                                                                                                                                                                                                                                                     
Handling connection for 9001                                                                                                                                                                                                                                                                                                     
E1229 14:24:40.046628   69982 portforward.go:379] error copying from remote stream to local connection: readfrom tcp6 [::1]:9001->[::1]:58450: write tcp6 [::1]:9001->[::1]:58450: write: broken pipe                                                                                                                            
Handling connection for 9001 

• later on, there are Timed out errors:

Connection to localhost port 9001 [tcp/etlservicemgr] succeeded!
Handling connection for 9001                                                                                                                                                                                                                                                                                                     
Connection to localhost port 9001 [tcp/etlservicemgr] succeeded!
E1229 14:23:34.674227   69851 portforward.go:345] error creating error stream for port 9001 -> 9001: Timeout occurred                                           
Handling connection for 9001                                                                                                                                                                                                                                                                                                     
E1229 14:23:34.710293   69851 portforward.go:345] error creating error stream for port 9001 -> 9001: Timeout occurred                                           
Handling connection for 9001                                                    
E1229 14:23:36.936280   69851 portforward.go:345] error creating error stream for port 9001 -> 9001: Timeout occurred                                                                                                                                                                                                            
Handling connection for 9001                                                                                                                                                                                                                                                                                                     
Connection to localhost port 9001 [tcp/etlservicemgr] succeeded!
E1229 14:23:38.616971   69851 portforward.go:345] error creating error stream for port 9001 -> 9001: Timeout occurred                                           
Handling connection for 9001 
Handling connection for 9001                                                                                                                                                                                                                                                                                                     
Handling connection for 9001                                                                                                                                                                                                                                                                                                     
E1229 14:23:41.950806   69851 portforward.go:345] error creating error stream for port 9001 -> 9001: Timeout occurred                                           
Handling connection for 9001                                                    
Connection to localhost port 9001 [tcp/etlservicemgr] succeeded!
E1229 14:23:46.966488   69851 portforward.go:345] error creating error stream for port 9001 -> 9001: Timeout occurred                                           
Handling connection for 900

• the behavior in the web ui is odd. authentication never succeeds.

Possible Solution

my coworker and i believe there's something going on w/ the minio web ui's around the login screen and authentication that causes the port-forwarding to break. we've gotten around this by brute-forcing the port forward to restart explicitly and manually during authentication. eventually, with enough port-forward restarts, it succeeds. once the browser session is authenticated, there are no port-forwarding issues.

workaround

1. while true; do kubectl port-forward pod/myminio-blah-blah 9001; done
2. open browser to localhost:9001
3. whenever there are timeouts or broken pipes in the cli w/ the port-forward, CTRL-C to restart the port-forward
4. interact w/ the web ui to login (revisiting 4) until fully authenticated and loaded
5. once loaded, port-forwarding is not brittle

Context

we use minio locally as a fake s3 for local flink development. we use eks in prod for flink, so we use minikube for a local k8s env and so we try to keep all flink development parts in the local minikube setup

Regression

yes. port-forwarding worked fine using the super-old helm chart: https://github.com/minio/charts

we noticed that this version doesn't have a split console and api port. it just has port 9000 that suffices for both endpoints. we had to version bump b/c this old version of minio has bugs that prevents proper s3 expectations.

Your Environment

• Version used (minio --version):

bash-4.4$ minio --version
minio version RELEASE.2022-12-12T19-27-27Z (commit-id=a469e6768df4d5d2cb340749fa58e4721a7dee96)
Runtime: go1.19.4 linux/amd64
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Copyright: 2015-2022 MinIO, Inc.

• Server setup and configuration:
  we tried using replicated and standalone setups for our minio install

• Operating System and version (uname -a):

Darwin myhost.mylan.whatever 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:08:47 PST 2022; root:xnu-8792.61.2~4/RELEASE_X86_64 x86_64

NOTE: we replicated this using both an m1 and intel mac. the m1 mac uses minikube over docker and the intel mac uses minikube over hyperkit.

• helm values overrides:

persistence:
  size: 10Gi

resources:
  requests:
    memory: 256Mi

buckets:
  - name: promoted-event-logs
    versioning: false
    purge: true

mode: standalone

replicas: 1

environment:
  MINIO_API_SELECT_PARQUET: "on"

[jinyius]

more context:

port-forwarding various kafka, redis and flink ports from the same minikube all work fine w/o issue in our setup. it's just this latest minio console. moreover, forwarding minio's api port (9000) and using mc locally in the host context against the api port *9000) works fine w/o any forward breaks.

[harshavardhana]

Forwarding won't work as MinIO Console UI requires websockets.

Moving this to relevant repo for more guidance..

[jinyius]

interesting. did the older version not use websockets?

[harshavardhana]

interesting. did the older version not use websockets?

yes, there have been some changes to the bucket browser that started using WebSockets for more efficient (responsive) UI.

So I don't know if port-forwarding handles these scenarios properly, as it's not meant to be used for production.

[jinyius]

it seems there's only issues when loading the login page based on more testing. for my setup, i've refined the workaround to just need one restart:

1. do the while true loop for the port forward in a cli session
2. just load up the login window (see the errors in the cli session)
3. back in the cli session, ctrl-c the port-forward to restart it
4. back in the browser, actually perform the login, ideally with one click (minimize interactions)

[jinyius]

So I don't know if port-forwarding handles these scenarios properly, as it's not meant to be used for production.

i agree that port-forwarding isn't for production use. the use case minio satisfies for us right now is stubbing s3 for local development which hopefully is a valid minio use case.

[clovis-dugue]

I also have websocket-related issues, on a console deployed via Minio Operator and Tenant

I see the following error in my pod: error upgrade connection:websocket: response does not implement http.Hijacker

While I understand port-forwarding and websockets together may not be meant for production use, I have done nothing but follow the documentation, which is essentially "pop up an operator, use the krew plugin to proxy your traffic, create a tenant and enjoy". The documentation should probably be updated to reflect the fact that issues should be expected when using port-forwarding

[ravindk89]

@clovis-dugue that's a fair ask. I missed this discussion as I typically monitor minio/minio and minio/operator, but forgot minio/console.

@harshavardhana am I right in thinking that moving forward, we'll need to use ingress to get this to work? that is, kubectl minio proxy likely wont work either as Operator Console is Console under the hood?

[ravindk89]

I'll note that this might be an issue related mostly to containerized K8s setups - asking internally, we haven't seen it come up yet in fully-deployed upstream Kubernetes. If that's the case, we might need to add a warning in about this behavior and the workaround. Documenting nginx/traefik ingress will take a little longer.

[pasztorl]

I've tested it with kubeadm/kubernetes on aws using aws nlb->haproxy-ingress->minio-operator->minio-console and websocket based communication not worked and got the "error upgrade connection.." message in the console log