Incorrect `$PATH` affecting many executables (including python3) #664

samirotiv · 2022-02-06T08:39:28Z

Bug description

Both python and python3 need to come from pyenv. Currently python comes from pyenv but python3 comes from /bin - the in-built python.

Steps to reproduce

gitpod /workspace/microappsv2 (development) $ docker run --rm -it gitpod/workspace-full bash
gitpod ~ $ which python
/home/gitpod/.pyenv/shims/python
gitpod ~ $ which python3
/bin/python3

Expected behavior

gitpod /workspace/microappsv2 (development) $ docker run --rm -it gitpod/workspace-full bash
gitpod ~ $ which python
/home/gitpod/.pyenv/shims/python
gitpod ~ $ which python3
/home/gitpod/.pyenv/shims/python3

Example repository

Just run the code sample above, anywhere.

Anything else?

$PATH seems to be getting overridden in some build step.

The text was updated successfully, but these errors were encountered:

kylos101 · 2022-02-16T01:54:06Z

As an example, users are working around path and other issues like so -https://github.com/couchbase-examples/python-quickstart/pull/4/files

huweiATgithub · 2022-02-18T16:30:42Z

The reason why python works while python3 doesn't work is because python is not in system path.
I also notice this problem. And find that this is due to the PATH variable having been appended somewhere after the pyenv PATH has been set.
The pyenv related PATH variable has been set in

workspace-images/chunks/lang-python/Dockerfile

Line 10 in ffc68a3

ENV PATH=$HOME/.pyenv/bin:$HOME/.pyenv/shims:$PATH

But then, after entering the workspace, take the workspace-full-vnc as an example, the PATH becomes,

echo -n $PATH | awk -v RS=: '!($0 in a) {a[$0]; printf("%s%s", length(a) > 1 ? ":" : "", $0)}' | tr : "\n"
/home/gitpod/.pyenv/plugins/pyenv-virtualenv/shims
/home/gitpod/.nix-profile/bin
/ide/bin/remote-cli
/home/gitpod/.sdkman/candidates/maven/current/bin
/home/gitpod/.sdkman/candidates/java/current/bin
/home/gitpod/.sdkman/candidates/gradle/current/bin
/home/gitpod/.rvm/gems/ruby-2.7.5/bin
/home/gitpod/.rvm/gems/ruby-2.7.5@global/bin
/home/gitpod/.rvm/rubies/ruby-2.7.5/bin
/workspace/.cargo/bin
/home/gitpod/go-packages/bin
/workspace/.rvm/bin
/usr/sbin
/bin
/workspace/.pip-modules/bin
/home/gitpod/go/bin
/usr/local/sbin
/usr/local/bin
/usr/bin
/sbin
/home/linuxbrew/.linuxbrew/sbin/
/home/gitpod/.cargo/bin
/home/linuxbrew/.linuxbrew/bin
/home/gitpod/.pyenv/bin
/workspace/go/bin
/home/gitpod/.pyenv/shims
/home/gitpod/.nvm/versions/node/v16.13.2/bin

This is after deduplication, there are actually many duplicates. Anyway, as seen from the result, the following paths now precedence the /home/gitpod/.pyenv/shims path, which is how pyenv works.

/usr/sbin
/usr/local/sbin
/usr/local/bin
/usr/bin
/sbin

I would suggest not setting it in the Dockerfile, instead, we can set it in the /home/gitpod/.bashrc.d/60-python script (which I tested, works for python). Besides, this can be easily done calling the pyenv generated script eval "$(pyenv init --path)", which execute PATH="/home/gitpod/.pyenv/shims:${PATH}”.

I think this will not break other tools in the workspace. As the added path only hook commands are related to python, which is why pyenv is used in the first place.

If the gitpod team think this is ok, I can submit a PR.

huweiATgithub · 2022-02-18T17:06:12Z

besides, I think this is due to how dazzle merges the PATH variables. It seems that dazzle neither preserves the order in the PATH nor remove duplicates in the PATH. [there are duplicates even in the workspace-base image]

utam0k · 2022-02-20T23:51:14Z

@huweiATgithub @samirotiv (cc @Furisto @kylos101 )
I'm trying to fix this issue. Please wait a little longer 🙏
gitpod-io/dazzle#43

huweiATgithub · 2022-02-28T23:29:26Z

I see the related PR has been merged but in a fresh workspace-full image, the order in PATH is still not correct,

gitpod ~ $ echo $PATH | tr : '\n'
/home/gitpod/.sdkman/candidates/maven/current/bin
/home/gitpod/.sdkman/candidates/java/current/bin
/home/gitpod/.sdkman/candidates/gradle/current/bin
/home/gitpod/.rvm/gems/ruby-2.7.5/bin
/home/gitpod/.rvm/gems/ruby-2.7.5@global/bin
/home/gitpod/.rvm/rubies/ruby-2.7.5/bin
/home/gitpod/.pyenv/plugins/pyenv-virtualenv/shims
/home/gitpod/.nix-profile/bin
/workspace/go/bin
/workspace/.pip-modules/bin
/sbin
/usr/local/sbin
/home/gitpod/go-packages/bin
/workspace/.rvm/bin
/home/gitpod/.cargo/bin
/home/linuxbrew/.linuxbrew/bin
/home/linuxbrew/.linuxbrew/sbin/
/home/gitpod/go/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/home/gitpod/.pyenv/shims
/bin
/workspace/.cargo/bin
/usr/bin
/usr/local/bin
/home/gitpod/.nvm/versions/node/v16.13.2/bin
/usr/sbin
/home/gitpod/.pyenv/bin
/home/gitpod/.nvm/versions/node/v16.13.2/bin
/home/gitpod/.rvm/bin

utam0k · 2022-02-28T23:57:19Z

@huweiATgithub Thanks for your checking. But we haven't released a new image which is included fixing this problem yet. The last update was 7 days ago in dockerhub and a PR to fix this problem was merged 3 days ago. Please wait until the next release.
https://hub.docker.com/r/gitpod/workspace-full/tags
#727

If you want to fix this issue in a hurry, you can commit like this.
utam0k/kedro@1c2e11d

jab · 2022-03-01T17:00:07Z

Should this issue title be changed to something like "incorrect PATH affecting many executables (including python3)", since it affects e.g. "pip" and many other executables as well?
Even with the eval "$(pyenv init --path)" workaround in my init task that @utam0k suggested in the previous comment, I'm still seeing very buggy behavior. Once my init task completes and the GitPod environment is ready, it's as if nothing in my init task had ever even run, even though the console output shows that everything ran successfully.

Specifically, you can see in https://github.com/jab/bidict/blob/09c1705973dd75c95e7771cdca5b4f3ebac3c4b5/.gitpod.yml#L20-L27

tasks:
  - init: |
      eval "$(pyenv init --path)"  # https://github.com/gitpod-io/workspace-images/issues/664#issuecomment-1054827300
      pyenv install 3.10.2
      pyenv global 3.10.2
      pip install -U pip
      pip install -r requirements/dev.txt

And the console output includes, for example, the following:

Requirement already satisfied: setuptools>=20.0 in /home/gitpod/.pyenv/versions/3.10.2/lib/python3.10/site-packages (from astroid==2.9.3->-r requirements/dev.txt (line 9)) (58.1.0)

Note that "setuptools" was found in the path /home/gitpod/.pyenv/versions/3.10.2. This shows that pyenv successfully installed python 3.10.2, the "python" command now resolves to that version (rather than the system version 3.8), and "pip" resolves to the pip executable installed for that version of python, rather than /usr/bin/pip corresponding to the system python.

But if you load https://gitpod.io/#https://github.com/jab/bidict and then try to actually use the environment, it's as if none of the stuff in the init task ever happened:

gitpod /workspace/bidict (main) $ pyenv versions
  system
* 3.8.12 (set by /home/gitpod/.pyenv/version)

gitpod /workspace/bidict (main) $ ls /home/gitpod/.pyenv/versions
3.8.12

gitpod /workspace/bidict (main) $ which pip
/usr/bin/pip

gitpod /workspace/bidict (main) $ pip -V
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)

At this point I've burned many hours trying to figure out what's going on with GitPod here, so any help from the community would be greatly appreciated!

huweiATgithub · 2022-03-01T17:19:16Z

@jab That calling eval "$(pyenv init --path)" will append pyenv's path into PATH and only affect the current shell. For a workaround, you may also want to echo that into the /home/gitpod/.bashrc.d/60-python so that it is being invoked for every shell.

jab · 2022-03-01T19:09:59Z

Thanks, @huweiATgithub, I appreciate your trying to help. I tried your suggestion though and unfortunately it doesn't help.

If you open https://gitpod.io/#https://github.com/jab/bidict and run cat /home/gitpod/.bashrc.d/60-python, you can see for yourself:

gitpod /workspace/bidict (main) $  HISTFILE=/workspace/.gitpod/cmd-0 history -r; {
> echo 'eval "$(pyenv init --path)"' >> ~/.bashrc.d/60-python  # https://github.com/gitpod-io/workspace-images/issues/664
> bash -l
> pyenv install 3.10.2
> pyenv global 3.10.2
> pip -V | fgrep 'python 3.10'  # 'pip' should resolve to ~/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pip
> pip install -U pip
> pip install -r requirements/dev.txt
> pre-commit install --install-hooks
> 
> }

gitpod /workspace/bidict (main) $ cat ~/.bashrc.d/60-python  # implements your suggestion
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
eval "$(pyenv init --path)"

gitpod /workspace/bidict (main) $ pyenv versions  # `pyenv install 3.10.2` never happened?
  system
* 3.8.12 (set by /home/gitpod/.pyenv/version)

Any other suggestions?

At this point it really feels like GitPod is gaslighting me. 🥴

Really hope a GitPod community or team member can help!

jab · 2022-03-01T19:57:18Z

Wow, I think compressing my init task into a single line actually did the trick!

- init: echo 'eval "$(pyenv init --path)"' >> ~/.bashrc.d/60-python `# https://github.com/gitpod-io/workspace-images/issues/664` && eval $(pyenv init --path) && pyenv install 3.10.2 && pyenv global 3.10.2 && pip -V | fgrep 'python 3.10' && pip install -U pip && pip install -r requirements/dev.txt && pre-commit install --install-hooks

Is this the best way to work around this for now?
Should the https://www.gitpod.io/docs/languages/python docs be updated somewhere to mention that a workaround is needed, in case other users are hitting this and struggling too?

huweiATgithub · 2022-03-01T22:35:31Z

@jab I think the single line is not the key.
echo eval "$(pyenv init --path)" into the 60-python file will ensure each time a new bash is started, the pyenv's path variable is prepending to the top.
Your first try doesn't work because the init bash did not execute the command (as seen from your bash history, but bashes started later should work fine).
You one-line try both execute the command and write to the 60-python file so that it works.

utam0k · 2022-03-01T23:11:45Z

@jab LGTM. I've changed the title of this issue.

Should this issue title be changed to something like "incorrect PATH affecting many executables (including python3)", since it affects e.g. "pip" and many other executables as well?

utam0k · 2022-03-01T23:18:00Z

Maybe yes. Anyway, you need to override the $PATH so that pyenv takes precedence.

Is this the best way to work around this for now?

But as for the code base, this problem has been solved. gitpod-io/dazzle#43 However, it has not been released yet. In other words, It has been fixed, but not yet delivered to the user.
We plan to make improvements that will allow users to use the new, modified images more quickly. Please wait a little longer.
cc: @princerachit @kylos101

utam0k · 2022-03-01T23:19:35Z

Thanks for your suggestion. I think we don't need to do it because this problem will soon be fixed.

Should the https://www.gitpod.io/docs/languages/python docs be updated somewhere to mention that a workaround is needed, in case other users are hitting this and struggling too?

utam0k · 2022-03-01T23:27:16Z

@jab I think the single line is not the key.
echo eval "$(pyenv init --path)" into the 60-python file will ensure each time a new bash is started, the pyenv's path variable is prepending to the top.
Your first try doesn't work because the init bash did not execute the command (as seen from your bash history, but bashes started later should work fine).
You one-line try both execute the command and write to the 60-python file so that it works.

@huweiATgithub
Thanks for your help. It's very helpful. I'm sure this is what you're suggesting, right? I think this is probably correct as well. My previous suggestion created a lot of confusion. Sorry about that.

tasks:
  - init: |
      echo 'eval "$(pyenv init --path)"' >> ~/.bashrc.d/60-python
      eval "$(pyenv init --path)"  # https://github.com/gitpod-io/workspace-images/issues/664#issuecomment-1054827300
      pyenv install 3.10.2
      pyenv global 3.10.2
      pip install -U pip
      pip install -r requirements/dev.txt

jab · 2022-03-02T14:49:04Z

Thank you @utam0k and @huweiATgithub, @utam0k's suggestion in the previous comment works! Notice the only difference between that and what I already tried in #664 (comment) is, after appending the eval line to ~/.bashrc.d/60-python, I ran bash -l, rather than running just the eval line in the current shell session. My thinking was that this should have the same effect, and at the same time, test that the modified ~/.bashrc.d/60-python is working correctly for new shell sessions. Not sure why that didn't work.

The main problem I'm having now though is, even though we got my init task to change the $PATH so that "python", "python3", "pip", etc. now all resolve to Python 3.10 as intended, VS Code is still choosing the system Python 3.8 by default every time you start it up: https://gitpod.io/#https://github.com/jab/bidict

And even though the https://www.gitpod.io/docs/languages/python#python-versions docs explain that you can use pyenv to install a different Python version, they don't say anything about how to get VS Code to actually use it.

Worse, if a user who wants to contribute to my project clicks my "edit in GitPod" link and then somehow actually notices they need to change VS Code to a different Python version, when they try to, the intended version Python 3.10.2 is not a directly available option (neither via ~/.pyenv/versions/3.10.2 nor via ~/.pyenv/shims/python):

So they have to somehow know to manually type in ~/.pyenv/shims/python to get the right version of Python:

Any ideas?

utam0k · 2022-03-03T00:05:42Z

@jab I came up with another solution for PATH. Can you use this image until a new one is released? This one is before the new dazzle and does not contain the PATH issue.
https://hub.docker.com/layers/gitpod/workspace-full/legacy-dazzle-v1/images/sha256-46859378f3067e50e8d9509e270f67a0447bb28f5d2a2f0c723dc916411a46c8?context=explore
https://github.com/utam0k/bidict/blob/1ab76f43adadb6ef861742fec3559deea38db97c/.gitpod.yml#L1

utam0k · 2022-03-03T00:10:33Z

The main problem I'm having now though is, even though we got my init task to change the $PATH so that "python", "python3", "pip", etc. now all resolve to Python 3.10 as intended, VS Code is still choosing the system Python 3.8 by default every time you start it up: https://gitpod.io/#https://github.com/jab/bidict

And even though the https://www.gitpod.io/docs/languages/python#python-versions docs explain that you can use pyenv to install a different Python version, they don't say anything about how to get VS Code to actually use it.

Worse, if a user who wants to contribute to my project clicks my "edit in GitPod" link and then somehow actually notices they need to change VS Code to a different Python version, when they try to, the intended version Python 3.10.2 is not a directly available option (neither via ~/.pyenv/versions/3.10.2 nor via ~/.pyenv/shims/python):

So they have to somehow know to manually type in ~/.pyenv/shims/python to get the right version of Python:

Any ideas?

@jab Thanks for reporting the problem with the python version in vscode. But perhaps it would be good to speak on a different issue.

@axonasif @pawlean Can I ask you to handle this issue? Shall I create an issue in gitpod-io/gitpod?

utam0k · 2022-03-03T02:43:56Z

@jab @samirotiv @huweiATgithub
Hi, I'm sorry for waiting for you. I've released new images which are included in fixing this bug, and it worked fine.
utam0k/kedro@12511e8

Sorry for the confusion about the name of the image tag, but carefully that the tag I used in the commit above is newer than latest.

huweiATgithub · 2022-03-04T05:11:00Z

@utam0k That tag you mentioned works! Thank you. Hope the latest tag will get that fix soon.

@jab you may try to reload your vscode (close and re-open the window, don't close the entire workspace). The stuff in init is running with the launching of vscode I guess (as you can see the terminal that runs init commands in the opened workspace). I tested, after reloading vscode, it can discover the newly installed python interpreter.
Another two solutions I can think of are:

maintain a vscode setting that sets interpreter path
create a Dockerfile that build from workspace-full and install python3.10.2 there so that it is there before the start of vscode

kylos101 · 2022-03-04T17:54:35Z

Hi @samirotiv and @huweiATgithub ,

Should be all set now. Have a nice weekend!

Status: Downloaded newer image for gitpod/workspace-python:latest
gitpod ~ $ which python
/home/gitpod/.pyenv/shims/python
gitpod ~ $ which python3
/home/gitpod/.pyenv/shims/python3

jab · 2022-03-07T19:42:36Z

Thanks, everyone.

@utam0k wrote:

@jab Thanks for reporting the problem with the python version in vscode. But perhaps it would be good to speak on a different issue.

@axonasif @pawlean Can I ask you to handle this issue? Shall I create an issue in gitpod-io/gitpod?

Looks like no one did this yet, so I just opened gitpod-io/gitpod#8646

utam0k · 2022-03-07T23:54:19Z

@jab Thank you for opening new issue 👍

huweiATgithub · 2022-04-22T05:50:12Z

I re-open this issue, as in the latest full image (e249a43d75e6), the path is still in a mess, i.e. it is in the wrong order, and it contains many duplicates.
Though the Python only image (gitpod/workspace-python:latest) works well (as the problem is introduced by merging which may not affect the python image), it will be good if the full image got fixed (it is the default image, so that there is no need to declare the image to use in .gitpod.yml).

Update: I cannot open this ... @utam0k @kylos101

utam0k · 2022-04-25T00:02:43Z

@huweiATgithub Thanks for your report. Please let me organize the problem you pointed out. What order do you expect and what is the actual order regarding $PATH? Or is it random?

I re-open this issue, as in the latest full image (e249a43d75e6), the path is still in a mess, i.e. it is in the wrong order, and it contains many duplicates.

kylos101 · 2022-04-25T03:34:10Z

@huweiATgithub may we ask you to pen a new issue, describing the problem with workspace-full that you see, with steps to recreate, expected results, and actual results? Depending on what you see, it may be a problem with Dazzle, rather than just the workspace-full image itself.

samirotiv added the bug Something isn't working label Feb 6, 2022

Furisto self-assigned this Feb 14, 2022

kylos101 added this to 🌌 Workspace Team Feb 15, 2022

kylos101 moved this to Scheduled in 🌌 Workspace Team Feb 15, 2022

kylos101 added team: workspace lang: python labels Feb 15, 2022

antonymilne mentioned this issue Feb 16, 2022

pip install in editable mode no longer works #694

Closed

utam0k mentioned this issue Feb 20, 2022

The order of $PATH is not guaranteed gitpod-io/dazzle#42

Closed

kylos101 unassigned Furisto Feb 21, 2022

kylos101 mentioned this issue Feb 22, 2022

Create new dazzle release & use for main and PR workflows #721

Closed

utam0k self-assigned this Feb 28, 2022

utam0k moved this from Scheduled to In Progress in 🌌 Workspace Team Feb 28, 2022

utam0k mentioned this issue Mar 1, 2022

Gitpod break the behavior of pre-commit of Python gitpod-io/gitpod#4886

Closed

utam0k changed the title ~~Incorrect path of python3 - not using pyenv~~ Incorrect $PATH affecting many executables (including python3) Mar 1, 2022

kylos101 closed this as completed Mar 4, 2022

Repository owner moved this from In Progress to Done in 🌌 Workspace Team Mar 4, 2022

jab added a commit to jab/bidict that referenced this issue Mar 7, 2022

rm workaround for gitpod-io/workspace-images#664 now that it's fixed

545416a

jab added a commit to jab/bidict that referenced this issue Mar 7, 2022

rm workaround for gitpod-io/workspace-images#664 now that it's fixed

062ccb6

Incorrect $PATH affecting many executables (including python3) #664

Incorrect $PATH affecting many executables (including python3) #664

Comments

samirotiv commented Feb 6, 2022

Bug description

Steps to reproduce

Expected behavior

Example repository

Anything else?

kylos101 commented Feb 16, 2022

Uh oh!

huweiATgithub commented Feb 18, 2022

Uh oh!

huweiATgithub commented Feb 18, 2022

Uh oh!

utam0k commented Feb 20, 2022

Uh oh!

huweiATgithub commented Feb 28, 2022

Uh oh!

utam0k commented Feb 28, 2022

Uh oh!

jab commented Mar 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huweiATgithub commented Mar 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jab commented Mar 1, 2022

Uh oh!

jab commented Mar 1, 2022

Uh oh!

huweiATgithub commented Mar 1, 2022

Uh oh!

utam0k commented Mar 1, 2022

Uh oh!

utam0k commented Mar 1, 2022

Uh oh!

utam0k commented Mar 1, 2022

Uh oh!

utam0k commented Mar 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jab commented Mar 2, 2022

Uh oh!

utam0k commented Mar 3, 2022

Uh oh!

utam0k commented Mar 3, 2022

Uh oh!

utam0k commented Mar 3, 2022

Uh oh!

huweiATgithub commented Mar 4, 2022

Uh oh!

kylos101 commented Mar 4, 2022

Uh oh!

jab commented Mar 7, 2022

Uh oh!

utam0k commented Mar 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huweiATgithub commented Apr 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

utam0k commented Apr 25, 2022

Uh oh!

kylos101 commented Apr 25, 2022

Uh oh!

Incorrect `$PATH` affecting many executables (including python3) #664

Incorrect `$PATH` affecting many executables (including python3) #664

jab commented Mar 1, 2022 •

edited

Loading

huweiATgithub commented Mar 1, 2022 •

edited

Loading

utam0k commented Mar 1, 2022 •

edited

Loading

utam0k commented Mar 7, 2022 •

edited

Loading

huweiATgithub commented Apr 22, 2022 •

edited

Loading