Skip to content

Update getcommandpath to use the PATH variable set through setenv=PATH=... #1148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Update getcommandpath to use the PATH variable set through setenv=PATH=... #1148

wants to merge 2 commits into from

Conversation

Bhargavasomu
Copy link

Fixes #1146

@Bhargavasomu
Copy link
Author

@gaborbernat could you please review?

@Bhargavasomu
Copy link
Author

Could you please help me with the errors that I am seeing.

@obestwalter
Copy link
Member

obestwalter commented Jan 25, 2019

https://toxdev.visualstudio.com/tox/_build/results?buildId=1152 - black needed to reformat your changes. Please run tox -e fix_lint locally.

@obestwalter
Copy link
Member

The other error seems unrelated to your changes on first look but does not occur on master, so this needs looking into more closely. Please run the tests locally and have a closer look.

Copy link
Member

@gaborbernat gaborbernat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before we can consider this to merge we should discuss what the expected behavior is for edge cases, and then add a few tests to check for validity and defend against regressions.

@Bhargavasomu Bhargavasomu changed the title Update os.environ[PATH] thorughout code when used setenv=PATH=... Update getcommandpath to use the PATH variable set through setenv=PATH=... Jan 26, 2019
@Bhargavasomu
Copy link
Author

@gaborbernat @obestwalter how come Tox CI hasn't run? Also please review the recent changes, I have fixed this in kind of an elegant way to read and understand. Please let me know for further changes.

@Bhargavasomu
Copy link
Author

Also when I installed this version of tox, the bug which I was personally experiencing got fixed !!

@Bhargavasomu
Copy link
Author

On python3.5 running the test tests/unit/test_venv.py is behaving randomly. Please apply this PR on a python3.5 environment and see the test behaving randomly on multiple runs. Could someone please explain me why that is happening?

@Bhargavasomu
Copy link
Author

EDIT : Fixed all the local errors in py3.5 and py3.4 etc. But it is not visible, because Tox CI is not running. Can someone please look into this?

@gaborbernat
Copy link
Member

This PR still lacks tests, without it it's small chance of getting accepted. Please add unit test for each feature improvement one by one, taking care of edge cases too. Furthermore I would prefer to have a one place path mangling, now you do it in two places which makes the code hard to maintain.

@Bhargavasomu
Copy link
Author

path mangling, now you do it in two places which makes the code hard to maintain

@gaborbernat I understand your view, but the following issue was what I faced when I tried to PATH mangling only once.

In the following snippet, the path order to be maintained is envbindir + os.environ["PATH"]

tox/src/tox/venv.py

Lines 514 to 518 in d6cd8d4

os.environ.pop("VIRTUALENV_PYTHON", None)
env = self._get_os_environ(is_test_command=is_test_command)
bin_dir = str(self.envconfig.envbindir)
env["PATH"] = os.pathsep.join([bin_dir, os.environ["PATH"]])
self.session.report.verbosity2("setting PATH={}".format(env["PATH"]))
.
But when I try to pass that variable to _normal_lookup, even envbindir would be passed, which in practice/theory is not correct. Hence I put it this way. Please correct me if I am wrong.

I will add test cases, in a while. Could you please let me know why Tox CI is not running?

@gaborbernat
Copy link
Member

@Bhargavasomu
Copy link
Author

Bhargavasomu commented Jan 26, 2019

@gaborbernat that was an old build. I have made 2 pushes after that. (force pushes). But it is not updating. Does force push not restart the Tox CI?

@gaborbernat
Copy link
Member

gaborbernat commented Jan 26, 2019

The CI sometimes gets stuck, you can push a new commit to try to retrigger it again, here's the latest https://dev.azure.com/toxdev/tox/_build/results?buildId=1156

@gaborbernat
Copy link
Member

@Bhargavasomu arguably the order should be setenv + (bindir + what is inherited from shell - in case the user does extends via the setenv - otherwise these latter two probably should not be included). That being said maybe we should not allow users to overwrite PATH, only prepend via setenv. @obestwalter @asottile what do you think?

Copy link
Contributor

@asottile asottile left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

took a look at the cod first, let me check the thing I was supposed to read next 😆

else:
env["PATH"] = os.pathsep.join([bin_dir, os.environ["PATH"]])
# Remove redundantly duplicated directories in PATH without changing order
env["PATH"] = ":".join(list(dict.fromkeys(env["PATH"].split(":"))))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is (for instance) incorrect on windows -- use os.pathsep instead of ':' -- also it's unnecessary (and imo a surprising behaviour) to remove duplicates so this modification probably shouldn't happen at all

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(above is true for .split(":") as well -- quite surprised this patch passes tests on windows?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not passing on the windows test cases, and you gave me a solution. Thankyou.

@asottile
Copy link
Contributor

hmm yeah overriding PATH might sidestep some of the helpful things that tox does (and further complicate tox's setup)

From a test isolation perspective (one of tox's goals) it seems odd that you'd add (for instance from the issue) a directory starting at HOME. Though I could see a usecase that's like setenv = {toxinidir}/bin:{env:PATH} or some such (though I'd never do that, I'd just invoke {toxinidir]/bin/foo)

So I guess I'd like to see a usecase that is more reasonable ? Or maybe convince me what you already have is reasonable? 🤔 (not trying to shoot this down, legitimately curious how this would / should work)

@Bhargavasomu
Copy link
Author

@asottile, one use case that I came across is as follows. If suppose the user installs an application using tox (via a bash script maybe), then he has to modify the path, so that the installed executable can be invoked from anywhere directly by its name insted of the whole absolute path. This is further needed, because the test cases (in my case), make use of the executable by calling it by its name instead of the whole path.

@Bhargavasomu
Copy link
Author

@asottile , further I am working on writing a test case, which will make things clear.

@Bhargavasomu
Copy link
Author

@asottile I have added the new testcase. Please check it out.

@gaborbernat
Copy link
Member

I'll be honest I still don't entirely see the reason for this. What you describe sounds like you want to side step the tox as entry point which I find very dangerous. The more I think about it the surer I am that the only reasonable thing which we could allow is injecting elements into the path before the environment path, but after the tox environments bin folder.

@gaborbernat
Copy link
Member

On the subject of simplifying the path I think it's ok and will help decrease log size and debug. We should definitely keep only the first instance of path occurances. We could also remove not existing paths (I'm not entirely settled on this yet though). That being said this change is orthogonal to this PR so maybe let's keep it separate.

@asottile
Copy link
Contributor

We could also remove not existing paths (I'm not entirely settled on this yet though).

(not sure how relevant it is to tox -- but) I often use a pattern like this when creating virtualenvs in docker:

# set up all env variables in one layer
ENV PATH=/venv/bin:$PATH ... ...  
RUN virtualenv /venv && pip install ...

I could see a case where removing non-existent paths would break stuff

@Bhargavasomu have you considered using something like pytest-env or maybe a session-scoped auto-use pytest fixture?

@Bhargavasomu
Copy link
Author

@asottile @gaborbernat I'm really confused from all the above comments. Could you please frame it in a better way as to do what next?

@gaborbernat
Copy link
Member

@asottile was alluding to the fact that's probably more robust to alter the PATH from within the tests (via some session fixture e.g.) rather than inside tox. This way your tests will work also for cases when users are not calling the test from within tox (e.g. they do from within an IDE).

@Bhargavasomu
Copy link
Author

Thankyou for the support.

@Bhargavasomu Bhargavasomu deleted the fix_setenv_path_bug branch February 7, 2019 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants