Windows python3 executable

[ptyork]

ISSUE:

On Windows, a Python 3 install directory contains python.exe. However, there is no python3.exe, python3.cmd or similar. Thus, a user following almost any online tutorial, copy/pasting installation scripts, etc. will fail because all are written for *nix platforms that standardize on python3 as the name of the executable.

Worse, there is a python3.exe in a default AppData\Local\Microsoft\WindowsApps directory which is a stub to open the Windows Store to download Python 3. So the unsuspecting user is greeted with the option to install Python 3 from the Windows Store, which could break an existing installation and certainly will not do what they intended.

This is obviously only an issue for novice users. But I teach novice users. And this comes up numerous times each semester despite posting FAQ's and warnings related to this issue. And it seems very easy to fix in the default install.

This seems like it should be an issue long since reported, but I cannot find it by searching. So I apologize if this is a duplicate.

REQUESTED FIX:

One option is simply to add a copy of python.exe named python3.exe in the base install directory. This appears to be how PIP is handled, with pip.exe, pip3.exe and pip3.X.exe all being copies of the same executable located in a Scripts subdirectory.

Alternatively, adding a python3.cmd script that calls python.exe and forwards all args would serve the same function. Though I'm unsure that saving 100MB would be worth the potential confusion.

Linked PRs

• gh-99185: Copy python executable (python.exe) to python3.exe #100184

[FFY00]

@zooba do you have any proposals for this? On unix, we have symlinks up to the executable with the minor version: python > python3 > python3.12. I assume this is not the case on Windows because symlinks aren't reliably supported, so it would require us to copy the interpreter. Do you know if there is any other mechanism that allow us to implement this reliably? If not, would it be reasonable to install very simple launcher executables that mimic the symlink behavior? I definitely think this is a goal worth pursuing.

One option is simply to add a copy of python.exe named python3.exe in the base install directory. This appears to be how PIP is handled, with pip.exe, pip3.exe and pip3.X.exe all being copies of the same executable located in a Scripts subdirectory.

pip's case is a bit different, all the executables are launchers for a Python module.

[pochmann]

a user following almost any online tutorial, copy/pasting installation scripts, etc. will fail because all are written for *nix platforms that standardize on python3 as the name of the executable.

Hmm, I checked about 10 of the top Google results for python tutorial, and that's not at all what I saw...

[ptyork]

Thanks for the reply and attention.

pip's case is a bit different, all the executables are launchers for a Python module.

True, though I believe on Windows, python.exe simply invokes functions in python3X.dll. Here's a directory listing:

08/30/2021  07:36 PM           101,608 python.exe
08/30/2021  07:36 PM            59,624 python3.dll
08/30/2021  07:36 PM         4,488,424 python39.dll
08/30/2021  07:36 PM         2,504,187 python39.zip

The executable stubs are even smaller than the pip ones:

11/03/2021  08:44 AM           106,337 pip.exe
11/03/2021  08:44 AM           106,337 pip3.9.exe
11/03/2021  08:44 AM           106,337 pip3.exe

Yes, symbolic links exist on Windows, but only for NTFS, not FAT32. IMO, perhaps just following pip's lead and copying the executable stubs might be the lowest friction solution.

Hmm, I checked about 10 of the top Google results for python tutorial, and that's not at all what I saw...

Well, the first result...python.org...currently uses python3.11. How about I sidestep this and just modify my original assertion and say that "much of the documentation I send my students to" uses python3. But even if it uses python, my Mac users will have issues.

The REAL problem is that there is inconsistency between platforms and documentation. python works by default on Windows but python3 or python3.x does not. python does not work by default on MacOS or Linux (Debian/Ubuntu anyway), but python3 and python3.x does. I sent students to the MariaDB and MySQL connector pages. Windows folks had issues running python3 from MariaDB docs and Mac users had issues with python in the MySQL docs.

Likewise, pip works by default on Windows but not on Mac/*nix, which need pip3; but many package installation guides give command line examples using pip and not pip3. And then there's the py launcher for Windows, which is cool but doesn't exist on Mac/*nix and is unfortunately referenced in a number of tutorials. And I think the /usr/bin/python symbolic link is apparently supposed to be managed using update-alternatives on Linux, which I guess is similar, but is not installed by default nor is it available on Mac.

Anyway, point is there is inconsistency across platforms. And while some inconsistency is unavoidable, these seem unnecessary and do cause confusion for students...and distress for teachers. If we can just get python, python3, python3.X, pip, pip3, and pip3.x all working by default across all major platforms, that would be fantastic. Short of this, at least the _3 and _3.x ones...

...I guess asking for a default /usr/bin/python and /usr/bin/pip links to be added for Mac/*nix is probably out of scope for the issue, huh? ;)

[pochmann]

100MB

You mean 100 kB.

The python.org tutorial explains what to use for different systems, showing python3.11, py and python. Does your own tutorial also do that (in the intro chapter, not just in the FAQ) but students just choose to ignore that? (Maybe include a note that asking about it will cost them points in their next exam :-)

[ptyork]

Pedantry aside, yes my CS majors must of course learn the intricacies, though you'd be surprised at how little motivation many show to learn even the basics of terminal operations. I can and will hold these students accountable as it is a learning outcome. HOWEVER, students in other majors take classes or are simply given ancillary assignments in Python to provide exposure and understanding (and recruiting), not to master it. They copy/paste blocks of code that they find online or that I write. Having to constantly provide "translations" and then having them still encounter roadblocks and frustrations as they have to seek and wait potentially hours for assistance due to the need to add or remove a '3'...is avoidable pain. Pain that can needlessly drive away potentially talented technology majors, which is counter to our mission.

This is adding little to this issue. java/javac, dotnet, node/npm, gcc, clang, go, etc., etc. all are CLIs that are at least semantically identical cross-platform. There seems little reason for Python to be the outlier, especially as it is billed as an approachable introduction to coding.

[FFY00]

True, though I believe on Windows, python.exe simply invokes functions in python3X.dll. Here's a directory listing:

AFAIK it parses the command line and envvars and invokes the interpreter via python3X.dll.

And I think the /usr/bin/python symbolic link is apparently supposed to be managed using update-alternatives on Linux, which I guess is similar, but is not installed by default nor is it available on Mac.

This is actually a Debian (and naturally Ubuntu and anything else Debian-based) specific thing btw, other distributions don't have that.

Yes, symbolic links exist on Windows, but only for NTFS, not FAT32.

And IIRC there's a toggle setting to enable support for them 🙃

IMO, perhaps just following pip's lead and copying the executable stubs might be the lowest friction solution.

Well, pip's executable stubs (aka packaging entrypoint launchers) are a bit different, it's not as straight-forward here. Like I said above, I am pretty sure the executable is more than just a stub, so we need to create stubs.

...I guess asking for a default /usr/bin/python and /usr/bin/pip links to be added for Mac/*nix is probably out of scope for the issue, huh? ;)

No, if we are fixing the inconsistencies, I think it should be done for all platforms. On macOS it can be done via symlinks though, I am actually surprised you mentioned it doesn't already do this.

This is adding little to this issue. java/javac, dotnet, node/npm, gcc, clang, go, etc., etc. all are CLIs that are at least semantically identical cross-platform. There seems little reason for Python to be the outlier, especially as it is billed as an approachable introduction to coding.

I agree, it's unfortunate. If you are teaching introduction to Python, I would consider using IPython or Jupyter though. Anyway, I am glad to see you are advocating for your students, I wish my teachers did that when I was in college 😅

[ptyork]

Thanks for the clarifications, @FFY00!

On macOS it can be done via symlinks though, I am actually surprised you mentioned it doesn't already do this.

Surprised me, too. I just installed the latest from python.org just to make sure:

% where python
python not found
% where python3
/Library/Frameworks/Python.framework/Versions/3.11/bin/python3
/Library/Frameworks/Python.framework/Versions/3.10/bin/python3
/usr/local/bin/python3
/usr/bin/python3
% ls -l /usr/local/bin/python*
lrwxr-xr-x  1 root  wheel  70 Dec 16 14:48 /usr/local/bin/python3 -> ../../../Library/Frameworks/Python.framework/Versions/3.11/bin/python3
lrwxr-xr-x  1 root  wheel  77 Dec 16 14:48 /usr/local/bin/python3-config -> ../../../Library/Frameworks/Python.framework/Versions/3.11/bin/python3-config
lrwxr-xr-x  1 root  wheel  78 Dec 16 14:48 /usr/local/bin/python3-intel64 -> ../../../Library/Frameworks/Python.framework/Versions/3.11/bin/python3-intel64
lrwxr-xr-x  1 root  wheel  73 Sep 23 12:19 /usr/local/bin/python3.10 -> ../../../Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10
lrwxr-xr-x  1 root  wheel  80 Sep 23 12:19 /usr/local/bin/python3.10-config -> ../../../Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10-config
lrwxr-xr-x  1 root  wheel  81 Sep 23 12:19 /usr/local/bin/python3.10-intel64 -> ../../../Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10-intel64
lrwxr-xr-x  1 root  wheel  73 Dec 16 14:48 /usr/local/bin/python3.11 -> ../../../Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
lrwxr-xr-x  1 root  wheel  80 Dec 16 14:48 /usr/local/bin/python3.11-config -> ../../../Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11-config
lrwxr-xr-x  1 root  wheel  81 Dec 16 14:48 /usr/local/bin/python3.11-intel64 -> ../../../Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11-intel64
% ls -l /usr/bin/python*
-rwxr-xr-x  76 root  wheel  167120 Aug 24 04:59 /usr/bin/python3

Would also be nice to have it do the same for pip as there's only a /usr/local/bin/pip3 and pip3.x.

Debian/Ubuntu .deb package seems also not to add a /usr/bin/python or /usr/local/bin/python symbolic link. python3-pip DOES however have pip, pip3 and pip3.x script replicas. Different teams making the installers, I guess.

FWIW, I'll go to my grave not understanding the real distinction between /usr/bin and /usr/local/bin. 😆

If you are teaching introduction to Python, I would consider using IPython or Jupyter though.

Agreed!! We include Python in a (very) intro course, a "scripting and automation" course, database, and data science. Each kinda lends itself to different environments. Intro uses some kind of simple IDE (currently IDLE maybe?). Automation I think uses Mu, Data Science uses Jupyter. And my latest pain is in Database, where I need them to "experience" accessing a database from an application...here I use VSCode and CLI since they are connecting to a local MariaDB daemon.

I guess the challenge of having such a powerful, general purpose scripting language is that it is used in so many different ways...

[zooba]

I've been hesitant to add more executables to the Python install directory, because enough users already have messed up PATH that adding more is only going to be a bigger problem.

Any changes at all are going to cause churn, which is going to break people, which is going to use up any remaining goodwill we have left. I'd rather save the churn until we actually have a good, long-term fix. But I also don't know what that fix will be.

My best guess right now is that all the scenarios that would benefit from having "python3" be slightly more consistent (it's still not going to behave the same) would benefit far more from using a distro (who can easily do this themselves) or a preinstalled/configured environment such as via Jupyter or Minecraft. The python.org install of Python is such a long way from being a beginner friendly option that I'd rather leave that space clearly open for someone who wants to focus on that audience.

The Store install is a significant concession, but also properly handles the PATH modifications so I'm far less concerned about multiple executables (and the stub that launches the store was added by Microsoft in support - we don't control that from here, though I personally have the contacts to make changes if needed). That said, if they don't fix up the execution alias UI some time soon I'll seriously consider dropping some of the entry points.

[zooba]

enough users already have messed up PATH that adding more is only going to be a bigger problem.

To offer a really concrete example here:

• User installs (hypothetical) 3.11.2 with added python3.exe. 3.11 gets pushed to the front of PATH
• User installs security update to 3.10. 3.10 gets pushed to the front of PATH
• Now python.exe is 3.10 and python3.exe is 3.11. The only way to fix it is to manually modify PATH, and a user who knows how to do that could've set it up manually in a way that wouldn't break like this.

[FFY00]

I understand, and agree with your point regarding churn, however if we go that route I think we should decide what is our stance regarding installing versioned executable in general and document it, along with whatever exceptions we might have (eg. Windows if we decide that we want to install versioned executables as a general practice, or Linux if we decide we don't).

Having somewhere in the documentation where we can point people to learn what exactly is the difference in behavior between the different platforms would go a long way to mitigate this issue.

[ptyork]

• Now python.exe is 3.10 and python3.exe is 3.11.

and pip3.exe presumably would be 3.10. Yeah. That is a mess. Yuck!

As a "convenience" maybe coming up with a managed "standard" method of setting the current version that is consistent across platforms? Perhaps pysetver or something. It could present a list of known installs (in the path on Windows and common install locations elsewhere) and allow you to pick one to set as active? Would modify the path on Windows and update the symlink elsewhere? Kinda sorta like venv, but for the base install? Dunno. Might just be more churn...

Having somewhere in the documentation where we can point people to learn what exactly is the difference in behavior between the different platforms would go a long way to mitigate this issue.

Doesn't fix the copy/paste issue, but I agree that it is a good middle ground if no universal fix can be envisioned.

[zooba]

As a "convenience" maybe coming up with a managed "standard" method of setting the current version that is consistent across platforms? Perhaps pysetver or something.

This would be great, but unfortunately doesn't work well (on Windows at least). A process can't change the environment of its parent process, which would mean it would have to be a batch file and a PowerShell script and a Bash script to handle the three most popular shells (on Windows - Git Bash gets a lot of use), and nobody has signed up for that.

Would modify the path on Windows and update the symlink elsewhere

This is actually my ideal design, but it requires OS changes to work properly. I'm trying to argue those into existence before I give up and look for an alternative (potentially renaming the py.exe launcher to python.exe and python3.exe... which it already supports, but obviously the default counts for a lot... and I already know how this will break some users).

A non-terminal interface such as Jupyter, Mu or VS Code, is always going to avoid this level of confusion. It's a shame, but also pretty irreversible, that terminal skills are a prerequisite to using Python.

[ptyork]

A process can't change the environment of its parent process, which would mean it would have to be a batch file and a PowerShell script and a Bash script to handle the three most popular shells (on Windows - Git Bash gets a lot of use), and nobody has signed up for that.

True...but...to change the path permanently, you're modifying the registry using setx or similar. Which will require closing and reopening the command prompt anyway. So using C/Win32 or python + the winreg module could do this. You'd just need to print something like "Changes won't be effective until you close and reopen the command prompt" at the end.

Would be happy to throw together some proof-of-concept code for this if interested.

But, yes, a more elegant "work immediately" option would need to BOTH modify the registry and refresh the current environment from the registry. Sadly, an age-old issue since...ages old. Like me. :D And one that would indeed require a bat/ps1/sh script.

potentially renaming the py.exe launcher to python.exe and python3.exe

Interesting. Would be the "most similar" to symlinks on other platforms. But I can imagine this would almost have to be an "opt in" option. At least if other Python installs are detected.

No perfect solutions here, huh? At least without a Tardis.