port shebang of tools from python2 to python3

[shihai1991]

BPO	41626
Nosy	@terryjreedy, @tiran, @ned-deily, @merwok, @encukou, @corona10, @shihai1991
PRs	bpo-41626: port shebang of tools from python2 to python3 #21948 gh-85792: Remove the shebang of internal Tools to avoid confusion. #23581

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-08-24.16:28:45.369>
labels = ['library', '3.10']
title = 'port shebang of tools from python2 to python3'
updated_at = <Date 2020-12-01.11:51:42.061>
user = 'https://github.com/shihai1991'

bugs.python.org fields:

activity = <Date 2020-12-01.11:51:42.061>
actor = 'christian.heimes'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Demos and Tools', 'Library (Lib)']
creation = <Date 2020-08-24.16:28:45.369>
creator = 'shihai1991'
dependencies = []
files = []
hgrepos = []
issue_num = 41626
keywords = ['patch']
message_count = 8.0
messages = ['375855', '379244', '379247', '379279', '379969', '379970', '382238', '382241']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'christian.heimes', 'ned.deily', 'eric.araujo', 'petr.viktorin', 'corona10', 'shihai1991']
pr_nums = ['21948', '23581']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue41626'
versions = ['Python 3.10']

Linked PRs

• gh-85792: Update shebang to python3 #108273
• gh-85792: Update shebang to python3 #130863

[shihai1991]

After the EOL of python2, the shebang of tools should be ported from python2 to python3.

some files like: https://github.com/python/cpython/blob/master/Objects/typeslots.py#L1

[merwok]

This is a minor thing, and not a particularly visible bug, but it could create confusion about how to run scripts so could be fixed.

1) For stdlib modules, I think the intended usage is python3 -m encodings.rot_13; I don’t expect anyone to run python3 /usr/lib/python3.10/encodings/rot_13.py to do a decoding in the terminal!
So for these easter eggs / hidden gems in the stdlib, I would delete the shebang and remove the executable bit. There are 18 such modules, ignoring test, lib2to3, turtledemo and a few others named below.

1.1) There may be some exceptions: for Lib/idlelib/pyshell.py for example, maybe the IDLE entry point is just a symlink to that file, and removing the executable bit would break it. So this can’t be a mass update but each case should be reviewed.

2) Some of the files in the PR are used during development of CPython: Tools/clinic/clinic.py Python/makeopcodetargets.py Parser/asdl_c.py Objects/typeslots.py Modules/_sha3/cleanup.py Lib/ctypes/macholib/fetch_macholib Mac/BuildScript/build-installer.py
I think some of these work with any Python, but some need the locally built interpreter to give correct results.
It could avoid confusion and mistakes to remove the shebangs and executable bits from the scripts that need a local Python, and make dure their documentation mentions ./python path/to/tool.

2.1) One recent example, Tools/peg_generator/pegen/__main__.py is documented to work with Python 3.8+, and confusingly is both a __main__ module (to support python3 -m pegen and an executable script with shebang!)

2.2) In Tools/ssl, we have /usr/bin/env python3 in make_ssl_data.py and ./python in multissltests.py. Are these working?!

3) The files in Tools/scripts are documented as «useful for building Python» or «generally useful scripts». They nearly all executable and nearly all use #!/usr/bin/env python3, so fixing the few python references there seems ok. Likewise for Tools/pynche/pynche.

3.1) For Tools/gdb/libpython.py I don’t know how it’s meant to be used; if it’s not distributed but only for debugging a locally built CPython, then removing the shebang seems correct.

(Oh and by the way, Tools/scripts/fixps.py is a tool to replace /usr/local/bin/python shebangs with /usr/bin/env python… You can judge if that is useful and/or correct.)

So if things are changed (let’s get a couple opinions here), maybe this needs separate PRs for 1/2/3, and make sure to get reviews to ensure we don’t break IDLE / Mac installers / pegen / etc.

[ned-deily]

To address some of the concerns:

• The shebang line in Mac/BuildScript/build-installer.py can be safely removed.

• "Lib/idlelib/pyshell.py for example, maybe the IDLE entry point is just a symlink to that file"
  It's not, at least for unix-y builds. In [install-prefix], an "idle*" script is installed of the form:

#![install-prefix]/bin/python3.[n]

from idlelib.pyshell import main
if __name__ == '__main__':
    main()

So there is an absolute link to the correct interpreter and the shebang in pyshell.py is not used.

[terryjreedy]

Thoughts I wrote on the PR that belong here. (Thanks for the reminder, Éric.):
*The use of #! in both stdlib and tools seems rather inconsistent.

• Stdlib modules are best run with <python> -m mod so as to run the code with the exact python binary they are meant for. So maybe the marker should be be removed at least from /Lib/*. But...

-- I was not really aware of idlelib.pyshell. Running with an explicit binary seems particularly important for IDLE. About once a month on SO, some beginner posts about not being able to import a module they downloaded when running IDLE (maybe only sometimes). Nearly always, they have 2 pythons, such as from Anaconda and python.org.

I may want or need to deprecate using pyshell as entry instead of idle.py/idlew.py/idle.bat as I want to move startup code from pyshell to either idle.py or a separate startup only file. I am not sure how to get from here to there, partly because I don't really know what 'here' is in practice.

• With 2.7 put to bed, the line is hardly needed to select between latest 2.x and latest 3.x.

• The response of py.exe to shebang lines needs to be detailed and considered. I believe it only knows about python.org installs, so it will only start the latest python.org install. I don't know what it does if 32- and 64- bit versions are both present. I also don't know what it runs if the 'default' installed version is not the latest installed version. Does py.exe know what the default is, or does that just affect what 'python' runs?

[shihai1991]

Sorry for my delay~

I think some of these work with any Python, but some need the locally built interpreter to give correct results.
It could avoid confusion and mistakes to remove the shebangs and executable bits from the scripts that need a local Python, and make dure their documentation mentions ./python path/to/tool.

I update Objects/typeslots.py in #PR21913 recently and using f-format in this script(it means that just local python or python3 can run). So I will accept your suggestion from this file after this PR merged:)

(Oh and by the way, Tools/scripts/fixps.py is a tool to replace /usr/local/bin/python shebangs with /usr/bin/env python… You can judge if that is useful and/or correct.)

Thanks, I will take a look.

So if things are changed (let’s get a couple opinions here), maybe this needs separate PRs for 1/2/3, and make sure to get reviews to ensure we don’t break IDLE / Mac installers / pegen / etc.

Agree. To corner case, a single sepereate PR is good.

[shihai1991]

typo error: PR21913->PR21931

[shihai1991]

I create PR23581 to remove the inner Tools' shebang.

But I am not remove Tools/ssl/make_ssl_data.py, Tools/gdb/libpython.py in PR23581.

• To make_ssl_data.py, I guess Christian may give us some guide.
• libpython.py will be installed in kinds of OS, some OS will use or maintain python2.7 for a long times. so I perfer like to keep it.

[tiran]

make_ssl_data.py is an internal tool that generates internal header files. It's rarely used. The tool is not bound to any Python version, too. Feel free to change the shebang or leave it as it is. I don't care.

[terryjreedy]

As near as I can tell, encodings.rot_13 is currently the only$ /Lib module with a shebang line with 'python' rather than 'python3'. #108273 proposes to modify this one file. Does this seem appropriate?

$ idlelib/idletest/example_noext has the same line (with a harmless bug), but it is a dummy file for testing whether IDLE recognizes an extensionless file with a python shebang line as python source. I intend to change it for consistency, but as part of a PR making other changes and changing other files and backporting them.

As noted above, idlelib.pyshell has a 'python3' shebang line. It has not been the official initial startup file for at least a decade. idlelib.idle does not have such a line, so it seems to be unneeded. I will think about removing that line (and exec bit?) in 3.13.

There seems to be as many /Lib files with if __name__ == '__main__': without a shebang line as with.

[encukou]

I think that today, it's best to close this issue without action. python2/python3 confusion should be history now.
I'll close next month if there are no objections.

[merwok]

This wasn’t only about 2 vs 3 ‑ there were also notes about stdlib modules having executable bit and shebang needlessly

[encukou]

OK, I'll only close the PR for internal tools.

[merwok]

That last part is handled in #118673

[sandy-lcq]

There are still several files using shebang "/usr/bin/env python", add a pull request to fix them.