diff --git a/peps/pep-0773.rst b/peps/pep-0773.rst index c26ce1f695c..a61dffa7033 100644 --- a/peps/pep-0773.rst +++ b/peps/pep-0773.rst @@ -24,6 +24,13 @@ existing installers for the platform, while avoiding most of their limitations, and provides the core team with the ability to manage releases for many years to come. +With the new installer, the recommended command to launch the default Python on +Windows will be ``python``, while the command for managing multiple versions +(including launching a specific one) will be ``py``. Users can also opt-in to +having global ``python3.x`` commands by adding an extra entry to their ``PATH`` +environment variable. We recommend starting with the "How to teach this" section +for the most concise high-level overview of the features. + Reader's Note ------------- @@ -173,18 +180,18 @@ to the latest release. The user's own data is stored in an OS-managed location in their user profile, and is able to be reset, backed-up and restored using regular OS functionality. -Nuget Package +NuGet Package ------------- -The Nuget packages for CPython are produced and published as part of our +The NuGet packages for CPython are produced and published as part of our normal release process. The contents are identical to the traditional -installer. A Nuget package is published to nuget.org, which is a package +installer. A NuGet package is published to nuget.org, which is a package manager typically associated with .NET languages, but highly integrated with any project supported by Visual Studio. This makes it a nice format for users who want a lightweight install of Python as part of their regular build process, and can simplify embedding scenarios. -The packages are installed using any tool capable of using the Nuget API, or +The packages are installed using any tool capable of using the NuGet API, or may be downloaded directly once the URL of the package is known. The package is a plain ZIP file with some metadata. It contains the CPython interpreter, the standard library, development headers and import libraries, and pip. It does @@ -303,7 +310,7 @@ to Microsoft). Without managing these, it is relatively easy for an undesired version to be launched, though in general the targets can only be changed manually by the user, and not by merely installing another app. -Both the Nuget package and the embeddable distro are as simple and reliable to +Both the NuGet package and the embeddable distro are as simple and reliable to install as extracting an archive file, though it's worth noting that for many Python users this is not a common task. They provide no install management at all, and cannot be reliably updated other than by deleting and re-extracting. @@ -315,8 +322,6 @@ failure. Overview of PyManager ===================== -("PyManager" name open for bikeshedding) - PyManager is the internal name of our proposed replacement installer tool. It will be distributed both in the Windows Store and on python.org as an MSIX package. Downloading from either source will get an identical package, and @@ -354,8 +359,9 @@ management use, due to its brevity. ``py install ...``, ``py list ...`` and so on. The proposed commands are detailed later. The existing behaviour of the :pep:`397` launcher is preserved, however, launching through ``py`` will not automatically install runtimes (by default). If one is requested but is not -installed, users will just get an error. The ``py run ...`` subcommand, however, -will install automatically, and supports the same options as bare ``py``. +installed, users will just get an error. The ``py exec ...`` subcommand, +however, will install automatically, and supports the same options as bare +``py``. These commands are added at very low priority in the user's ``PATH`` by the OS. Every existing configuration we may have created on a user's machine will take @@ -379,10 +385,10 @@ Replacing other installers -------------------------- Our intent is to immediately stop publishing individual versions to the Windows -Store, and to deprecate and phase out the traditional installer and Nuget -packages by Python 3.16. The embeddable distro would remain, but its listing on -python.org download pages would be phased out and it will be available only -through PyManager. +Store, and to deprecate and phase out the traditional installer by Python 3.16. +The embeddable distro will remain, but its listing on python.org download pages +will be phased out and it will be available only through PyManager. No changes +will be made to the NuGet packages. PyManager will be made available as an app package downloadable manually from python.org, and the double-click install experience is generally smooth. This @@ -394,8 +400,11 @@ runtime after install. Some automated deployment scenarios do not work with the newer MSIX format, and so a simple MSI will also be provided on python.org. This will have no options or user interface, and require administrative privileges, which are typically -available for these kinds of scenarios. This MSI would be discouraged for most -users, and the MSIX should be the default. +available for these kinds of scenarios. Additionally, the MSI will be required +by users on non-standard systems, such as Wine, that do not support the MSIX +format. While not officially supported, the MSI will enable these platforms to +continue using upstream distributions of CPython. The MSI would be discouraged +for most other users, and the MSIX is considered the default. It's worth noting that there is no way to make the MSI install fully compatible with the MSIX, and users with both will likely encounter confusion or problems. @@ -435,12 +444,12 @@ Exec subcommand .. code:: text - pymanager exec -V:tag [interpreter opts] [script.py|-m module|-c code] [script args] - pymanager exec -3.* ... - pymanager exec [--only-managed] [--[no-]install] [-V:|-3.*] ... - py [-V:|-3.*] ... + py [-V:] [interpreter opts] [script.py|-m module|-c code] [script args] + py [-3.*] [interpreter opts] [script.py|-m module|-c code] [script args] python ... python3 ... + pymanager exec -V:tag ... + pymanager exec -3.* ... This subcommand is used to select and launch a runtime. It is the default action for the ``py`` command, and the only action supported by the ``python`` and @@ -490,11 +499,15 @@ be allowed. The best installed runtime matching the tag is then selected and launched with the remaining command line. -If no matching runtime is found, and the ``--install`` option is set, it will be -installed and then launched. This is the default for ``py exec``, and also for -``python`` but only when no runtime was requested (which implies that no -runtimes were installed). In all other cases, if no install is found, an error -is printed and the process exits. +If no matching runtime is found, the ``py exec`` and ``pymanager exec`` commands +will automatically install and launch one (user configuration permitting). The +``py``, ``python`` and ``python3`` commands will report and error and exit. +However, in the case where no runtimes are available at all, including none +detected from other installers, the first runtime will be automatically +installed. This provides a useful first-launch experience, where a new user who +has just installed PyManager will directly launch the latest (or requested) +version of CPython. After this first time, errors will again be reported when a +requested runtime is not found. Install subcommand @@ -502,9 +515,9 @@ Install subcommand .. code:: text - py install [-s|--source ] [-f|--force] [-u|--upgrade] [tag ...] - py install [-s|--source ] [-t|--target ] [tag ...] - py install [-s|--source ] [-d|--download ] [tag ...] + py install [-s|--source ] [-f|--force] [-u|--upgrade] tag [...] + py install [-s|--source ] [-t|--target ] tag + py install [-s|--source ] [-d|--download ] tag [...] py install --refresh .. note:: @@ -518,7 +531,8 @@ and are used to search the index file. Company names match as case-insensitive prefixes, preferring a full match over a prefix, and tags use case-insensitive, number-aware matches, with dotted numbers treated as versions. Tags must match one of the listed "install for" tags, and entries list multiple such tags to -handle abbreviated requests. +handle abbreviated requests. The special tag ``default`` resolves to the user's +configured default (typically ``3``). For example, the ``3.10.5`` entry would list all of ``3``, ``3.10`` and ``3.10.5`` as tags to be installed for. A request for ``3.10`` would match one @@ -540,20 +554,20 @@ Users are expected to use shorter tags for convenience, rather than ranges. The default index file is hosted on python.org, and contains install information including package URLs and hashes for all installable versions. An alternate index may be specified by the user or their administrator (see Configuration -below). Entries in the index file list the full tags they should be installed -for, and if an exact match is found the package will be selected. In the case -of no exact match, a prefix match will be used. In both cases, numbers in the -tag are treated logically - that is, ``3.1`` is a prefix of ``3.1.2`` but not of -``3.10``. +below). Entries in the index file list the full set of tags they should be +installed for, and if an exact match is found the package will be selected. In +the case of no exact match, a prefix match will be used. In both cases, numbers +in the tag are treated logically - that is, ``3.1`` is a prefix of ``3.1.2`` but +not of ``3.10``. If a tag is already satisfied by an existing install, nothing will be installed. The user must pass an ``--upgrade`` or ``--force`` option to replace the existing install; the former will only replace it with a newer version, while the latter will remove and replace even with the same version. -Calling the command without providing any tags will install the latest default -version (in effect, the first non-prerelease entry in the index). -Passing ``--upgrade`` with no tags is an error. +Calling the command without providing any tags will not install anything, but +will display the help text and an error (as for any invalid option). However, +passing ``--upgrade`` with no tags will attempt to upgrade all installs. Passing ``--refresh`` will regenerate all metadata and shortcuts for all installs. This is intentionally applied to all installs at once, as shortcut @@ -588,6 +602,13 @@ be prompted before uninstalling each runtime. If the ``--purge`` option is passed with no tags, then (after confirmation) all runtimes will be removed, along with shortcuts and any cached files. +Uninstalling PyManager does not uninstall any runtimes that were installed. For +technical reasons, this would not be reliably possible (we cannot run arbitrary +code at uninstall time), and so instead we deliberately ensure that anything +that has been installed will continue to work. Reinstalling PyManager allows +management of these installs to resume. Running ``py uninstall --purge`` before uninstalling +PyManager will perform a complete uninstall. + List subcommand --------------- @@ -596,8 +617,7 @@ List subcommand py list [-f|--format ] [-1|--one] [--only-managed] [tag ...] py list [-f|--format ] [-1|--one] [--online] [--source ] [tag ...] - py [--list|-0] - py [--list-paths|-0p] + py [--list|--list-paths|-0|-0p] This subcommand will list any or all installs matching the specified tags or ranges. If no tags are provided, lists all installs. Runtimes not managed by @@ -767,13 +787,18 @@ stored in the root of each package archive. The entries in the bundled file will fill in any gaps from the index file. This is intended to allow the typically large ``shortcuts`` key to be removed from the index file, but may also extend to ``alias``, ``executable`` and ``executable_args``. Omitting other keys from -the index may result in problems installing the package. +the index may result in problems installing the package. Ensuring correct +behaviour is left to the implementation - this is not an interoperability point, +and so we do not intend to specify the details here (beyond the normal compatibility +requirements that apply). A second top-level key ``next`` contains an optional URL to another index. This may be used if PyManager cannot find a suitable package in the included versions. The intent is to allow for older indexes to be archived and only accessed when required, reducing the size of the initial download without -cutting off users from older versions. +cutting off users from older versions. When searching for a suitable install, +later indexes will not be searched if a viable candidate is found (in other +words, the first index consulted should have the latest versions in it). The initial schema is shown below: @@ -867,11 +892,14 @@ suitable runtime for the script. Unlike the support currently in the ``py.exe`` launcher, we propose to reduce this functionality to only support Python commands where the command matches -a global alias listed for an install. (The existing launcher is able to run any -executable, and attempts to extract version information from the command, rather -than simple matching.) +a global alias listed for an install, with anything not matching being treated +as an arbitrary executable path. In practice, this is expected to produce the +same (or better) results for non-contrived cases, but may result in subtle +changes where users are relying on unspecified edge case behaviours of the +existing launcher. -The specific patterns to be detected are left to the implementation. +The specific patterns to be detected are left to the implementation, but should +be largely compatible with the existing launcher. Rationale @@ -906,7 +934,8 @@ launches the most appropriate version of Python available on the user's machine. This is much closer to the desired behaviour of the global ``python`` command than any alternative. -TODO: Note that this follows Gentoo's design as well +It has been noted that Gentoo also distributes an intelligent ``python`` +command, in order to serve their users better than a simple symlink. Replacing py.exe @@ -935,6 +964,13 @@ always find ``py`` resolving to the launcher. Ultimately, the only way to resolve this in favour of PyManager is to uninstall the launcher, which can be done through the standard Installed Apps control panel. +We propose to update the existing launcher, which will continue to be released +with the traditional installer, to detect attempted use of the new subcommands +and provide a useful message for the user rather than the current error. These +warnings can also detect the unlikely case where a user is intentionally +launching extensionless files by those names and retain that behaviour, allowing +a viable alternative for users who cannot make other changes to their setup. + Interaction with venv --------------------- @@ -943,7 +979,9 @@ An activated virtual environment, as implemented by the standard library :mod:`venv` module, will modify the user's ``PATH`` environment variable to ensure that the venv launcher will take precedence over other executables. As a result, when a venv has been activated, PyManager can only be launched by its aliases -other than ``python``. +other than ``python``. When an active virtual environment is detected, it will +be treated as the user's default runtime (except for uninstall), which ensures +that other commands will also behave as expected. This means that working virtual environments will behave as they do today with no additional support from PyManager. @@ -975,7 +1013,8 @@ predictable, and so we have no recourse to do anything other than apologise and suggest users use our own tooling for downloads. The deprecation period for the traditional installer allows time for these users -to learn about the upcoming change. +to learn about the upcoming change. Where possible, we will add deprecation +warnings to the traditional installer. Scripted installs @@ -989,7 +1028,8 @@ intervention (that is, someone has to change the command already), this is considered an acceptable change. The deprecation period for the traditional installer allows time for these users -to learn about the upcoming change. +to learn about the upcoming change. Where possible, we will add deprecation +warnings to the traditional installer. Old runtime installed @@ -1046,7 +1086,7 @@ Old version availability Python versions prior to the first release of PyManager can be backfilled into the python.org index, either based on newly repackaged archives or using the -almost equivalent packages from Nuget (the latter does not include Tcl/Tk, +almost equivalent packages from NuGet (the latter does not include Tcl/Tk, making them significantly incompatible for some users, but this is likely okay for especially old versions). @@ -1061,8 +1101,9 @@ distribution, and so we will simply not provide an option for them. Third parties who desire this functionality are encouraged to provide their own distributions. -PyManager can only be installed for all users, and can be extensively configured -by an administrator, including to constrain the actual runtimes which users may +PyManager can only be installed for all users, though an MSIX does not require +administrative privileges to install, and can be extensively configured by an +administrator, including to constrain the actual runtimes which users may install. Additionally, PyManager supports local extraction for bundling, and so embedding apps can easily generate their own layout, which can be installed for all users if they so desire. @@ -1074,18 +1115,30 @@ changes, this will be documented only. Build-time installs ------------------- -Users currently using Nuget packages will also have to change to a new workflow. -Further investigation is required to determine how best to support this, as it -is possible that the PyManager MSIX package may not be installable on all -continuous integration systems. No differences are anticipated between a package -installed by PyManager compared to one installed by Nuget. - Users using the embeddable distro may have to change to a new method for discovering the URL to the packages, though the recommendation would be to use PyManager to discover and install. No differences are anticipated due to the change of installer, and the embeddable distro package would be identical to today. +No changes to the NuGet packages are proposed. + + +Single architecture installer +----------------------------- + +The current proposal only makes an Intel 64-bit build of PyManager available. +This will prevent users on 32-bit only operating systems or CPUs from being able +to install PyManager. As this is a vanishingly small proportion of machines +today, and an even smaller proportion of developer machines (the target audience +for PyManager), we are not concerned about excluding users. + +Windows ARM64 machines support running binaries built for Intel 64-bit through +efficient emulation. CPython 32-bit builds will still be available as they have +an important role in integrating with 32-bit executables, which does not allow +substitution of 64-bit binaries. As PyManager runs as a standalone executable, +this is not a necessary feature for the manager. + Security Implications ===================== @@ -1114,13 +1167,20 @@ use it to install Python. The admin-only configuration described earlier in attacker who can run PyManager is able to do whatever the user can do, and only a complete application whitelisting approach can prevent the use of Python. -TODO: Notes about security of package acquisition (tl;dr: TLS) +Acquisition of packages over HTTPS is protected by the connection security. We +use native Windows download mechanisms that support public and enterprise +certificates, as well as authenticated proxy servers. The feed may include +hashes for downloadable packages, which will be verified, but there is no +third-party hosted verification built into PyManager. This is consistent with +today's model. Runtime installs by PyManager are fully accessible by, and modifiable by, the current user. This is equivalent to typical installs using the traditional -installer or a Nuget package, but is more vulnerable to tampering than a Store +installer or a NuGet package, but is more vulnerable to tampering than a Store install or a per-machine install with the traditional installer. It is not possible to fully protect an install from the user who installed it. +Reinstalling or updating an install performs a clean install, which will revert +any tampering that may have occurred since the original install. The aliases generated by PyManager when installing a runtime are designed to use a signed, unmodified executable that uses an adjacent data file to launch @@ -1287,12 +1347,16 @@ lines: Reference Implementation ======================== -The reference implementation is available at https://github.com/zooba/pymanager/ -with a precompiled MSIX package under the Releases at -https://github.com/zooba/pymanager/releases. This sample includes a bundled -index, rather than a hosted one, and references a range of existing Nuget +The reference implementation is available at `the author's repository +`_ +with a precompiled MSIX package under `Releases +`_. This sample includes a bundled +index, rather than a hosted one, and references a range of existing NuGet packages to allow install testing. +Reference documentation can also be found in `the same repository +`_. + Rejected Ideas ============== @@ -1322,7 +1386,8 @@ for Windows. Due to the additional steps needed to make this functional on other platforms, and the fact that there isn't a need to replace existing installers for those platforms, we consider this idea out of scope for this PEP. It may be pursued in -the future. +the future (and the contributors most likely to do so have indicated that they +are looking into it and would be able to use a consistent interface). Include a runtime pre-installed with the manager @@ -1352,8 +1417,8 @@ as is appropriate (whether through a shebang or an active environment). Include a runtime INSTEAD OF the manager ---------------------------------------- -This is the current situation, which we are trying to change. If you made this -argument, and somehow still read this far, please go back and start again. +This is the current situation, which we are trying to change. If you read this +far and still chose to make this argument, please go back and start again. Use a built-in module rather than subcommands @@ -1523,7 +1588,7 @@ Just publish the plain ZIP file ------------------------------- Publishing the plain ZIP file is part of the plan, however, it will not be -visibly listed (for example, on the python.org downloads pages, though they will +visibly listed (for example, on the python.org download pages, though they will be visible in the FTP view). An alternative would be to publish and list these packages, and expect users to download and manually extract and configure them. @@ -1543,6 +1608,7 @@ The index protocol and download list will be available for tools that wish to use it, or for users who are willing to navigate JSON in order to find the URL. The ``--target`` option on the install command also provides a trivial download and extract operation, allowing users to have the same experience as a ZIP file. +And the ``--download`` option can give users the ZIP file still zipped. Only publish PyManager to one place @@ -1561,12 +1627,6 @@ Many users rely on the Windows Store app to install packages, and the built-in redirector in Windows can only open to a Store page. As such, removing the Store app is equivalent to denying hundreds of thousands of installs each month. -Additionally, automatic updates are only supported through the Store. We would -have to implement automatic updates manually if we did not publish there. It is -possible to have the MSIX from python.org find its own updates on the Store, and -we can assume that machines without access are responsible for their own -updates. - The two builds are practically identical. The only difference between the MSIX we provide to the Store and the one that goes to python.org is package signing: we sign the python.org package ourselves, while the Store package is signed as @@ -1606,7 +1666,36 @@ match its requirement, but this is not part of the initial proposal. Open Issues =========== -TODO: Raise open issues +Generate packages for old versions +---------------------------------- + +At present, we can install versions of Python back to 3.5 from the NuGet feed. +These are identical to the traditional installer releases of the same versions, +but do not include Tkinter, IDLE, documentation, or the test suite. However, it +is very little work (and no additional storage space) to use them for older +releases. + +New binaries releases will generate new packages, even for already released +versions (currently 3.12 and 3.13). So these will exist. The open issue is +whether we go back through old releases and generate full PyManager packages for +them all, and how far back to go. This is merely repackaging, and does not +require rebuilding or modifying the release. + + +Include test suite in PyManager packages +---------------------------------------- + +The test suite consumes the majority of the size of our current distribution, +but is not intended to be used by most of our users. By omitting it from the +standard PyManager package, all downloads become smaller and faster, at the cost +of some users having to obtain the test suite in another way (such as building +from source). + +It seems obviously preferable to have some way to access the exact test suite +intended for a particular build. So the question is whether to include it by +default (there are no optional components anymore), to publish it independently +(for manual download as a ZIP file), or to publish a second feed of packages +that include both the runtime and the test suite. Footnotes