Package with pyproject.toml cannot be installed with pip install -e

[erzoe]

OS version

Debian 11

Python version

Python 3.9.2

Pip version

pip 21.3.1

Guide link

https://packaging.python.org/en/latest/tutorials/packaging-projects/

Problem description

I have created an example project as explained in Packaging Python Projects with the following pyproject.toml file:

[build-system]
requires = [
    "setuptools>=42",
    "wheel"
]
build-backend = "setuptools.build_meta"

and setup.cfg:

[metadata]
name = example-pkg-erzo
version = file: src/example_package/version.txt
author = Example Author
author_email = author@example.com
description = A small example package
long_description = file: README.md
long_description_content_type = text/markdown
url = https://github.com/pypa/sampleproject
project_urls =
    Bug Tracker = https://github.com/pypa/sampleproject/issues
classifiers =
    Programming Language :: Python :: 3
    License :: OSI Approved :: MIT License
    Operating System :: OS Independent

[options]
package_dir =
    = src
packages = find:
python_requires = >=3.6

[options.packages.find]
where = src

[options.entry_points]
console_scripts =
    example-program = example_package.main:main
    hello-world = example_package.greet:main

Also I have added a setup.py because I need to include data files:

#!/usr/bin/env python3

import setuptools

setuptools.setup(package_data={"":["*.txt", "doc/*"]})

When I am trying to install the package with pip install -e . I am getting the error message listed in the next section.

Removing pyproject.toml resolves the problem. But does it cause other problems that I am missing right now?

Error message

Defaulting to user installation because normal site-packages is not writeable
Obtaining file://.../example_package_git
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Installing collected packages: example-pkg-erzo
  Running setup.py develop for example-pkg-erzo
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'.../example_package_git/setup.py'"'"'; __file__='"'"'.../example_package_git/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --user --prefix=
         cwd: .../example_package_git/
    Complete output (22 lines):
    running develop
    /tmp/pip-build-env-807d9_g2/overlay/lib/python3.9/site-packages/setuptools/command/easy_install.py:156: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    WARNING: The user site-packages directory is disabled.
    /tmp/pip-build-env-807d9_g2/overlay/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    error: can't create or remove files in install directory
    
    The following error occurred while trying to add or remove files in the
    installation directory:
    
        [Errno 13] Permission denied: '/usr/lib/python3.9/site-packages'
    
    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:
    
        /usr/lib/python3.9/site-packages/
    
    This directory does not currently exist.  Please create it and try again, or
    choose a different installation directory (using the -d or --install-dir
    option).
    
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'.../example_package_git/setup.py'"'"'; __file__='"'"'.../example_package_git/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --user --prefix= Check the logs for full command output.

[bhrutledge]

@erzoe Please don't run the sudo command that was suggested. I think the "permission denied" error is a symptom of your environment not being properly configured. I'm guessing you're trying to install into the system Python environment, which is leading to the error.

In general, it's best to run pip install from within a virtual environment; see https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/ for an overview.

To troubleshoot, can you share the output of the following commands?

which python

which pip

[erzoe]

Thank you for your fast reply. Here is the output of the requested commands:

$ which python
/usr/bin/python

$ which pip
/usr/bin/pip

[bhrutledge]

Yeah, that pip install is trying to install to the "system" Python environment, i.e. the one that's used by your OS, hence the permissions error. Projects should generally be installed in a virtual environment, esp. when you're using editable mode.

[erzoe]

Sorry, but I don't understand why I should install it in a virtual environment. Virtual environments are great for testing that the software works with other versions of python and other packages but this is the version that I want to use always (1) because I want to find the bugs that I am adding asap and (2) because I need the functionality that I am adding asap.

I guess I could install it in a virtual environment and start that virtual environment when I boot up my computer but then what's the point in the virtual environment?

[bhrutledge]

@erzoe I think the main issue with installing packages in a global Python environment is that there could be conflicts between their dependencies. For example, if package-a requires package-c==2.0, and package-b requires package-c==3.0, then installing package-a and package-b into the same Python environment will cause a conflict.

Also, the system Python environment is carefully constructed by your OS distributor (e.g. Debian) to provide the Python packages used by the OS and its packaged tools. So, it's best to heed the "permission denied" errors and leave that alone.

Your idea to create a "personal" virtual environment would address the second issue, but only partially address the first issue, because conflicts could still arise. An excellent tool that solves both issues is pipx, which handles creating isolated virtual environments and putting any command line scripts on your $PATH.

Further reading:

https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/
https://snarky.ca/a-quick-and-dirty-guide-on-how-to-install-packages-for-python/
https://snarky.ca/why-you-should-use-python-m-pip/

[erzoe]

Thank you for the information. Pipx seems to work fine. Thanks to your links I have also found this other interesting article on What to do when you botch a release on PyPI which might come in handy someday (well, hopefully not).

Nevertheless I am still going without a pyproject.toml file to avoid that error message because—as you have pointed out yourself—it tempts users to run pip with sudo which is in my experience more dangerous than installing a package outside of a virtual environment. But I am now showing how to install the program using pipx in my readme and have added a note that installing python packages outside of a virtual environment is discouraged and a link to your comment.

Also, it seems inconsistent if editable mode installs fail where normal installs work just fine.

[henryiii]

Pip is supposed to apply --user automatically if sudo doesn't work - you seem to have a recent pip version, so I'm not sure why it's not doing that - maybe a bug? pyproject.toml is the future of building, and is useful for other things too, so I'd not recommend avoiding it.

[henryiii]

Ahh, it could be because of the editable mode install. Pip might assume any user who knows enough to use an editable install will also know to use --user or a virtualenv? Even if it works, you should never make an editable install into a global site packages.

Technically, system python distributions are not intended to have any pip installs in them - those are for system installs only (apt, apk, yum, etc). Even upgrading pip is unsupported.

[bhrutledge]

@erzoe I agree with @henryiii: you should be using pyproject.toml, and I would avoid using -e unless you're actively developing the project, within a virtual environment.

More useful reading: https://snarky.ca/what-the-heck-is-pyproject-toml/

[erzoe]

@henryiii no, --user does not make a difference. And the first line says:

Defaulting to user installation because normal site-packages is not writeable

so pip recognizes correctly that --user should be used here.
I think the interesting part is this line:

WARNING: The user site-packages directory is disabled.

[erzoe]

It's not that I am opposed to using pyproject.toml, I would like to use that standard. My problem is that misleading error message which pip gives.

I looked at another project where I knew that this error message did not occur and noticed the difference was the existence of the pyproject.toml file. That is why I removed it from my project, it was the easiest way around this error message.

When I started using pip I, I did not know about --user yet and pip did not use it automatically yet. And when it prompted me with an error similar to this one I just tried again with sudo. And by doing that I broke the package manager of my operating system. That is why I don't want anyone to see this error message. So that they don't repeat my silly mistake.

On the other hand I have never experienced any problems with installing packages outside of virtual environments. (Although it's probably fair to mention that I have started using virtual environments after I learned about them and I knew that I would only need the package in a limited context and for a limited time, like for a university project. Maybe that has saved my some trouble.)

If pip gave a concise error message, one or two lines explaining that you should run that command inside of a virtual environment that would be ok. But with the error as it is I want to make sure that none of my users is accidentally running into it.

[erzoe]

It seems there is the possibility to avoid that error message while still having a pyproject.toml by adding the following lines in setup.py before calling setuptools.setup:

import site
import os

site.ENABLE_USER_SITE = os.geteuid() != 0

I have found that variable in this issue.

[henryiii]

none of my users

Are your users downloading your git source, then running pip install -e .? If so, then whatever docs directed them to do that should also mention what to do if they run into this sort of problem, and warn against sudo. If you are worried about "normal" users that do pip install package from pypi, that can't be editable.

If you want to suggest a better error message for pip, that's probably something you could request (check first to make sure no one already made the request).

tried again with sudo. And by doing that I broke the package manager

That's the point of sudo. It means you know what you are doing and take responsibility for breaking stuff.

The user site-packages directory is disabled.

Yep, that's why. You should never install to the system site packages, so if user site packages are disabled, you have a completely locked down system, no installs allowed.

System Pythons were not intended for users to mess with. They were intended to run system scripts and support system packages. Feel free to ask the packagers, they will tell you the one and only way to distribute Python packages is to make apt/yum/etc. packages, and pip should not be used. 🤦 That's why virtual environments are useful - they give you something you can edit, with an easy revert (delete the venv) and they manage to share the binary part of Python (so they are easy to setup).

I'd think your OS disabling user site packages is a mistake, you could look into why that was done. It's obviously pushing people toward sudo installs, which is much worse.

[henryiii]

seems there is the possibility to avoid that error message

I'd guess this would install to the user location, but since it's globally disabled, no one would be able to use it after it's installed?

[merwok]

Debian does not disable user site-packages, I am sure there is something more to understand why this happens.