[BUG] `pip install -e .` on Windows creates broken .pth files #3937

simaoafonso-pwt · 2023-05-29T17:37:16Z

setuptools version

67.8.0

Python version

3.8.10

OS

Windows 10

Additional environment information

Tried with newer Python versions, same problem.

Description

I tried to install my PEP571 package in editable mode and it works, but since my installation path is not ASCII, the .pth file is created as UTF-8 and Python just ignores it.

Expected behavior

Using pip install -e should create non-UTF-8 .pth files.

How to Reproduce

Start with an empty folder
Setup the venv and the project:

> git clone https://support.powertools-tech.com/PowertoolsTech/tkmilan.git -b setup/pyproject tkmilan_ç_utf8
Cloning into 'tkmilan_ç_utf8'...
remote: Enumerating objects: 2520, done.
remote: Counting objects: 100% (1011/1011), done.
remote: Compressing objects: 100% (624/624), done.
remote: Total 2520 (delta 557), reused 484 (delta 236), pack-reused 1509
Receiving objects: 100% (2520/2520), 995.17 KiB | 21.63 MiB/s, done.
Resolving deltas: 100% (1445/1445), done.

> py -3.8 -m venv venv

> venv\Scripts\activate.bat

> pip install -U pip setuptools wheel
Requirement already satisfied: pip in %TEMP%\setuptools_test\venv\lib\site-packages (21.1.1)
Collecting pip
  Using cached pip-23.1.2-py3-none-any.whl (2.1 MB)
Requirement already satisfied: setuptools in %TEMP%\setuptools_test\venv\lib\site-packages (56.0.0)
Collecting setuptools
  Using cached setuptools-67.8.0-py3-none-any.whl (1.1 MB)
Collecting wheel
  Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 56.0.0
    Uninstalling setuptools-56.0.0:
      Successfully uninstalled setuptools-56.0.0
  Attempting uninstall: pip
    Found existing installation: pip 21.1.1
    Uninstalling pip-21.1.1:
      Successfully uninstalled pip-21.1.1
Successfully installed pip-23.1.2 setuptools-67.8.0 wheel-0.40.0

>pip install -e tkmilan_ç_utf8
Obtaining file:///%TEMP%/setuptools_test/tkmilan_%C3%A7_utf8
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: tkmilan
  Building editable for tkmilan (pyproject.toml) ... done
  Created wheel for tkmilan: filename=tkmilan-0.31.dev5+g89be321-0.editable-py3-none-any.whl size=7606 sha256=af651bdbad009100b19fe77bf9e920808d738d260e908894f46d95b1e22d3977
  Stored in directory: %TEMP%\pip-ephem-wheel-cache-hzsqdlvi\wheels\0b\a3\d4\32ec164451dad8b10051ddc54df87ea215cc03fcb18d742dfa
Successfully built tkmilan
Installing collected packages: tkmilan
Successfully installed tkmilan-0.31.dev5+g89be321

Try and run the showcase, notice it fails:

> tkmilan-showcase
Traceback (most recent call last):
  File "C:\Program Files\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "%TEMP%\setuptools_test\venv\Scripts\tkmilan-showcase.exe\__main__.py", line 4, in <module>
ModuleNotFoundError: No module named 'tkmilan'

Edit venv\Lib\site-packages\__editable__.tkmilan-0.31.dev5+g89be321.pth on Notepad and save as ANSI format.
Try and run the showcase, notice it works:

> tkmilan-showcase

Output

See above

The text was updated successfully, but these errors were encountered:

abravalheri · 2023-05-30T11:35:36Z

Hi @simaoafonso-pwt, could you please post here what would be the contents of the .pth file?
Any chance this is a bug with Python itself? Python3 is supposed to have support for UTF-8 isn't it?

simaoafonso-pwt · 2023-05-30T11:41:03Z

Hi @simaoafonso-pwt, could you please post here what would be the contents of the .pth file?

It's a single line the correct path (%TEMP%/setuptools_test/tkmilan_ç_utf8/src), but as UTF-8 text. As mentioned, if I open the file in Notepad and save it as ANSI text, python picks it up correctly.

Any chance this is a bug with Python itself? Python3 is supposed to have support for UTF-8 isn't it?

I think paths on Windows are a special case, they are not UTF-8.
I guess Python itself reads these .pth files using the default locale. So the producer of .pth files should write it using the default locale too, which I think it's setuptools, correct me if I'm wrong.

abravalheri · 2023-05-30T11:52:30Z

Does anything get added to sys.path at all, or is it all ignored?

simaoafonso-pwt · 2023-05-30T12:09:13Z

It's not added to sys.path. I tried adding the correct path manually and the package is found.

As UTF-8:

> python
Python 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', 'C:\\Program Files\\Python38\\python38.zip', 'C:\\Program Files\\Python38\\DLLs', 'C:\\Program Files\\Python38\\lib', 'C:\\Program Files\\Python38', '%CD%', '%CD%\\lib\\site-packages']
>>> import tkmilan
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'tkmilan'
>>> sys.path.append(r"%HOME%\tkmilan\src")
>>> import tkmilan
>>>

As ANSI:

>python
Python 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', 'C:\\Program Files\\Python38\\python38.zip', 'C:\\Program Files\\Python38\\DLLs', 'C:\\Program Files\\Python38\\lib', 'C:\\Program Files\\Python38', '%CD%', '%CD%\\lib\\site-packages', '%HOME%\\tkmilan\\src']

abravalheri · 2023-05-30T13:31:26Z

@simaoafonso-pwt, I am trying to create a reproducer with a windows container to ask the Python maintainers for advice about this matter.

When I try to change the encoding to ASCII I am no longer able to use the ç character in the terminal (but I see that you are able to do that in your example). What is the encoding your system is using?

In my machine I have $OutputEncoding set to US-ASCII, [System.Text.Encoding]::Default set to iso-8859-1 and chcp configured to 437 (I got these values using PowerShell).
When I run a windows container (python:3.10-windowsservercore-1809), I get $OutputEncoding => US-ASCII, [System.Text.Encoding]::Default => UTF-8 and chcp => 0.

Could you please tell me what are the values in your system?

simaoafonso-pwt · 2023-05-30T14:19:14Z

Not sure if this answers your question, I'm not familiar with all these PowerShell commands.

PS %HOME%> Get-SystemLanguage
pt-PT

PS %HOME%> [System.Text.Encoding]::Default

IsSingleByte      : True
BodyName          : iso-8859-1
EncodingName      : Western European (Windows)
HeaderName        : Windows-1252
WebName           : Windows-1252
WindowsCodePage   : 1252
IsBrowserDisplay  : True
IsBrowserSave     : True
IsMailNewsDisplay : True
IsMailNewsSave    : True
EncoderFallback   : System.Text.InternalEncoderBestFitFallback
DecoderFallback   : System.Text.InternalDecoderBestFitFallback
IsReadOnly        : True
CodePage          : 1252

PS %HOME%> echo $OutputEncoding

IsSingleByte      : True
BodyName          : us-ascii
EncodingName      : US-ASCII
HeaderName        : us-ascii
WebName           : us-ascii
WindowsCodePage   : 1252
IsBrowserDisplay  : False
IsBrowserSave     : False
IsMailNewsDisplay : True
IsMailNewsSave    : True
EncoderFallback   : System.Text.EncoderReplacementFallback
DecoderFallback   : System.Text.DecoderReplacementFallback
IsReadOnly        : True
CodePage          : 20127
PS %HOME%> chcp
Active code page: 850

abravalheri · 2023-05-30T18:15:08Z

Thank you very much Simão.

I did not manage to reproduce this with a Windows container (for some reason they seem to always be UTF-8, and I did not find out how to change that), however I managed to run this in a Windows machine I have access to, with the following steps:

Open the run prompt with Windows+R
Run intl.cpl
Go to the Administrative tab, then Change system locale
Make sure the UTF-8 option is not used and that the appropriate encoding is used (compatible with iso8859-1)
Using PowerShell check if the [System.Text.Encoding]::Default is compatible with iso8859-1/ Windows 1252
(if not it might be necessary to change the system locale again and/or restart)

Create virtual environment using PowerShell:

# rm -R -Force $env:TEMP\test
mkdir $env:TEMP\test
cd $env:TEMP\test
python3 -m venv .venv

In the same PowerShell terminal, simulate an UTF-8 .pth file:

new-item create_files.py
@"
from pathlib import Path
proj = Path("tkmilan_ç_utf8")
(proj / "src/mypkg").mkdir(parents=True, exist_ok=True)
(proj / "src/mypkg/__init__.py").write_text("print('hello world')", encoding="utf-8")
Path(".venv/Lib/site-packages/custom.pth").write_text(str((proj / "src").resolve()), encoding="utf-8")
"@ | add-content -Encoding utf8 -Path create_files.py
.venv\Scripts\python create_files.py

Verify that the UTF-8 entry in the custom.pth is not added to sys.path when Python runs:
```
.venv\Scripts\python -c 'import sys; print(sys.path)'
```

Modify the custom.pth file to the iso8859-1 encoding:

rm create_files.py
new-item create_files.py
@"
from pathlib import Path
proj = Path("tkmilan_ç_utf8")
Path(".venv/Lib/site-packages/custom.pth").write_text(str((proj / "src").resolve()), encoding='iso8859-1')
"@ | add-content -Encoding utf8 -Path create_files.py
.venv\Scripts\python

Verify that the ISO 8859 entry is custom.pth is added to sys.path when Python runs:
```
.venv\Scripts\python -c 'import sys; print(sys.path)'
```

abravalheri · 2023-05-30T18:26:31Z

@jaraco do you have any advice here?

In the editable_wheel command, we write the .pth file to the wheel using writestr (which accepts a bytes object. To create this bytes object we currently use utf-8:

setuptools/setuptools/command/editable_wheel.py

Lines 404 to 405 in 1aed0ff

    
           contents = bytes(f"{entries}\n", "utf-8") 
        
           wheel.writestr(f"__editable__.{self.name}.pth", contents)

We could change the utf-8 to locale.getpreferredencoding(), however I am in doubt about this approach because of the following:

locale.getpreferredencoding() has side effects and is not thread safe (it can be invoked with do_setlocale=False, but then, the docs seem to indicate it may fail) -- Not sure if side-effects and thread-safety is something we have to worry about in setuptools.
Python3 uses UTF-8 by default for source files... .pth files are kind of source files, they can even contain Python code, it would seem to me that they should be treated as UTF-8.
There is an open issue in the CPython repository about it .pth files cannot contain folders with utf-8 names python/cpython#77102, but I don't think there is a clear position from the core maintainers if the issue is considered a pending bug, a wontfix, or not a bug at all...
I am not sure what will happen in other versions of Python/Windows... What if we change to locale.getpreferredencoding(), any chance we end up having the opposite problem later?
Additionally, if we produce a .pth file with ISO 8859 encoding and later the user sets the PYTHONUTF8 environment variable or use python -X utf8, then the .pth file will not work.

abravalheri · 2023-08-09T14:05:23Z

Hi @simaoafonso-pwt.
We received feedback from Python core developers that I tried to reflect on #4009.

I run the following as a manual test (PowerShell):

cd $env:TEMP
git clone https://github.com/pypa/sampleproject tkmilan_ç_utf8
cd tkmilan_ç_utf8
python3 -m venv .venv
.venv\Scripts\python -m pip install -U 'pip==23.2'
.venv\Scripts\python -m pip install -U wheel https://github.com/abravalheri/setuptools/archive/refs/heads/issue-3937.zip
.venv\Scripts\python -m pip install -e . --no-build-isolation

Then I tried to open .venv\Lib\site-packages\__editable__.sampleproject-3.0.0.pth with the Notepad app and it says Windows CRLF, ANSI.

I also tried to run the following (with no error):

.venv\Scripts\python -c 'import sample; print(sample)'
<module 'sample' from '...TEMP..\\tkmilan_ç_utf8\\src\\sample\\__init__.py'>

However that might not be a good indication if things are working or not, because I never managed to reproduce the error localy.

Can you have a look to see if the PR solves the problem in your system (see in the steps above how I install the version of setuptools provided by the PR and how I disable the build isolation to ensure that is the version of setuptools that is used during the build.).

simaoafonso-pwt · 2023-08-09T16:48:24Z

Hi @abravalheri

Can you have a look to see if the PR solves the problem in your system (see in the steps above how I install the version of setuptools provided by the PR and how I disable the build isolation to ensure that is the version of setuptools that is used during the build.).

Can confirm the problem is fixed with that setuptools version. Using Python 3.11, but I confirmed with Python 3.8 too.
I updated to the latest pip version (23.2.1) and it works as well.

Thanks for taking care of that.

simaoafonso-pwt added bug Needs Triage Issues that need to be evaluated for severity and status. labels May 29, 2023

abravalheri mentioned this issue May 30, 2023

.pth files cannot contain folders with utf-8 names python/cpython#77102

Closed

abravalheri added Waiting Upstream Input and removed Needs Triage Issues that need to be evaluated for severity and status. labels Jun 20, 2023

abravalheri mentioned this issue Aug 9, 2023

Use default encoding when creating .pth files in editable_wheel #4009

Merged

2 tasks

abravalheri closed this as completed in #4009 Aug 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[BUG] `pip install -e .` on Windows creates broken .pth files #3937

[BUG] `pip install -e .` on Windows creates broken .pth files #3937

simaoafonso-pwt commented May 29, 2023

abravalheri commented May 30, 2023

Uh oh!

simaoafonso-pwt commented May 30, 2023

Uh oh!

abravalheri commented May 30, 2023

Uh oh!

simaoafonso-pwt commented May 30, 2023

Uh oh!

abravalheri commented May 30, 2023 •

edited

Loading

Uh oh!

simaoafonso-pwt commented May 30, 2023

Uh oh!

abravalheri commented May 30, 2023

Uh oh!

abravalheri commented May 30, 2023 •

edited

Loading

Uh oh!

abravalheri commented Aug 9, 2023

Uh oh!

simaoafonso-pwt commented Aug 9, 2023

Uh oh!

Uh oh!

[BUG] pip install -e . on Windows creates broken .pth files #3937

[BUG] pip install -e . on Windows creates broken .pth files #3937

Comments

simaoafonso-pwt commented May 29, 2023

setuptools version

Python version

OS

Additional environment information

Description

Expected behavior

How to Reproduce

Output

abravalheri commented May 30, 2023

Uh oh!

simaoafonso-pwt commented May 30, 2023

Uh oh!

abravalheri commented May 30, 2023

Uh oh!

simaoafonso-pwt commented May 30, 2023

Uh oh!

abravalheri commented May 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simaoafonso-pwt commented May 30, 2023

Uh oh!

abravalheri commented May 30, 2023

Uh oh!

abravalheri commented May 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

abravalheri commented Aug 9, 2023

Uh oh!

simaoafonso-pwt commented Aug 9, 2023

Uh oh!

[BUG] `pip install -e .` on Windows creates broken .pth files #3937

[BUG] `pip install -e .` on Windows creates broken .pth files #3937

abravalheri commented May 30, 2023 •

edited

Loading

abravalheri commented May 30, 2023 •

edited

Loading