Regression in handling concurrency when creating environments

[vlad-ivanov-name]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

pixi run --frozen --manifest-path pixi.toml some-command |
pixi run --frozen --manifest-path pixi.toml another-command -
⠁ updating packages in 'default'                                                                                                                                                                                                                                                                                                                          
⠁ updating packages in 'default'
⠁ updating packages in 'default'
  ✔ downloading          [00:00:01] Done!
  ✔ linking              [00:00:01] Done!                                                                                                                                                                                                                                                                                                                   × failed to post process the environment (unclobbering)
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

[project]
name = "requirements-3.10.13"
version = "0.1.0"
channels = ["conda-forge"]
platforms = ["linux-64"]

[tasks]

[dependencies]
python = "==3.10.13"
pip-tools = ">=7.3.0,<7.4"

Issue description

In this example, pixi runs a command in an environment, and pipes the output to another command. This seems to break (seemingly after switch to uv but could be unrelated).

The example is run from a clean state.

Doesn't work with 0.16.1

Expected behavior

Works with 0.15.2

[ruben-arts]

He @vlad-ivanov-name,
I'm can't seem to easily reproduce it, or think of a reason this would have changed. Could you give the actual commands that would trigger this error?

[vlad-ivanov-name]

I tried a few times and got a different error now but I think the root cause is probably related:

pixi run --frozen --manifest-path pixi.toml python3 file.py |
pixi run --frozen --manifest-path pixi.toml pip-compile -r - --output-file requirements.txt

⠁ updating packages in 'default'
  × failed to link 'bin/python3.10'
  ├─▶ could not open destination file for writing
  ╰─▶ Text file busy (os error 26)

file.py

if __name__ == "__main__":
    print("pytest==7.1.3")

Perhaps there's no lock on the .pixi environment dir so initializing it from both commands fails?

[ruben-arts]

On my Fedora machine I can not reproduce. Is this on windows perhaps?

That said maybe @baszalmstra you have any ideas?

[vlad-ivanov-name]

I’m also testing on Fedora 🤔

[wolfv]

I wonder if it's something related to this? rust-lang/rust#46016

Rye also had this issue: astral-sh/rye#861

[vlad-ivanov-name]

Just to clarify - the issue only occurs for me when the .pixi folder is deleted. The lockfile is present. The error above is on 0.17.1

[wolfv]

One other idea would be to set PYTHONUNBUFFERED=1, does that change anything?

[vlad-ivanov-name]

One other idea would be to set PYTHONUNBUFFERED=1, does that change anything?

Unfortunately no, still getting errors.

I noticed that the errors are sporadic, it's probably a race condition. I used this snippet to try to run the command until the error appears:

for i in $(seq 1 100) ; do rm -rf .pixi && ( pixi run --frozen --manifest-path pixi.toml python3 file.py | pixi run --frozen --manifest-path pixi.toml pip-compile -r - --output-file requirements.txt ) || break ; done

[vlad-ivanov-name]

I think perhaps broken pipe error is just a consequence of the consumer end of the pipe failing to properly initialize and read stdin

Screen.Recording.2024-03-26.at.09.40.49.mov

[vlad-ivanov-name]

Actually the minimal reproducible example for me is even simpler, no python involved at all:

pixi run --frozen --manifest-path pixi.toml echo | pixi run --frozen --manifest-path pixi.toml cat

[vlad-ivanov-name]

Same bug on MacOS:

[vlad-ivanov-name]

Just tried the latest version, the bug is still there:

⠁ updating packages in 'py38'
⠁ updating packages in 'py38'
  × failed to post process the environment (unclobbering)
  × failed to link 'include/readline/tilde.h'
  ╰─▶ EOF while parsing a value at line 1 column 0

  ├─▶ could not update destination file permissions
  ╰─▶ No such file or directory (os error 2)