Skip to content

gh-87901: os.popen: Fix new encoding argument. #92415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from

Conversation

methane
Copy link
Member

@methane methane commented May 7, 2022

  • Make it keyword only.
  • Fix it wasn't passed to subprocess.Popen

#87901

@methane methane added the type-bug An unexpected behavior, bug, or error label May 7, 2022
@methane methane changed the title gh-87901: os.popen: Make encoding keyword only gh-87901: os.popen: Fix new encoding argument. May 7, 2022
Copy link
Contributor

@hauntsaninja hauntsaninja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks for fixing!

@JelleZijlstra
Copy link
Member

You should also add a unit test that uses this new parameter.

@methane methane added the needs backport to 3.11 only security fixes label May 8, 2022
Copy link
Member

@serhiy-storchaka serhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a What's New entry?

@@ -998,7 +998,7 @@ def _empty_mapping(self):
def test_update2(self):
os.environ.clear()
os.environ.update(HELLO="World")
with os.popen("%s -c 'echo $HELLO'" % unix_shell) as popen:
with os.popen("%s -c 'echo $HELLO'" % unix_shell, encoding="utf-8") as popen:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not a test for os.popen. It works without explicit encoding, so no need to change it.

Instead look at test_popen. Needed tests for encoding and errors, and they should fail if revert changes in os.popen, so use non-ASCII data and encodings which give different result in comparison with common locale encodings: utf-8, latin1 or cp1252.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to add a new test for the private attribute to check for the encoding? To check encoding="utf-8".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer not to rely on private attributes if possible. Checking the result of the decoding is enough.

@@ -998,7 +998,7 @@ def _empty_mapping(self):
def test_update2(self):
os.environ.clear()
os.environ.update(HELLO="World")
with os.popen("%s -c 'echo $HELLO'" % unix_shell) as popen:
with os.popen("%s -c 'echo $HELLO'" % unix_shell, encoding="utf-8") as popen:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to add a new test for the private attribute to check for the encoding? To check encoding="utf-8".

@methane methane removed the skip news label May 13, 2022
@serhiy-storchaka
Copy link
Member

There is a dedicated test Lib/test/test_popen.py for this.

Test:

  • Non-ASCII output with default encoding and errors.
  • Non-ASCII output with different values of encoding.
  • Non-ASCII output with a specified encoding and different values of errors.
  • Non-ASCII command arguments and how they interact with the specified encoding.

Include tests for the locale encoding and the filesystem encoding if they are relevant.

@methane
Copy link
Member Author

methane commented May 13, 2022

os.popen() is just a simple wrapper of subprocess.Popen()
And since os.popen() uses shell=True with string cmd, writing/maintain such a test in cross platform is 100x difficult than its implementation.

@methane
Copy link
Member Author

methane commented May 13, 2022

Or should we revert the encoding argument and recommend using subprocess instead?

@vstinner
Copy link
Member

os.popen() is just a simple wrapper of subprocess.Popen()

I dislike this API since it always use a shell and so is more likely to be vulnerable to shell code injection. It may also be less efficient than avoiding a shell. I would prefer to deprecate this function, but I was never brave enough to propose deprecating it. At least, I removed platform.popen() ;-)

@methane
Copy link
Member Author

methane commented May 16, 2022

I created #92836 that remove encoding argument and add document instead.

@methane methane closed this May 19, 2022
@methane methane deleted the os-popen-encoding branch May 19, 2022 03:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting core review needs backport to 3.11 only security fixes type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants