Skip to content

test_tarfile fails on 32 bit systems in NoneInfoExtractTests_Data #107811

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
kanavin opened this issue Aug 9, 2023 · 6 comments
Closed

test_tarfile fails on 32 bit systems in NoneInfoExtractTests_Data #107811

kanavin opened this issue Aug 9, 2023 · 6 comments
Assignees
@encukou
Labels
tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@kanavin
Copy link
Contributor

kanavin commented Aug 9, 2023 •
edited by bedevere-bot
Loading

These tests are newly added, and they produce the following fails on 32 bit x86 systems (running python 3.11.4):

======================================================================
ERROR: test_extractall_none_gid (test.test_tarfile.NoneInfoExtractTests_Data.test_extractall_none_gid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3067, in test_extractall_none_gid
    with self.extract_with_none('gid'):
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3026, in extract_with_none
    self.tar.extractall(DIR, filter='fully_trusted')
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

======================================================================
ERROR: test_extractall_none_gname (test.test_tarfile.NoneInfoExtractTests_Data.test_extractall_none_gname)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3075, in test_extractall_none_gname
    with self.extract_with_none('gname'):
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3026, in extract_with_none
    self.tar.extractall(DIR, filter='fully_trusted')
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

======================================================================
ERROR: test_extractall_none_mode (test.test_tarfile.NoneInfoExtractTests_Data.test_extractall_none_mode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3053, in test_extractall_none_mode
    with self.extract_with_none('mode') as DIR:
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3026, in extract_with_none
    self.tar.extractall(DIR, filter='fully_trusted')
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

======================================================================
ERROR: test_extractall_none_mtime (test.test_tarfile.NoneInfoExtractTests_Data.test_extractall_none_mtime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3034, in test_extractall_none_mtime
    with self.extract_with_none('mtime') as DIR:
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3026, in extract_with_none
    self.tar.extractall(DIR, filter='fully_trusted')
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

======================================================================
ERROR: test_extractall_none_uid (test.test_tarfile.NoneInfoExtractTests_Data.test_extractall_none_uid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3063, in test_extractall_none_uid
    with self.extract_with_none('uid'):
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3026, in extract_with_none
    self.tar.extractall(DIR, filter='fully_trusted')
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: gid is greater than maximum

======================================================================
ERROR: test_extractall_none_uname (test.test_tarfile.NoneInfoExtractTests_Data.test_extractall_none_uname)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3071, in test_extractall_none_uname
    with self.extract_with_none('uname'):
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3026, in extract_with_none
    self.tar.extractall(DIR, filter='fully_trusted')
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

======================================================================
ERROR: setUpClass (test.test_tarfile.NoneInfoExtractTests_Default)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3002, in setUpClass
    tar.extractall(cls.control_dir, filter=cls.extraction_filter)
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

======================================================================
ERROR: setUpClass (test.test_tarfile.NoneInfoExtractTests_FullyTrusted)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3002, in setUpClass
    tar.extractall(cls.control_dir, filter=cls.extraction_filter)
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

======================================================================
ERROR: setUpClass (test.test_tarfile.NoneInfoExtractTests_Tar)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3002, in setUpClass
    tar.extractall(cls.control_dir, filter=cls.extraction_filter)
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

----------------------------------------------------------------------
Ran 545 tests in 8.309s

FAILED (errors=9, skipped=8)

I believe the file they are tripping over is:

alex@Zen2:/srv/work/alex/cpython$ tar --list --numeric-owner -vf Lib/test/testtar.tar|grep regtype-gnu-uid
-rw-r--r-- 4294967295/4294967295 7011 2003-01-06 00:19 gnu/regtype-gnu-uid

Linked PRs
@kanavin kanavin added the type-bug An unexpected behavior, bug, or error label Aug 9, 2023
@Eclips4 Eclips4 added the tests Tests in the Lib/test dir label Aug 9, 2023
@encukou encukou self-assigned this Aug 9, 2023
@encukou
Copy link
Member

encukou commented Aug 10, 2023 •
edited
Loading

What OS is this? I tried on 32-bit Debian but could not reproduce.

Looks like the tests uncovered a real issue on systems with a small uid_t -- extraction should behave like when chmod fails in any other way.

@kanavin
Copy link
Contributor Author

kanavin commented Aug 10, 2023

What OS is this? I tried on 32-bit Debian but could not reproduce.

Looks like the tests uncovered a real issue on systems with a small uid_t -- extraction behave like when chmod fails in any other way.

It's Yocto's poky using qemu x86 target. I can run local tests to help diagnose this if needed.

@encukou
Copy link
Member

encukou commented Aug 10, 2023

Can you try out this patch?

diff --git a/Lib/tarfile.py b/Lib/tarfile.py
index d62323715b4..5ef2394ad3a 100755
--- a/Lib/tarfile.py
+++ b/Lib/tarfile.py
@@ -2557,7 +2557,8 @@ def chown(self, tarinfo, targetpath, numeric_owner):
                     os.lchown(targetpath, u, g)
                 else:
                     os.chown(targetpath, u, g)
-            except OSError as e:
+            except (OSError, OverflowError) as e:
+                # OverflowError can be raised if an ID doesn't fit in `id_t`
                 raise ExtractError("could not change owner") from e
 
     def chmod(self, tarinfo, targetpath):

@kanavin
Copy link
Contributor Author

kanavin commented Aug 10, 2023

Can you try out this patch?

With the patch the test is passing:

root@qemux86:/usr/lib/python3.11# python3 -m test test_tarfile
0:00:00 load avg: 0.28 Run tests sequentially
0:00:00 load avg: 0.28 [1/1] test_tarfile

== Tests result: SUCCESS ==

1 test OK.

Total duration: 10.2 sec
Tests result: SUCCESS

encukou added a commit to encukou/cpython that referenced this issue Aug 23, 2023
@encukou
pythongh-107811: tarfile: treat overflow in UID/GID as failure to set it
3afaea1
@bedevere-bot bedevere-bot mentioned this issue Aug 23, 2023
Merged
encukou added a commit that referenced this issue Aug 23, 2023
@encukou
gh-107811: tarfile: treat overflow in UID/GID as failure to set it (#…
5d18715 
…108369)
@encukou
Copy link
Member

encukou commented Aug 23, 2023

Merged. Thanks for the report and testing!
As this is not a supported architecture, and it's only come up because the tests are now more comprehensive, I don't think it should be backported to 3.12.0rc. I assume poky has ways to patch it in if needed.

@encukou encukou closed this as completed Aug 23, 2023
@kanavin
Copy link
Contributor Author

kanavin commented Aug 23, 2023

No problem. I'll do a local backport.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@encukou encukou

Labels
tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@encukou @kanavin @Eclips4