Skip to content

pkgutil direct use of zipimport._zip_directory_cache is no longer safe #121342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
AdamWill opened this issue Jul 3, 2024 · 0 comments
Open

pkgutil direct use of zipimport._zip_directory_cache is no longer safe #121342

AdamWill opened this issue Jul 3, 2024 · 0 comments
Labels
type-bug An unexpected behavior, bug, or error

Comments

@AdamWill
Copy link
Contributor

AdamWill commented Jul 3, 2024 •
edited by bedevere-app bot
Loading

Bug report

Bug description:

pkgutil directly uses zipimport._zip_directory_cache, in this block:

       def iter_zipimport_modules(importer, prefix=''):
           dirlist = sorted(zipimport._zip_directory_cache[importer.archive])

since #103208 , this is no longer safe, because that made zipimport no longer always refresh the cache on invalidation. I think that if the cache has been invalidated but not accessed 'normally' since then, this direct access attempt can cause a crash as it doesn't trigger a refresh:

E           KeyError: '/tmp/dask-scratch-space/scheduler-belyj1dp/myfile.zip'

I observed this in the test suite for dask-distributed - see dask/distributed#8708 - but it seems like a general issue, and @desmondcheongzx asked me to file an issue, so here it is. That test suite hits it in a test that runs async and creates archives under the same name repeatedly, but there are probably other ways to hit this, I guess.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs
@AdamWill AdamWill added the type-bug An unexpected behavior, bug, or error label Jul 3, 2024
beachmachine added a commit to beachmachine/cpython that referenced this issue Jul 13, 2024
@beachmachine
pythongh-121342: Fixed pkgutil.iter_zipimport_modules on an invalid…
2b4efef 
…ated cache (python#121342)

It is no longer safe to directly access `zipimport._zip_directory_cache` since python#103208. It is
not guaranteed that the cache is acutally filled. This changed fixes this by using the
internal method `_get_files` instead, which may not be the best solution, but fixes the issue.
beachmachine added a commit to beachmachine/cpython that referenced this issue Jul 13, 2024
@beachmachine
pythongh-121342: Fixed pkgutil.iter_zipimport_modules on an invalid…
d244b82 
…ated cache (python#121342)

It is no longer safe to directly access `zipimport._zip_directory_cache` since python#103208. It is
not guaranteed that the cache is acutally filled. This changed fixes this by using the
internal method `_get_files` instead, which may not be the best solution, but fixes the issue.
@bedevere-app bedevere-app bot mentioned this issue Jul 13, 2024
Open
beachmachine added a commit to beachmachine/cpython that referenced this issue Jul 13, 2024
@beachmachine
pythongh-121342: Fixed pkgutil.iter_zipimport_modules on an invalid…
35bc2ba 
…ated cache (python#121342)

It is no longer safe to directly access `zipimport._zip_directory_cache` since python#103208. It is
not guaranteed that the cache is acutally filled. This changed fixes this by using the
internal method `_get_files` instead, which may not be the best solution, but fixes the issue.
beachmachine added a commit to beachmachine/cpython that referenced this issue Jul 13, 2024
@beachmachine
pythongh-121342: Fixed pkgutil.iter_zipimport_modules on an invalid…
7e655ab 
…ated cache (python#121342)

It is no longer safe to directly access `zipimport._zip_directory_cache` since python#103208. It is
not guaranteed that the cache is acutally filled. This changed fixes this by using the
internal method `_get_files` instead, which may not be the best solution, but fixes the issue.
beachmachine added a commit to beachmachine/cpython that referenced this issue Jul 14, 2024
@beachmachine
pythongh-121342: Fixed pkgutil.iter_zipimport_modules on an invalid…
1e52c23 
…ated cache (python#121342)

It is no longer safe to directly access `zipimport._zip_directory_cache` since python#103208. It is
not guaranteed that the cache is acutally filled. This changed fixes this by using the
internal method `_get_files` instead, which may not be the best solution, but fixes the issue.
beachmachine added a commit to beachmachine/cpython that referenced this issue Jul 14, 2024
@beachmachine
pythongh-121342: Unreferenced pkgutil.iter_zipimport_modules in NEWS.d
bc9c996 
As `pkgutil.iter_zipimport_modules` isn't supposed to be consumed directly
by anything other than `pkgutils` itself, and is registered via as an
importer module for `zipimporter`, I've decided to not include it in the
public documentation. Thus, the `:func:` reference
of `pkgutil.iter_zipimport_modules` in the news file is removed now.
beachmachine added a commit to beachmachine/cpython that referenced this issue Jul 14, 2024
@beachmachine
pythongh-121342: Make the Sphinx linter happy again
7d2a3fb
@jrbourbeau jrbourbeau mentioned this issue Nov 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@AdamWill