Skip to content

TraversalError in MultiplexedPath.joinpath when parent in compound path is missing #253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MattiasDC opened this issue Jul 1, 2022 · 18 comments · Fixed by #254
Closed
Assignees

Comments

@MattiasDC
Copy link

MattiasDC commented Jul 1, 2022

I have a regression between version 5.7.1 and 5.8.0. I think the cause is this commit.

I have a reproduction scenario in the zip.
To reproduce:

  • unzip repro.zip
  • cd repro
  • pip install -e .
  • python -m test.test --info

If you change the setup.cfg to use version 5.7.1, no error is raised
repro.zip

@MattiasDC
Copy link
Author

MattiasDC commented Jul 1, 2022

The exception callstack that should occur:

Traceback (most recent call last):
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/importlib_resources/abc.py", line 119, in joinpath
    match = next(matches)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/MDC/git/repro/test/test.py", line 8, in <module>
    test()
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/main.py", line 49, in decorated_main
    _run_hydra(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/utils.py", line 399, in _run_hydra
    hydra.show_info(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 655, in show_info
    options[info](config_name=config_name, overrides=overrides)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 588, in _print_all_info
    self._print_config_info(config_name, overrides)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 470, in _print_config_info
    self._print_search_path(config_name=config_name, overrides=overrides)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 403, in _print_search_path
    cfg = self._get_cfg(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 163, in _get_cfg
    cfg = self.compose_config(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 564, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 146, in load_configuration
    return self._load_configuration_impl(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 239, in _load_configuration_impl
    defaults_list = create_defaults_list(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 719, in create_defaults_list
    defaults, tree = _create_defaults_list(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 689, in _create_defaults_list
    defaults_tree = _create_defaults_tree(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 337, in _create_defaults_tree
    ret = _create_defaults_tree_impl(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 420, in _create_defaults_tree_impl
    return _expand_virtual_root(repo, root, overrides, skip_missing)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 262, in _expand_virtual_root
    subtree = _create_defaults_tree_impl(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 429, in _create_defaults_tree_impl
    update_package_header(repo=repo, node=parent)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 244, in update_package_header
    loaded = repo.load_config(config_path=node.get_config_path())
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 337, in load_config
    ret = self.delegate.load_config(config_path=config_path)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 86, in load_config
    source = self._find_object_source(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 130, in _find_object_source
    if source.is_config(config_path):
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/core_plugins/importlib_resources_config_source.py", line 91, in is_config
    res = files.joinpath(config_path)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/importlib_resources/readers.py", line 87, in joinpath
    return super().joinpath(*descendants)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/importlib_resources/abc.py", line 121, in joinpath
    raise TraversalError(
importlib_resources.abc.TraversalError: ('Target not found during traversal.', 'hydra', ['config.yaml'])

@MattiasDC
Copy link
Author

MattiasDC commented Jul 1, 2022

Notice that I would've expected that this exception would be catched by this frame:
File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/importlib_resources/readers.py", line 87, in joinpath

As per:
image
Am I missing something obvious here?

@MattiasDC
Copy link
Author

Hi @jaraco, is this something you could have a look at as you recently committed in this code?

@jaraco jaraco self-assigned this Jul 2, 2022
@jaraco
Copy link
Member

jaraco commented Jul 2, 2022

Yes. I'll take a look. As described, it's difficult for me to see what's going on, so I'll need to download and unzip the repro to investigate deeper.

@jaraco
Copy link
Member

jaraco commented Jul 3, 2022

I've pushed the repro to the issue-253 branch for easier testing.

@jaraco
Copy link
Member

jaraco commented Jul 3, 2022

It works fine for me:

 repro issue-253 $ py -3.10 -m venv .venv
 repro issue-253 $ py -m pip install -q -U pip
 repro issue-253 $ py -m pip install -e .
Obtaining file:///Users/jaraco/draft/repro
  Preparing metadata (setup.py) ... done
Collecting hydra-core==1.1.0
  Using cached hydra_core-1.1.0-py3-none-any.whl (144 kB)
Collecting hydra-joblib-launcher==1.1.5
  Using cached hydra_joblib_launcher-1.1.5-py3-none-any.whl (5.2 kB)
Collecting importlib_resources==5.8.0
  Using cached importlib_resources-5.8.0-py3-none-any.whl (28 kB)
Collecting omegaconf==2.1.*
  Using cached omegaconf-2.1.2-py3-none-any.whl (74 kB)
Collecting antlr4-python3-runtime==4.8
  Using cached antlr4_python3_runtime-4.8-py3-none-any.whl
Collecting joblib>=0.14.0
  Using cached joblib-1.1.0-py2.py3-none-any.whl (306 kB)
Collecting PyYAML>=5.1.0
  Using cached PyYAML-6.0-cp310-cp310-macosx_11_0_arm64.whl (173 kB)
Installing collected packages: antlr4-python3-runtime, PyYAML, joblib, importlib_resources, omegaconf, hydra-core, hydra-joblib-launcher, test
  Running setup.py develop for test
Successfully installed PyYAML-6.0 antlr4-python3-runtime-4.8 hydra-core-1.1.0 hydra-joblib-launcher-1.1.5 importlib_resources-5.8.0 joblib-1.1.0 omegaconf-2.1.2 test-0.0.0
 repro issue-253 $ py -m test.test --info
Hydra 1.1.0
===========
Installed Hydra Plugins
***********************
        ConfigSource:
        -------------
                FileConfigSource
                ImportlibResourcesConfigSource
                StructuredConfigSource
        CompletionPlugin:
        -----------------
                BashCompletion
                FishCompletion
                ZshCompletion
        Launcher:
        ---------
                BasicLauncher
                JoblibLauncher
        Sweeper:
        --------
                BasicSweeper
        SearchPathPlugin:
        -----------------
                Dummy

Config search path
******************
| Provider                 | Search path                           |
--------------------------------------------------------------------
| hydra                    | pkg://hydra.conf                      |
| main                     | file:///Users/jaraco/draft/repro/test |
| dummy-search-path-plugin | pkg://test.config                     |
| schema                   | structured://                         |
--------------------------------------------------------------------

Defaults Tree
*************
<root>:
  hydra/config:
    hydra/output: default
    hydra/launcher: joblib
    hydra/sweeper: basic
    hydra/help: default
    hydra/hydra_help: default
    hydra/hydra_logging: default
    hydra/job_logging: default
    hydra/callbacks: null
    hydra/env: default
    _self_
  run

Defaults List
*************
| Config path                 | Package             | _self_ | Parent       | 
------------------------------------------------------------------------------
| hydra/output/default        | hydra               | False  | hydra/config |
| hydra/launcher/joblib       | hydra.launcher      | False  | hydra/config |
| hydra/sweeper/basic         | hydra.sweeper       | False  | hydra/config |
| hydra/help/default          | hydra.help          | False  | hydra/config |
| hydra/hydra_help/default    | hydra.hydra_help    | False  | hydra/config |
| hydra/hydra_logging/default | hydra.hydra_logging | False  | hydra/config |
| hydra/job_logging/default   | hydra.job_logging   | False  | hydra/config |
| hydra/env/default           | hydra.env           | False  | hydra/config |
| hydra/config                | hydra               | True   | <root>       |
| run                         |                     | False  | <root>       |
------------------------------------------------------------------------------
Config
******
{}

@jaraco
Copy link
Member

jaraco commented Jul 3, 2022

Aha. So the issue seems only to arise only on older versions of Python.

 repro issue-253 $ py -3.8 -m venv .venv
 repro issue-253 $ py -m pip install -e .
Obtaining file:///Users/jaraco/draft/repro
  Preparing metadata (setup.py) ... done
Collecting hydra-core==1.1.0
  Using cached hydra_core-1.1.0-py3-none-any.whl (144 kB)
Collecting hydra-joblib-launcher==1.1.5
  Using cached hydra_joblib_launcher-1.1.5-py3-none-any.whl (5.2 kB)
Collecting importlib_resources==5.8.0
  Using cached importlib_resources-5.8.0-py3-none-any.whl (28 kB)
Collecting omegaconf==2.1.*
  Using cached omegaconf-2.1.2-py3-none-any.whl (74 kB)
Collecting antlr4-python3-runtime==4.8
  Using cached antlr4-python3-runtime-4.8.tar.gz (112 kB)
  Preparing metadata (setup.py) ... done
Collecting joblib>=0.14.0
  Using cached joblib-1.1.0-py2.py3-none-any.whl (306 kB)
Collecting zipp>=3.1.0
  Using cached zipp-3.8.0-py3-none-any.whl (5.4 kB)
Collecting PyYAML>=5.1.0
  Using cached PyYAML-6.0.tar.gz (124 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Using legacy 'setup.py install' for antlr4-python3-runtime, since package 'wheel' is not installed.
Building wheels for collected packages: PyYAML
  Building wheel for PyYAML (pyproject.toml) ... done
  Created wheel for PyYAML: filename=PyYAML-6.0-cp38-cp38-macosx_12_0_arm64.whl size=45335 sha256=e703e181e9d6429ca0b78de2c2229d95d127275b99c98a5711870bde22f1ac0c
  Stored in directory: /Users/jaraco/Library/Caches/pip/wheels/52/84/66/50912fd7bf1639a31758e40bd4312602e104a8eca1e0da9645
Successfully built PyYAML
Installing collected packages: antlr4-python3-runtime, zipp, PyYAML, joblib, omegaconf, importlib_resources, hydra-core, hydra-joblib-launcher, test
  Running setup.py install for antlr4-python3-runtime ... done
  Running setup.py develop for test
Successfully installed PyYAML-6.0 antlr4-python3-runtime-4.8 hydra-core-1.1.0 hydra-joblib-launcher-1.1.5 importlib_resources-5.8.0 joblib-1.1.0 omegaconf-2.1.2 test-0.0.0 zipp-3.8.0
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/Users/jaraco/draft/repro/.venv/bin/python -m pip install --upgrade pip' command.
 repro issue-253 $ py -m test.test --info
Hydra 1.1.0
===========
Installed Hydra Plugins
***********************
        ConfigSource:
        -------------
                FileConfigSource
                ImportlibResourcesConfigSource
                StructuredConfigSource
        CompletionPlugin:
        -----------------
                BashCompletion
                FishCompletion
                ZshCompletion
        Launcher:
        ---------
                BasicLauncher
                JoblibLauncher
        Sweeper:
        --------
                BasicSweeper
        SearchPathPlugin:
        -----------------
                Dummy

Config search path
******************
Traceback (most recent call last):
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/abc.py", line 119, in joinpath
    match = next(matches)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.8.13_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/homebrew/Cellar/[email protected]/3.8.13_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/jaraco/draft/repro/test/test.py", line 8, in <module>
    test()
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/main.py", line 49, in decorated_main
    _run_hydra(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 399, in _run_hydra
    hydra.show_info(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 655, in show_info
    options[info](config_name=config_name, overrides=overrides)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 588, in _print_all_info
    self._print_config_info(config_name, overrides)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 470, in _print_config_info
    self._print_search_path(config_name=config_name, overrides=overrides)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 403, in _print_search_path
    cfg = self._get_cfg(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 163, in _get_cfg
    cfg = self.compose_config(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 564, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 146, in load_configuration
    return self._load_configuration_impl(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 239, in _load_configuration_impl
    defaults_list = create_defaults_list(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 719, in create_defaults_list
    defaults, tree = _create_defaults_list(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 689, in _create_defaults_list
    defaults_tree = _create_defaults_tree(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 337, in _create_defaults_tree
    ret = _create_defaults_tree_impl(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 420, in _create_defaults_tree_impl
    return _expand_virtual_root(repo, root, overrides, skip_missing)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 262, in _expand_virtual_root
    subtree = _create_defaults_tree_impl(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 429, in _create_defaults_tree_impl
    update_package_header(repo=repo, node=parent)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 244, in update_package_header
    loaded = repo.load_config(config_path=node.get_config_path())
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 337, in load_config
    ret = self.delegate.load_config(config_path=config_path)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 86, in load_config
    source = self._find_object_source(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 130, in _find_object_source
    if source.is_config(config_path):
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/core_plugins/importlib_resources_config_source.py", line 91, in is_config
    res = files.joinpath(config_path)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/readers.py", line 87, in joinpath
    return super().joinpath(*descendants)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/abc.py", line 121, in joinpath
    raise TraversalError(
importlib_resources.abc.TraversalError: ('Target not found during traversal.', 'hydra', ['config.yaml'])

@jaraco
Copy link
Member

jaraco commented Jul 3, 2022

A bit of debugging:

 repro issue-253 $ py -m pdb -m test.test --info
> /Users/jaraco/draft/repro/test/test.py(1)<module>()
-> import hydra
(Pdb) c
Hydra 1.1.0
===========
Installed Hydra Plugins
***********************
        ConfigSource:
        -------------
                FileConfigSource
                ImportlibResourcesConfigSource
                StructuredConfigSource
        CompletionPlugin:
        -----------------
                BashCompletion
                FishCompletion
                ZshCompletion
        Launcher:
        ---------
                BasicLauncher
                JoblibLauncher
        Sweeper:
        --------
                BasicSweeper
        SearchPathPlugin:
        -----------------
                Dummy

Config search path
******************
Traceback (most recent call last):
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/abc.py", line 119, in joinpath
    match = next(matches)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.8.13_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py", line 1703, in main
    pdb._runmodule(mainpyfile)
  File "/opt/homebrew/Cellar/[email protected]/3.8.13_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py", line 1547, in _runmodule
    self.run(code)
  File "/opt/homebrew/Cellar/[email protected]/3.8.13_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "/Users/jaraco/draft/repro/test/test.py", line 1, in <module>
    import hydra
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/main.py", line 49, in decorated_main
    _run_hydra(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 399, in _run_hydra
    hydra.show_info(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 655, in show_info
    options[info](config_name=config_name, overrides=overrides)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 588, in _print_all_info
    self._print_config_info(config_name, overrides)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 470, in _print_config_info
    self._print_search_path(config_name=config_name, overrides=overrides)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 403, in _print_search_path
    cfg = self._get_cfg(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 163, in _get_cfg
    cfg = self.compose_config(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 564, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 146, in load_configuration
    return self._load_configuration_impl(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 239, in _load_configuration_impl
    defaults_list = create_defaults_list(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 719, in create_defaults_list
    defaults, tree = _create_defaults_list(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 689, in _create_defaults_list
    defaults_tree = _create_defaults_tree(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 337, in _create_defaults_tree
    ret = _create_defaults_tree_impl(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 420, in _create_defaults_tree_impl
    return _expand_virtual_root(repo, root, overrides, skip_missing)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 262, in _expand_virtual_root
    subtree = _create_defaults_tree_impl(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 429, in _create_defaults_tree_impl
    update_package_header(repo=repo, node=parent)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 244, in update_package_header
    loaded = repo.load_config(config_path=node.get_config_path())
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 337, in load_config
    ret = self.delegate.load_config(config_path=config_path)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 86, in load_config
    source = self._find_object_source(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 130, in _find_object_source
    if source.is_config(config_path):
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/core_plugins/importlib_resources_config_source.py", line 91, in is_config
    res = files.joinpath(config_path)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/readers.py", line 87, in joinpath
    return super().joinpath(*descendants)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/abc.py", line 121, in joinpath
    raise TraversalError(
importlib_resources.abc.TraversalError: ('Target not found during traversal.', 'hydra', ['config.yaml'])
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/abc.py(121)joinpath()
-> raise TraversalError(
(Pdb) up 2
> /Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/core_plugins/importlib_resources_config_source.py(91)is_config()
-> res = files.joinpath(config_path)
(Pdb) files
MultiplexedPath('/Users/jaraco/draft/repro/test/config')
(Pdb) config_path
'hydra/config.yaml'
(Pdb) d
> /Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/readers.py(87)joinpath()
-> return super().joinpath(*descendants)
(Pdb) descendants
('hydra/config.yaml',)
(Pdb) d
> /Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/abc.py(121)joinpath()
-> raise TraversalError(
(Pdb) l
116                 traversable for traversable in self.iterdir() if traversable.name == target
117             )
118             try:
119                 match = next(matches)
120             except StopIteration:
121  ->             raise TraversalError(
122                     "Target not found during traversal.", target, list(names)
123                 )
124             return match.joinpath(*names)
125  
126         def __truediv__(self, child: StrPath) -> "Traversable":
(Pdb) !list(self.iterdir())
[PosixPath('/Users/jaraco/draft/repro/test/config/run.yaml')]
(Pdb) target
'hydra'

It appears from the code that hydra is looking for hydra/config.yaml in /Users/jaraco/draft/repro/test/config and because hydra doesn't exist in test/config, the TraversalError is raised.

@jaraco
Copy link
Member

jaraco commented Jul 3, 2022

Looking at the calling code, it does appear as if the code is designed to trap exceptions when importlib_resources.files() is called, but not when .joinpath() is called.

Perhaps the refactoring that happened in 5.8.0 changed the behavior around when an exception is raised.

@MattiasDC
Copy link
Author

Hi @jaraco, but shouldn´t this code catch already?

@jaraco
Copy link
Member

jaraco commented Jul 3, 2022

Here's what changed from 5.7.1 to 5.8.0.

@MattiasDC
Copy link
Author

Hi @jaraco, but shouldn´t this code catch already?

Ah, but that code can reraise..

@MattiasDC
Copy link
Author

Can you confirm that this re-raise, which is new, is as intended?
If that's the case, this issue can be closed and I'll log an issue on hydra instead.

@jaraco
Copy link
Member

jaraco commented Jul 3, 2022

Aha. I see the issue. Previously, MultiplexedPath.joinpath(), when passed a compound child path (like "hydra/config.yaml"), would incorrectly assume that the child was not a compound path but instead was a simple filename, so would join it to the first path (i.e. test/config/hydra/config.yaml) not raising an Exception and since that doesn't .exist(), the hydra code could handle it fine.

With 5.8 (#250 in particular), however, joinpath is more assertive about resolving parents of any path. If a parent (in this case test/config/hydra doesn't exist, it cannot continue traversal, so raises an exception.

Can you confirm that this re-raise, which is new, is as intended? If that's the case, this issue can be closed and I'll log an issue on hydra instead.

The change was intended, but it wasn't necessarily intended that errors would be raised on joinpath where they weren't before. I need to consider whether it's possible and worthwhile to retain the prior, more lenient behavior.

To be sure Hydra could work around the issue by simply moving the joinpath() up into the try block, but I don't want to recommend that until I'm certain that importlib_resources shouldn't address this issue.

@jaraco
Copy link
Member

jaraco commented Jul 3, 2022

I also want to figure out if the issue affects Traversables other that MultiplexedPath.

Edit: After review, I believe the issue only affects MultiplexedPath, because the other implementation that changed (simple.ResourceContainer) would have raised a StopIteration under the same conditions.

@jaraco
Copy link
Member

jaraco commented Jul 3, 2022

I've put together a PR that (a) captures the expectation on the prior release, (b) demonstrates that test fails on the current main, and (c) fixes the issue by removing the re-raise. I'm not yet certain that's the right approach.

One of the things that makes this behavior interesting is that MultiplexedPath is being used at all. The purpose behind MultiplexedPath is to facilitate resources that exist in namespace packages (packages whose implementation and resources can be found in multiple directories), but in this case, the multiplexed path (and thus the namespace path) appears to be for test.config, which is a directory inside a package.

I'm uncertain whether this usage (either by hydra or the repro) is correct. What does it mean, for example, when path="pkg://test/config" here?

Does pkg://test/config mean the test.config Python package? If so, that seems incorrect, because config is not a Python package, and in that case, adding a test/config/__init__.py may correct the underlying expectation as well as workaround the error encountered. See ac61cce, where the repro no longer fails.

On the other hand, if pkg://test/config means "the 'config' directory found in the 'test' package", then hydra isn't using importlib.resources properly. To get resources from there, the correct usage is files('test').joinpath('config'), which produces a PosixPath and thus won't be subject to the unexpected behavior in MultiplexedPath.

Regardless, I believe you've identified a real potential defect here and if FFY00 agrees in the review of the PR, this will be fixed in 5.8.1 (and pertinent Pythons where 5.8 was merged). I urge you to determine if hydra or the repro are doing something incorrect that can be improved.

@jaraco jaraco changed the title TraversalError thrown when it should have been catched? TraversalError in MultiplexedPath.joinpath when parent in compound path is missing Jul 3, 2022
@MattiasDC
Copy link
Author

MattiasDC commented Jul 3, 2022

Some additional information. Documentation for hydra search path plugins can be found here. Notice especially this part:

pkg:// points to an importable Python module, with . being the separator. __init__.py files are needed in directories for Python to treat them as packages.

The documentation seems to imply that sometimes you don´t want to treat it as packages though. There is an example provided by Hydra, which can be found here. Notice however that the separator in the example it is not separated by '.', but by '/'. As can be seen here. Furthermore the most nested directory which contains configuration, doesn´t have an __init__.py file. Upper directories do have an __init__.py file.

I will check with someone of the hydra developer team whether they can give some input.

@MattiasDC
Copy link
Author

I asked a question to hydra maintainers here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants