[pypi] Support pulling dependencies using direct url specifiers without using an index #2363
Comments
|
Please see the documentation here: https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#bazel-downloader-and-multi-platform-wheel-hub-repository. This currently has a limitation that it needs hashes for all of the files in the |
|
@aignas How can I use |
|
The support for direct references has not been added yet, I'll reopen this issue and rename it to scope it to supporting it, but I am not sure when I can work on it.
For the second one it may be harder - I am not sure how to handle git repos yet, however, with bazel 7.4 the extension can pass a labels within the extension more easily, so one could look into how to do that. For now the workaround would be to build the wheels from |
aignas
changed the title
rctx.download()
aignas
changed the title
This PR adds support for installing wheels via direct urls in the requirements lock file: ``` foo==0.0.1 @ https://someurl.org/package.whl bar==0.0.1 @ https://someurl.org/package.tar.gz ``` This is to improve parity between bazel downloader and pip behavior. Before this change, direct urls used fallback to pip install. Partially addresses #2363 as it does not add support for git urls.
…ied (#2695) This pull request modifies the SimpleAPI HTML parsing to add a new field where we can get the `sha256` values by package version. This allows us to very easily fallback to all packages of a particular version when using `experimental_index_url` if the hashes are not specified. The code deciding which packages to query the SimpleAPI for has been also modified to only omit queries for packages that are included via direct URL references. If we fail to get the data from the SimpleAPI, we will fallback to `pip` and try to install it via the legacy behaviour. Fixes #2023 Work towards #260 Work towards #1357 Work towards #2363 --------- Co-authored-by: Ignas Anikevicius <[email protected]>
Whilst integrating bazel-contrib#2695 I introduced a regression and here I add a test for that and fix it. The code that was getting the filename from the URL was too eager and would break if there was a git ref as noted in the test. Before this commit and bazel-contrib#2695 the code was not handling all of the cases that are tested now either, so I think now we are in a good place. I am not sure how we should handle the `git_repository` URLs. Maybe having `http_archive` and `git_repository` usage would be nice, but I am not sure how we can introduce it at the moment. Work towards bazel-contrib#2363
Whilst integrating #2695 I introduced a regression and here I add a test for that and fix it. The code that was getting the filename from the URL was too eager and would break if there was a git ref as noted in the test. Before this commit and #2695 the code was not handling all of the cases that are tested now either, so I think now we are in a good place. I am not sure how we should handle the `git_repository` URLs. Maybe having `http_archive` and `git_repository` usage would be nice, but I am not sure how we can introduce it at the moment. Work towards #2363
|
As @mering suggests the However, we should support I'd like to mark this as done and add a note to the |
|
@aignas thanks for fixing the direct download case 1. When we applied this to our code base, we noted that it still tries to reach out to an index-like URL. Could this be skipped in that case? I am fine with continuing the conversation about the git repository in the other issue. |
Summary: - Better handle git references for sdists. - Better handle direct whl references. - Add an extra test that turned out to be not needed in the end, but I left it to increase the code coverage. Work towards bazel-contrib#2363 Fixes bazel-contrib#2828
Summary: - Make the requirement line the same as the one that is used in whls. It only contains extras and the version if it is present. - Add debug log statements if we fail to get the version from a direct URL reference. - Move some tests from `parse_requirements_tests` to `index_sources_tests` to improve test maintenance. - Replace the URL encoded `+` to a regular `+` in the filename. - Correctly handle the case when the `=sha256:` is used in the URL. I cannot think of anything else that we can do for this as of now, so will mark the associated issue as resolved. Fixes bazel-contrib#2363
|
Spent more time on this, could you please check if the linked PR fixes the remaining issues? |
…ust (#2871) Summary: - Make the requirement line the same as the one that is used in whls. It only contains extras and the version if it is present. - Add debug log statements if we fail to get the version from a direct URL reference. - Move some tests from `parse_requirements_tests` to `index_sources_tests` to improve test maintenance. - Replace the URL encoded `+` to a regular `+` in the filename. - Correctly handle the case when the `=sha256:` is used in the URL. Once this is merged I plan to tackle #2648 by changing the `parse_requirements` code to de-duplicate entries returned by the `parse_requirements` function. I cannot think of anything else that we can do for this as of now, so will mark the associated issue as resolved. Fixes #2363 Work towards #2648
When given .whl file URLs and no file name, `whl_library` writes the wheel it downloads to a file with the same file name as the first URL's. But then at extraction time, it always consults ctx.attr.filename for that file name, leading to failure when that attribute is None. This patch should fix that. Related #2363
🐞 bug report
Affected Rule
Module extension:
Is this a regression?
Not sure.
Description
The repository rule for pip dependencies doesn't use
repository_ctx.download(). This breaks usages of--experimental_downloader_configfor example to use a transparent dependency mirror.🔬 Minimal Reproduction
https://github.com/bazelbuild/rules_python/blob/b6fc2a080d4884510dea02cc77b4e0b8fd3a0ccd/python/private/pypi/whl_library.bzl#L245-L256
https://github.com/bazelbuild/rules_python/blob/b6fc2a080d4884510dea02cc77b4e0b8fd3a0ccd/python/private/pypi/whl_installer/wheel_installer.py#L168-L190
🔥 Exception or Error
Stack trace prints to the code linked above.
🌍 Your Environment
Operating System:
Linux
Output of
bazel version:7.0.2
Rules_python version:
0.37.2
Anything else relevant?
The text was updated successfully, but these errors were encountered: