webhdfs: implement support for basic auth and proxy #10075
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was already supported according to documentation, but missing in code Fixes: 10062
This key is needed to construct a potential rewrite dictionary in webhdfs to deal with HDFS behind High Availability Proxy servers
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #10075 +/- ##
==========================================
- Coverage 90.54% 90.31% -0.24%
==========================================
Files 499 499
Lines 37950 37950
Branches 5514 5514
==========================================
- Hits 34363 34273 -90
- Misses 2943 3011 +68
- Partials 644 666 +22 ☔ View full report in Codecov by Sentry. |
efiop
changed the title
efiop
previously requested changes
Nov 8, 2023
efiop
changed the title
|
I think we're in some kind of limbo right now. Apparently the changes have been merged into dvc-webhdfs and the documentation has been updated (which now states that we can use "user", "password" and "data_proxy_target" - but the main repo doesn't yet support any of these parameters. 😕 |
skshetry
added a commit
to iterative/dvc-webhdfs
that referenced
this pull request
Dec 26, 2023
BradyJ27
pushed a commit
to BradyJ27/dvc
that referenced
this pull request
Apr 22, 2024
* remote: add user key to config schema webhdfs This was already supported according to documentation, but missing in code Fixes: 10062 * remote: add password key to support Basic Auth * remote: add data_proxy_target for webhdfs remote This key is needed to construct a potential rewrite dictionary in webhdfs to deal with HDFS behind High Availability Proxy servers * bump dvc-webhdfs --------- Co-authored-by: skshetry <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the current documentation of the webhdfs remote it was stated you could provide a
userparameter. However, this parameter was not explicitly defined in theconfig_schema.jsonfile.Since I wanted to add support for Basic Auth for WebHDFS (possible since my PR for fsspec has been merged fsspec/filesystem_spec#1409, but not yet released) I also added support for a
passwordparameter.Finally, for cases where the edgenode is behind a HA proxy, rewriting of the URLs might be needed. This is supported by fsspec using the data_proxy parameter (which can be either a dictionary or a callable). After a talk on discord opted to implement this using only the target of the rewrite instead of providing source and target.
This target (
data_proxy_target) also needed to be added to theconfig_schema.json.The relevant dvc-webhdfs PR for the change is: iterative/dvc-webhdfs#16
The relevant dvc.org PR for doc change is: iterative/dvc.org#4980
The combination of the 3 code changes (fsspec / dvc / dvc-webhdfs) has been running without any errors in the situation where the basic auth + rewrite is required.
Fixes: #10062
❗ I have followed the Contributing to DVC checklist.
📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.
Thank you for the contribution - we'll try to review it as soon as possible. 🙏