Support limited non-alphanumeric metadata keys #3354
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dbutenhof
added
Server
API
webbnh
previously approved these changes
Mar 20, 2023
Member
webbnh
left a comment
webbnh
left a comment
There was a problem hiding this comment.
Looks OK to me, but...do we guard against metadata keys containing dots, colons, and any other characters that we have special semantics for?...and do we have tests that ensure that we disallow the things we're supposed to disallow? And, I found a typo and I have a code suggestion.
webbnh
previously approved these changes
Mar 20, 2023
PBENCH-1111 Internally defined and managed Pbench Server metadata keys are lowercase alphanumeric strings; however the dataset.metalog namespace is extracted directly from the tarball metadata.log file and some benchmarks (especially fio and uperf generate tool and iteration keys that aren't accessible because they fail the normal key path validation checks. This PR loosens validation checks in some paths so `GET` APIs that retrieve metadata will be able to find these keys, for example uperf iterations like `dataset.metalog.iterations/1-tcp_stream-64B-1i`. (NOTE: I first noticed this problem when I added general metadata filtering, and made special allowances in the `filter` keyword processing, but at the time didn't want to expand the scope to deal with more general handling. I was reminded when I added the key aggregator, and finally I'm fixing it.)
(There could always be more, but I think this covers the basics.)
Some more test cases, but also now recursively validating JSON keys to avoid that loophole.
Confirm that multiple unacceptable JSON keys are collected across various levels of the JSON tree.
siddardh-ra
approved these changes
Mar 22, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PBENCH-1111
Internally defined and managed Pbench Server metadata keys are lowercase alphanumeric strings; however the dataset.metalog namespace is extracted directly from the tarball metadata.log file and some benchmarks (especially fio and uperf generate tool and iteration keys that aren't accessible because they fail the normal key path validation checks.
This PR loosens validation checks in some paths so
GETAPIs that retrieve metadata will be able to find these keys, for example uperf iterations likedataset.metalog.iterations/1-tcp_stream-64B-1i.(NOTE: I first noticed this problem when I added general metadata filtering, and made special allowances in the
filterkeyword processing, but at the time didn't want to expand the scope to deal with more general handling. I was reminded when I added the key aggregator, and finally I'm fixing it.)