Implement POST /api/v1/relay (#3425)

* Implement POST /api/v1/relay

PBENCH-1143

This API integrates with the file-relay server by assuming that the client
(e.g., the `pbench-results-move` command) creates *two* separate files on the
relay. The main artifact is the tarball, identified only by the tarball's
computed SHA256 hash, as `http://relay.example.com/<name>/<sha256>`. However
the Pbench Server needs the MD5 for the formal `resource_id` and a *name* for
the dataset before we can begin "intake". We also want a mechanism to override
the default "private" access level, and to set metadata, so by convention we
add a second JSON file to which I'm referring as the "Relay configuration
file". This provides the dataset "metadata" we require, including the tarball
URI:

```json
{
  "name": "tarball.tar.xz",
  "md5": "myMD5",
  "access": "private",
  "metadata": ["server.origin:relay", "global.server.relay:webb"],
}
```

The Pbench Server will first `GET` the URI parameter, which it expects to be
the Relay Configuration File. It will then `GET` the `"uri"` parameter in the
configuration file.

To accomplish this, the old `Upload` code has been refactored into a base
class defining most of the logic, and two subclasses which provide "helper"
methods called `_prepare` and `_access` to encapsulate the operational
differences between the "push" and "pull" models. Essentially the `Upload`
class `_prepare` processes the URI and query parameters while the `Relay`
class `_prepare` does the first `GET` and processes the JSON configuration
parameters. The `_access` helpers return the content length and an `IO[byte]`
stream which is read and processed by the common code.

This has been tested manually by standing up an instance of the `relay` server
and I added unit tests to cover the distinct functions of the relay helper
methods while not duplicating the `upload` unit test cases that cover the
common code.

Author: dbutenhof

lib/pbench/client/__init__.py

@@ -49,6 +49,7 @@ class API(Enum):
     DATASETS_VALUES = "datasets_values"
     ENDPOINTS = "endpoints"
     KEY = "key"
+    RELAY = "relay"
     SERVER_AUDIT = "server_audit"
     SERVER_SETTINGS = "server_settings"
     UPLOAD = "upload"

lib/pbench/server/api/__init__.py

@@ -33,9 +33,10 @@
     SampleValues,
 )
 from pbench.server.api.resources.query_apis.datasets_search import DatasetsSearch
+from pbench.server.api.resources.relay import Relay
 from pbench.server.api.resources.server_audit import ServerAudit
 from pbench.server.api.resources.server_settings import ServerSettings
-from pbench.server.api.resources.upload_api import Upload
+from pbench.server.api.resources.upload import Upload
 import pbench.server.auth.auth as Auth
 from pbench.server.database import init_db
 from pbench.server.database.database import Database
@@ -146,6 +147,12 @@ def register_endpoints(api: Api, app: Flask, config: PbenchServerConfig):
         endpoint="server_settings",
         resource_class_args=(config,),
     )
+    api.add_resource(
+        Relay,
+        f"{base_uri}/relay/<path:uri>",
+        endpoint="relay",
+        resource_class_args=(config,),
+    )
     api.add_resource(
         Upload,
         f"{base_uri}/upload/<string:filename>",