updater: abstract out the network IO

[jku]

This might be relevant to Updater redesign (#1135) and if accepted would deprecate #1142 and the PR #1171

We (me, Joshua, Martin, Teodora) have been talking about abstracting some of the client functionality out of the Updater itself. The biggest issue from my perspective is network IO. Teodora already made a PR to let the application download targets but it seems like there are still issues with TUF handling metadata downloads.

Why is this needed?

• In the real world applications are already using a network stack and will be using it after integrating TUF as well: we should not force another one on them
• Even if the network stacks of the application and TUF are same, the fact that they use different sessions and configurations is not great
• Complex applications have legitimate needs to configure a lot of things we don't want to provide API for: user agent, proxies, basic authentication, custom request headers. This applies to both metadata and targets
• Complex applications have legitimate needs to control the download process (e.g. progress information, canceling)
• Complex applications have (legitimate?) needs to poke at low level details like timeouts

Potential solutions

We identified two main solutions to this:

1. Make a new event-based non-blocking client API. This would be most flexible but also more complex for TUF maintainers to maintain and application developers to customize
2. Keep the current API but add a new Fetcher interface that applications can optionally implement. This is likely fairly easy and non-invasive to implement but remains a blocking API

I'm proposing option 2 but for reference please see the draft of option 1 as well.

Proposal

Add a Fetcher interface that applications can implement. Provide a default implementation of Fetcher. Add a new method to Updater that Fetcher can use to provide the data it fetches.

The Updater processes (refresh(), get_one_valid_targetinfo() and download_target()) will now look like this:

• Whenever a remote file (metadata or target) is needed:
  • setup a temporary file to write results to
  • call Fetcher.fetch()
    • fetcher calls Updater.provide_fetched_data() zero or more times to provide chunks of data. Updater writes these chunks into the file
  • when fetcher returns without exceptions, the download is finished and written to the file

This is like the go-tuf RemoteStore abstraction with two differences: 1. Python does not have reasonable stream abstractions like io.ReadCloser (that would actually be implemented by any of the network stacks) so we cannot return something like that: instead our implementation blocks and adds a provide_fetched_data() callback into Updater instead. 2. Metadata and target fetching is not separated: this way the Fetcher does not need any understanding of TUF or server structure, it's just a dumb downloader.

# Only new/changed methods mentioned for Updater
class Updater(object):
    # init now accepts an optional fetcher argument
    def __init__(self, repository_name, repository_mirrors, fetcher: Fetcher = None):

    # Accepts content of the url that is being currently fetched.
    # Can be called only from Fetcher.fetch() that this Updater called.
    def provide_fetched_data(self, data: bytes)

# New interface for applications to implement
class Fetcher(metaclass=abc.ABCMeta):
    # Fetches the contents of HTTP/HTTPS url from a remote server. Calls 
    # self.updater.provide_fetched_data() to forward sequential chunks of
    # bytes to the updater. Returns when the download is complete and all
    # bytes have been fed to updater.
    @abc.abstractmethod
    def fetch(self, url: str, length: int):
        pass

    # Called by updater init
    def set_updater(self, updater: Updater):
        self.updater = updater

I think this is fairly straight-forward to implement even without a client redesign (and will be backwards-compatible). download.py is split into two parts: one part contains the Tempfile handling bits and _check_downloaded_length() and are used by the updater itself; the rest of download.py form the default Fetcher implementation.

[trishankatdatadog]

💯

This actually happened with the Datadog Agent: some of our customers needed to use TLS proxies, and we were using custom networking code back then that simply couldn't handle it. Hence the switch to requests, but as you say, it's not great that we use our own configuration and sessions there...

[jku]

with regards to parallel downloads: Making sure that a fancy custom Fetcher could download multiple urls in parallel seems relatively simple -- the design doesn't have to implement that but it should be taken into account so adding it later would be possible without breaking API.

[lukpueh]

Thanks for sketching out two approaches in such great detail, @jku!

Regarding (2), may I ask why Updater.fetcher.fetch()calls back into Updater.fetcher.updater.provide_fetched_data()? And where is fetch() called in the first place?

Couldn't we just make fetch return the data directly to the caller (i.e. some updater method). I (naively) picture this similar to how we use securesystemslib.storage.StorageBackendInterface-methods.

Either way, and as @joshuagl has mentioned in #1142 (comment), if we're blocked until the I/O provider gives us all what they fetched, then we can not protect against endless data or slow retrieval attacks (see #932).

Maybe we could tell our users that, if they give us a blocking fetch they have to protect against those themselves. Or they give us a non-blocking fetch that allows us to incrementally receive data and perform checks on size and rate constraints.

[jku]

Regarding (2), may I ask why Updater.fetcher.fetch()calls back into Updater.fetcher.updater.provide_fetched_data()? And where is fetch() called in the first place?

fetch is called in the same two places in updater that now call tuf.download.*safe_download().

The issues that provide_fetched_data() tries to solve are:

• we don't want to return an array of bytes from fetch() because in the case of targets that could be very large
• as far as I can tell in python there's no nice commonly supported "stream" abstraction that fetch could immediately return in python like the go-tuf version does (in e.g. requests the only thing that's close is requests.Response.raw and it's a file object but ... a little rough to use. In aiohttp the comparable object is a aiohttp.StreamReader which looks fine... but are these two equivalent 🤷 ?)
• even if fetch() could return a stream/fileobject, one of the goals of the design is that the application should be able to do things like progress notifications: so the application has to be able to process the stream "as it flows" as well anyway

So as a compromise the design requires fetch() to deliver the bytes in chunks by calling provide_fetched_data() multiple times before returning from fetch() -- I think this leads to simplest Fetcher and simplest Updater implementations: this way

• large files are never fully in memory
• we don't make assumptions about the implementation of Fetcher: any http stack should be able to return bytes in chunks (and the default implementation is already 99% done in tuf.download)
• fetch() will indeed block but tuf can still do the endless data and slow retrieval protections (if we figure out how to do them) because Updater.provide_fetched_data() runs regularly -- with the caveat that individual chunk download and the chunk size is in Fetcher control
• Also, we can still make sure that the data is only processed once (by updating the hash digest in provide_fetched_data() before writing data to the tempfile) so this should still allow us to be as efficient as we want to be

Does that explain my thinking?

[lukpueh]

Thanks a ton for the clarification, @jku! This is a pretty smart solution. I am a bit concerned about the coupling in this design, but that may be ill-founded. Here is what I'm concerned about:

1. It may not be obvious to implementers of fetch that they need to call an unrelated provide_fetched_data.
2. provide_fetched_data has a side-effect, i.e. write to file, which depends on state managed two function calls earlier in the stack, i.e. when our updater code calls fetch we need to create a temporary file and let provide_fetched_data know that it shall henceforth write to that file until fetch returns to us and we close and persist.

What if instead of writing to a function we asked fetch to write to a passed tmp file pointer? This might be a slightly more obvious usage pattern to the fetch interface implementor and we get built-in enter/exit handling...

with tempfile.SpooledTmpFile() as fp_for_fetched_data: # use spooled tmp file to indeed unburden memory
  self.fetcher.fetch(url, length, fp_for_fetched_data) # fetch must write to passed file
  # Persist file once fetch has returned...

And to protect against endless data and slow retrieval protection, we can pass a custom tmp file object with an augmented write method that adds length and rate check. The file already knows how much has been written so far (endless data) and it could easily be extended to know when we started writing (slow retrieval).

What do you think?

[jku]

What if instead of writing to a function we asked fetch to write to a passed tmp file pointer?

I like it, it has advantages (and does not require Fetcher to figure out what a "file object" actually means).

The only dislike is that it still leaves a "file object" in the API and python file objects seem to defy definition... I know this is probably not a pythonic way to think but I just do not understand what is the actual interface that we promise to implement there. A non-seekable, non-readable io.RawIOBase maybe? or just "any object that has a write(bytes) method"?

Anyway, I can get over that ambiguity: passing a file object from updater to fetcher looks like a reasonable solution

[lukpueh]

You are right, I didn't think about "hiding/protecting" the file object from the fetch implementer. But I guess it's in their best interest to not do anything other than write(bytes) akin to what they'd do with your provide_fetched_data.

[trishankatdatadog]

What guarantees the Fetcher would only call provide_fetched_data()on the Updater, and nothing else?

[lukpueh]

Good question, @trishankatdatadog. In my proposal the fetcher doesn't need a reference to the updater object.