[Epic] Add support for the Bitswap transport

[jacobheun]

Summary

Storage Providers should be able to serve requests to the IPFS network as several storage clients, such as web3.storage, Estuary, and IPFS pinning services, and their end users are part of the IPFS network. Adding support for Bitswap removes the need for intermediary services, such as autoretrieve. Leveraging the flexibility of Boost we should enable this as a standalone service, similar to booster-http, so that it can be independently scaled as needed.

As part of this effort we are working with the Kubo and IPFS.io gateway teams to enable e2e tracing of the retrieval pipeline as part of our efforts to improve retrieval reliability. This will ensure we have visibility into each component in the retrieval pipeline to better resolve reliability and performance issues. The metrics/logging added for Boost will also enable us to troubleshoot retrieval problems seen by other retrieval transports (Graphsync and HTTP) at lower layers of the Boost code base (DAG store, piece store, etc)

Milestones

1. E2E Integration with Kubo for free, public content

With the completion of this milestone Kubo will be able to query Network Indexers, such as cid.contact, for a given CID and discover SP's who are running the Bitswap transport.

Requirements

• SP's must be able to restrict access to content via existing retrieval access patterns
• Only 0 cost retrieval will initially be supported
• Bitswap content records should be discoverable on network indexers, such as cid.contact

Issue tracking

Enabling the Bitswap retrieval server

• Add support for a standalone Bitswap retrieval service - booster bitswap MVP executable #707
• Announce Bitswap service multiaddrs to the Network Indexers #711
• booster-bitswap add option to limit the number of simultaneous peer connections #716
• booster-bitswap block filter #731

Adding metrics/logging

• Metrics infrastructure (OpenCensus) for Boost #772
• Audit the split bitswap server tracing to determine what data is currently being tracked, and what, if any additional tracking we need
  • Supplement the bitswap server tracing code if needed
• Add opt in metrics #365
  • Add support for uploading these to a central service for the Boost team to analyze
• ...

Related Issues

• [Tracking issue] http delegated routing used in production ipfs/kubo#9150
• Split server & client and create message dispatcher object ipfs/go-bitswap#568

References & Related Items

• IPFS Gateway <> SP Planning
• [Epic] Improving retrieval reliability #645

[dd45e640b42e6da7da96faee3996ef7c]

SPs need to be able to turn this feature off

[dirkmc]

It will run as a service that is started separately from the boostd process