Embedded JS-IPFS in Brave (experiment)

[lidel]

This issue tracks remaining work around embedded js-ipfs running in ipfs-companion in Brave with access to chrome.sockets APIs. For more context, see:

• Option (B) from notes on bringing IPFS to Brave in 2019
  (IPFS Integration - Roadmap and discussion brave/brave-browser#819 (comment)))
• Support chrome.sockets.* APIs (Support chrome.sockets.* APIs #664)
• Old issues: IPFS Integration in the Brave Browser in-web-browsers#64, Muon-based Brave Browser Integrations  #312

High Level Goals

• Leveling up js-ipfs to match go-ipfs in features relevant to in web browser uses
• Brave users can use IPFS with Companion alone, without installing go-ipfs or ipfs-desktop
• IPFS nodes in Brave can discover each other on the same network and exchange data in p2p fashion

Kanban Board

• https://github.com/ipfs-shipyard/ipfs-companion/projects/4

How to opt-in

Stable release of IPFS Companion can be enabled via Brave Settings (chrome://settings/extensions):

By default, it works like regular ipfs-companion and External node is used by default, assuming user has go-ipfs or IPFS Desktop installed and running.

Brave users can opt-in and try Embedded + chrome.sockets (see /docs/node-types for more details)

TODO

🎨 UX

• UI for changing Gateway port
  • Right now its just raw JSON with js-ipfs config
• Redirect to embedded Gateway
• New embedded node type exists (embedded:chromesockets)
• Companion UI is aware of embedded Gateway (Support chrome.sockets.* APIs #664 (comment))
• Play nice with go-ipfs and js-ipfs daemons (Support chrome.sockets.* APIs #664 (comment))
  • user can switch between embedded (default in Brave) and external IPFS nodes
  • use custom ports by default
• Post-install welcome screen enables user to start IPFS with one click
  • Add option to start embedded js-ipfs
  • See if we able to make it useful on Android as well
• Brave detects URLs from embedded gateway and displays them as ipfs:// and ipns://
  • Ideally, along with Origin isolation
  • UX tracked in Improving UX of IPFS in Address Bar brave/brave-browser#5218

🔌 chrome.sockets

• feature-detection for chrome.sockets (Support chrome.sockets.* APIs #664)
  • the same package should run on Google Chrome and Brave, chrome.sockets should enable additional powers at runtime

• Brave overrides Manifest on install from Chrome Web Store to add sockets section

🌐 Embedded HTTP Gateway

• Companion can use HTTP gateway provided by embedded JS-IPFS node (Support chrome.sockets.* APIs #664 (comment))
  • Decouple HTTP Servers from cli/commands/daemon (Decouple HTTP Servers from cli/commands/daemon js-ipfs#1950)
• Web UI loads and can be opened from Companion's main menu (Support chrome.sockets.* APIs #664 (comment))
• HTTP Responses are valid
  • streaming (Gateway Improvements: Streaming, Conditional and Range Requests js-ipfs#1989)
  • content-type sniffing (Gateway Improvements: Streaming, Conditional and Range Requests js-ipfs#1989)
  • missing last-modified (Gateway Improvements: Streaming, Conditional and Range Requests js-ipfs#1989)
  • support if-none-match and 304 Not Modified (Gateway Improvements: Streaming, Conditional and Range Requests js-ipfs#1989)
  • support ?filename and Content-Disposition (Gateway Improvements: Streaming, Conditional and Range Requests js-ipfs#1989)
  • range requests via video player (Gateway Improvements: Streaming, Conditional and Range Requests js-ipfs#1989)
  • Check usual suspects from HTTP Headers Cleanup: API and Gateway in-web-browsers#132
• DNSLink websites (e.g. /ipns/docs.ipfs.io/) are not supported by js-ipfs
  • upstream fix Resolving DNSLink Paths: /ipns/<fqdn> js-ipfs#1918
• HAMT-sharded directories, eg. /ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco/wiki/ do not work with js-ipfs
  • tracked upstream in HTTP Gateway fails to load sharded website js-ipfs#1963
• HTTP Proxy mode is supported by js-ipfs and ipfs-companion (Act as HTTP PROXY for http://<cidv1b32>.ipfs.localhost js-ipfs#1877)
  • this is "nice to have" – low priority unless go-ipfs ships with it
• ... in directory listing points at current directory instead of its parent
• Broken bandwidth graph in Web UI (Bandwidth Graph Anomaly in Brave ipfs-webui#1012)
• Unicode in paths works (eg /ipns/tr.wikipedia-on-ipfs.org/wiki/Venüs.html)

🕸️ Local Discovery and UDP/TCP Transports

• Embedded JS IPFS node is using P2P Transport over chrome.sockets.tcp and chrome.sockets.tcpServer)
  • connecting to other peers
  • accepting incoming connections
    • discover own IP (webrtc-ips)
    • discover all IPs chrome.system.network.getNetworkInterfaces ?
• Embedded JS IPFS node performs DNS-based Local Discovery (In-browser discovery of local-network IPFS nodes in-web-browsers#45)
  • chrome.mdns implements this dead spec and is not useful for our use case, but we should be able to implement working doscovery on top of chrome.sockets.udp
  • short-term interop with go-ipfs: feat: compatibility with go-libp2p-mdns libp2p/js-libp2p-mdns#80
    • discover go-ipfs
    • broadcast own TCP multiaddr (tracked in Brave: LAN discovery & announcement (mDNS, DNS-SD) #758)
  • long-term interop (new spec): [RFC] peer discovery with mDNS libp2p/specs#80

⚡ Performance

• 🕵️‍♂️ Finding/loading content
  • noticeable especially when preload nodes are under load and slow to respond
  • may just go away when we add p2p transport that can interop with go nodes (assuming DHT issues are addressed by then)
  • for now, we may mitigate this by enabling DHT with autorelay, or doing preloads and delegated routing:
    • preload things requested via gateway (Enable preload on MFS commands that accept IPFS paths js-ipfs#2335)
    • delegated routing: Add delegated routing to js-ipfs browser js-ipfs#2155
      • unable to enable 🙃 fix: allow setting Addresses.Delegates js-ipfs#2253
      • blocked due to request flood: Request flood in web browser libp2p/js-libp2p-delegated-content-routing#12
    • Enable DHT
      • Overview of issues and why its disabled for now: Cannot retrieve content via the DHT js-ipfs#1984
      • Closest peers query is not efficient: Closest peers query is not efficient libp2p/js-libp2p-kad-dht#86
• UI / defaults for storage limits and GC in js-ipfs
  • js-ipfs does not have GC yet: Implement Garbage Collection js-ipfs#2022
• inspect libp2p connection manager limits
• DNS lookups hit ipfs.io/api/v0/dns: we need to add cache to this, or support DoH (Support DNS over HTTPS helia-ipns#53) or implement chrome.sockets.udp* API
  • Throttling and caching implemented in fix: limit concurrent HTTP requests in browser js-ipfs#2304