New RPC system

[keks]

There's a few issues with the http API and it would be cool to support different kinds of transports like Unix domain sockets or websockets. So having a protocol that runs atop any reliable transport would be desirable.

This issue is about exploring the problem space and having a place to put ideas that need to be considered when we finally start working on this.

Relevant issues:

• Wrap results in something indicating the type of the result #31 Make sure we can distinguish error from value at all times
• Support Unix Sockets kubo#4218 Add support for Unix domain sockets

[magik6k]

Sending cbor rpcs both ways should be good enough.

CBOR seems to be perfect for RPC over reliable transports as it supports streaming, has defined boundaries and is very compact when data structure is designed well.

I'm not sure about the exact requirements, but what would probably work well would be client sending requests with some sort of unique identifier in cbor frames, and then the server would respond to those requests, not necessarily in the order they came, possibly with multiple responses over time (commands like ipfs ping). Only problem I see here is figuring out backpressure.

[keks]

While I agree that CBOR seems like a good fit, I'd probably advocate using multistream for this.
But yeah, CBOR would probably be the default format.

re back pressure: don't most reliable protocols handle that? At some boint the receive buffer is full and the client does acknowledge further packets. We can call TCPConn.SetReadBuffer(somethingSmall) to make sure the sender doesn't send too much, but I'm not sure how to find a good value here. Unix domain sockets have hard backpressure, the sender won't be able to write when the receiver doesn't read.

[Stebalien]

While I agree that CBOR seems like a good fit, I'd probably advocate using multistream for this.
But yeah, CBOR would probably be the default format.

YES! Multi all the things.

Only problem I see here is figuring out backpressure.

I'm not sure this is necessary. This is a privileged, local interface.

---

We may also be able to get away with using an existing RPC system.

[magik6k]

I was thinking about throwing multistream at it, one problem is that adds 1rtt (even if local this has some effect on short lived sessions, though it can be probably turned into 0rtt with some clever hacks), another is the added complexity - API wrappers will have to implement multistream handshake. Advantage is that we could probably reuse API port and try to detect if the incoming request is HTTP or multistream.

We may also be able to get away with using an existing RPC system.

That would be optimal

[dignifiedquire]

Some thoughts, as I have been thinking about this as well.

• Adding multistream is quite a large requirement if you want to make it easy for other clients to connect to.
• List of transports I would like to see are
  • unix sockets
  • windows pipes
  • websockets
  • in process
• CBOR is great but does have overhead, so I think there should be ways to avoid this for certain operations, i.e. streaming a file shouldn't go through cbor, but rather should be piped directly through with the most minimal overhead possible.

[dignifiedquire]

As reference, the RPC api from ethereum has quite a nice implementation of the various transport options here: https://github.com/ethereum/go-ethereum/tree/master/rpc. (this does not mean I am suggesting to use a json rpc api).

I think it is important to keep in mind that this would be an end user interface ideally, an the CLI just one of many clients, so the requirements should be kept as simple as we can on the client side. Which makes me think using any sort of multistream would be too much overhead there.

[Kubuxu]

Other option could be capnproto https://capnproto.org/.

I would love to see the API in RPC form with pressure on performance. As an example, possible use case could be FUSE.

IMO it could mimic core-api defined in go-ipfs. As far as I know JS's interface-core-api closely mimics current HTTP api and so the current CLI.
We want to rewrite current commands to use go's core-api so it would be great if later we could just plug'n'play it over the RPC.

---

streaming a file shouldn't go through cbor, but rather should be piped directly through with the most minimal overhead possible.

This would require muxer if there are to be multiple concurrent requests from one client (something we most certainly want). GRPC provides it but mostly because it is over QUIC so they get it for free.

[Stebalien]

Really, it would be kind of nice to be able to use libp2p. That would also make remote administration easier. @vyzo actually just wrote a dead-simple unix socket based transport for libp2p but we could use any transport, really. The client doesn't have to implement the entire libp2p stack, just the individual transport (and probably multistream-select but that's quite simple).

[keks]

Let me collect design constraints we've identified so far:

• Errors should always be distinguishable from values
  • Corollary: values should always be identified as such
  • we have a situation where we request a raw object and that may look like an error and we don't can't know whether we correctly received the stored object that looks like an error or whether we actually received an error...
• Web friendly
• Add support for UNIX sockets
• Keep an eye on performance
• make it easy for clients to use (i.e. to implement the API transport protocol)
• Allow multiple long-running commands over a single connection
  • this requires muxing (with backpressure on individual streams)
• don't reinvent the wheel

---

My main concern is that requiring the client to mux multiple command call streams over a single connection is decidedly not easy to implement, unless you take something off-the-shelf (QUIC, HTTP/2, yamux(?)). However, if we decide to go for one of these, I guess we already are at 60% of basic libp2p, so maybe it does make sense to just use that.

[Kubuxu]

Allow multiple long-running commands over a single connection
this requires muxing (with backpressure on individual streams)

This isn't true if you go with RPC and handles. Then you can have multiple long running commands. Example imagine a Read on file.

The way you are thinking about is just streaming the file. In this case, muxing is required.

Another way is to give the user a handle for the open file, then user requests a 2KiB chunk of the file. You prepare a buffer on the server side and after it is filled to 2KiB you send it over with minimal framing referencing the handle you've given.

This way there is no blocking but implementation itself is more complex. We have to decide if that complexity is worst than using a muxer.

[Stebalien]

Yeah, muxing complexity could be a concern but I think we can just make it a performance tradeoff. We can expose the API over as many protocols as we want so users can use connection pools with no muxers.

Requirement: web friendly.

[dignifiedquire]

as much as I like libp2p, adding the requirement of loading all of js-libp2p into the browser sounds like a very high overhead for an important use case. And makes it much less attractive at least for me

…

On 3. Oct 2018, 02:17 +0200, Steven Allen ***@***.***>, wrote: Yeah, muxing complexity could be a concern but I think we can just make it a performance tradeoff. We can expose the API over as many protocols as we want so users can use connection pools with no muxers. Requirement: web friendly. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.