http/2 support in ipfs api server

[sunboshan]

Version information:

➜ ipfs version --all
go-ipfs version: 0.4.18-aefc746
Repo version: 7
System version: amd64/darwin
Golang version: go1.11.2

Type:

bug

Description:

Repro step

http/1.1 request to cat a file works

➜ curl -v --http1.1 http://127.0.0.1:5001/api/v0/cat\?arg\=QmUL7wDowvNk3y7KeEYFAATmz43727FwXKhBJJrqQu813a
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 5001 (#0)
> GET /api/v0/cat?arg=QmUL7wDowvNk3y7KeEYFAATmz43727FwXKhBJJrqQu813a HTTP/1.1
> Host: 127.0.0.1:5001
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Access-Control-Allow-Headers: X-Stream-Output, X-Chunked-Output, X-Content-Length
< Access-Control-Expose-Headers: X-Stream-Output, X-Chunked-Output, X-Content-Length
< Content-Type: text/plain
< Server: go-ipfs/0.4.18
< Trailer: X-Stream-Error
< Vary: Origin
< X-Content-Length: 3
< X-Stream-Output: 1
< Date: Thu, 07 Feb 2019 19:17:46 GMT
< Transfer-Encoding: chunked
<
* Connection #0 to host 127.0.0.1 left intact
123

http/2 request does not

➜ curl -v --http2-prior-knowledge http://127.0.0.1:5001/api/v0/cat\?arg\=QmUL7wDowvNk3y7KeEYFAATmz43727FwXKhBJJrqQu813a
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 5001 (#0)
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fa06f80b200)
> GET /api/v0/cat?arg=QmUL7wDowvNk3y7KeEYFAATmz43727FwXKhBJJrqQu813a HTTP/2
> Host: 127.0.0.1:5001
> User-Agent: curl/7.54.0
> Accept: */*
>
* http2 error: Remote peer returned unexpected data while we expected SETTINGS frame.  Perhaps, peer does not support HTTP/2 properly.
* Send failure: Broken pipe
* Failed sending HTTP2 data
* Recv failure: Connection reset by peer
* Failed receiving HTTP2 data
* Closing connection 0
curl: (56) Send failure: Broken pipe

I'm not sure if this is a bug, or ipfs api server doesn't support http/2 yet? Thanks!

[magik6k]

AFAIK we use the built-in go http server, which currently doesn't support http2 (there is a WIP implementation in golang.org/x/net/http2, but it's not supposed to be used for production code)

Quote from https://github.com/golang/net/blob/master/http2/README:

This is a work-in-progress HTTP/2 implementation for Go.

It will eventually live in the Go standard library and won't require
any changes to your code to use. It will just be automatic.

[wstrm]

Hmm, the docs says that it is used automatically since 1.6:

This package is low-level and intended to be used directly by very few people. Most users will use it indirectly through the automatic use by the net/http package (from Go 1.6 and later).
https://godoc.org/golang.org/x/net/http2

I believe the problem is that the connection is done without TLS.

As it seems to work with a really simple demo:

package main

import (
	"log"
	"net/http"
)

func hello(w http.ResponseWriter, r *http.Request) {
	w.Write([]byte("Hello there"))
}

func main() {
	http.HandleFunc("/", hello)
	if err := http.ListenAndServeTLS(":8888", "server.crt", "server.key", nil); err != nil {
		log.Fatalln(err)
	}
}

~ ξ go version                                                                                                      2
go version go1.11.4 linux/amd64
~ ξ curl -v --insecure --http2-prior-knowledge https://localhost:8888                                               7
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8888 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
*  start date: Feb 12 15:07:05 2019 GMT
*  expire date: Feb  9 15:07:05 2029 GMT
*  issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5584226e6e80)
> GET / HTTP/2
> Host: localhost:8888
> User-Agent: curl/7.63.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200 
< content-type: text/plain; charset=utf-8
< content-length: 11
< date: Tue, 12 Feb 2019 15:08:51 GMT
< 
* Connection #0 to host localhost left intact
Hello there%

I do, however, get the same error as posted above if I'm not using TLS. :)

[Kubuxu]

Providing an HTTPS on local API is hard. It is quite possible that we depend on some HTTP1.X semantics in our commands library that facilitates the HTTP API.

This is why I'm leaning toward closing this issue as won't fix.

[sunboshan]

The motivation behind HTTP/2 support on IPFS api server is prevent head of line blocking in HTTP/1.1, since when do ipfs cat <some-hash>, the ipfs api server will not return but trying to find the file over the network, and then causing all the following requests stuck there on the same tcp connection. The only workaround now is to kill the connection and start a new one. HTTP/2 shall solve this issue completely.

I do remember that Google mentioned on they only support HTTP/2 via TLS, in order to enforce a more secure internet. So Google Chrome and golang's net/http only support HTTP/2 over TLS. Just wondering does IPFS supports a node providing a self-signed certs and set up a HTTP/2 enabled api server?

[Kubuxu]

go-ipfs' API port has connection reuse disabled, so there is no possibility of ahead of the line blocking.

In past, we explicitly decided not to implement any HTTPS related functionality in go-ipfs and I see little to no benefit in introducing HTTP/2. Unless we find that benefit and fix already existing issues there is no need for it.

[Stebalien]

go-ipfs' API port has connection reuse disabled, so there is no possibility of ahead of the line blocking.

Not entirely. We had to disable this for some requests due to #5168 and golang/go#15527 but we can otherwise reuse connections.

But, in this case, I agree that the right approach is one connection per request (or a large connection pool). The API is generally designed for use on localhost so we're not worried about RTT.

[sunboshan]

@Kubuxu I have a use case where there are two ipfs nodes in a private network(with swarm.key).

In one node, add a file

➜ echo -n "hello" | ipfs add
added QmWfVY9y3xjsixTgbd9AorQxH7VtMpzfx2HaWtsoUYecaX QmWfVY9y3xjsixTgbd9AorQxH7VtMpzfx2HaWtsoUYecaX
 5 B / ? [--------------------------------------------------------------------------------------------------=----] 

In another node, cat the file, it works

➜ ipfs cat QmWfVY9y3xjsixTgbd9AorQxH7VtMpzfx2HaWtsoUYecaX
hello% 

However, if I cat a non-existing file hash(changing the last letter to lower case), it stuck there and not return anything

➜ ipfs cat QmWfVY9y3xjsixTgbd9AorQxH7VtMpzfx2HaWtsoUYecax
  # nothing returns, just stuck here forever

Since I only have 2 nodes in my entire network, if it cannot find the file in the entire network, it should at least return something instead of stuck there. Not sure if this is the desired behavior for ipfs or a bug? Or am I missing anything here?

[Stebalien]

I'm not seeing how this is related to the current issue.

Note, the issue here is that the second peer has no way of knowing that it won't eventually get the file (the first peer may add it later, more peers may join, etc.). Related to: #5541

[sunboshan]

If i'm using a HTTP/1.1 connection and send a cat request to ipfs api server, it's gonna stuck here and all the following request send via the same connection will be stuck as well due to the head of line blocking. Though HTTP/2 would resolve this issue. I guess the only option for now is to use a new connection per request.

However, if on the ipfs server end, the connection won't close in a reasonable time but waiting there and hoping eventually it will get the file, it's pretty easy to DDOS attack the ipfs server right? I can just send 10000 request per second with non-existing hash, then the server would try to get those files and eventually eat up all the memory?

[Stebalien]

If i'm using a HTTP/1.1 connection and send a cat request to ipfs api server, it's gonna stuck here and all the following request send via the same connection will be stuck as well due to the head of line blocking. Though HTTP/2 would resolve this issue. I guess the only option for now is to use a new connection per request.

Yes, the solution here is just to use a new connection. Reusing the same connection is important on the open web but doesn't really make a difference for localhost. Independent calls to ipfs cat will always use new connections.

However, if on the ipfs server end, the connection won't close in a reasonable time but waiting there and hoping eventually it will get the file, it's pretty easy to DDOS attack the ipfs server right? I can just send 10000 request per second with non-existing hash, then the server would try to get those files and eventually eat up all the memory?

Against a gateway? Possibly but you'd need to keep all those connections open. I'm not sure what our DoS mitigation is at the moment but, if necessary, we could introduce rate-limits and/or connection limits.

Note: IPFS really isn't designed to be used as a "server" in this way. We run public gateways but that's just a stop-gap to allow people to access content on IPFS without having to install it.

[magik6k]

I can just send 10000 request per second with non-existing hash, then the server would try to get those files and eventually eat up all the memory?

Even if we just returned errors there, you could just not read the header. Or do that really slowly..

https://en.wikipedia.org/wiki/Slowloris_(computer_security)

[lidel]

5 years later, there are two alternatives to TCP-based http 1.1 in ipfs --api these days:

• --api now supports both unix sockets (since Kubo 0.30)
• --api over HTTPS multiaddr will be supported in 0.33

Closing. Feel free to open new issue if not enough or you find any paper-cuts related to the two ways above.