golang ethclient recieve gzip response

[toby127]

I used ethclient(https) to call BlockNumber and more interface, but i found the Accept-Encoding header of https request always was gzip, it cause a error, how should i config it?

[toby127]

code:

        httpClient, err := ethclient.Dial("https://xxxx.xxx.xx")
	if err != nil {
		logger.LOG.Error("connect eth http node error", zap.Error(err))
		return
	}

	block, err := httpClient.BlockByNumber(context.Background(), big.NewInt(int64(Height)))
	if err != nil {
		logger.LOG.Error("get BlockByNumber err", zap.Error(err))
		return
	}

error: get BlockByNumber err {"error": "invalid character 'H' looking for beginning of value"}

this was i recieved : H4sIAAAAAAAA/6pWyirOzysqSFayUjLSM1DSUcpMUbIy1FEqSi0uzSlRslIyqEgzM7W0MFWq5QIEAAD//zIR804tAAAA

[s1na]

I can reproduce this if I explicitly set Accept-Encoding: gzip, i.e.:

     cl, err := rpc.DialContext(ctx, path)
     if err != nil {
         fmt.Errorf("err: %v\n", err)
         return
     }
     cl.SetHeader("Accept-Encoding", "gzip")
     ecl := ethclient.NewClient(cl)
     num, err := ecl.BlockNumber(context.Background())

Interestingly if I don't set the header, the request and response will look exactly the same over the wire, but somehow there is no error.

The gzip write was modified in #25457 but this issue doesn't seem to stem from there. It seems to have existed before.

[s1na]

https request always was gzip, it cause a error, how should i config it?

BTW to configure the headers you can use my example from above and change this line:

    cl.SetHeader("Accept-Encoding", "identity")

[toby127]

https request always was gzip, it cause a error, how should i config it?

BTW to configure the headers you can use my example from above and change this line:

    cl.SetHeader("Accept-Encoding", "identity")

Thanks for your help, that's useful！

[s1na]

I read somewhere that if we manually set Accept-Encoding: gzip the go http client won't decompress the result automatically. That explains why my example returns an error.

I still don't understand the cause of error for you. Can you explain how is the RPC server setup? geth doesn't support HTTPS API so it seems you have a custom setup. Edit: you're probably using an external service, right?

[toby127]

I use geth's ethclient but it doesn't set any headers. But by capturing the network packets, I found that it automatically adds the header Accept-Encoding: gzip to the client request. As you said, I use Amazon Managed Blockchain as my RPC server.

[holiman]

this was i recieved : H4sIAAAAAAAA/6pWyirOzysqSFayUjLSM1DSUcpMUbIy1FEqSi0uzSlRslIyqEgzM7W0MFWq5QIEAAD//zIR804tAAAA

I'm not convinced it has anything to do with gzip, what you have there is base64-encoded data. I think the problem might be with that remote server, rather than the client?

@s1na when you reproed the problem, did your data also look like base64 data?

[s1na]

@holiman this is the error I get invalid character '\x1f' looking for beginning of value. It could be that these are 2 separate things. Mine could be simply because I manually set gzip header and the client doesn't know it has to decompress. And the OG issue could stem from the remote server.

[holiman]

invalid character '\x1f' looking for beginning of value means you are actually getting some binary data, namely the character 1F. The gzip header starts with 1f 8b, so you are definitely seeing gzip.

However, the original reporter @0127abcd is not having gzip problems, but rather problems with base64. (Which, incidentally, is how golang by default would encode a []byte response over rpc. )

[holiman]

I read somewhere that if we manually set Accept-Encoding: gzip the go http client won't decompress the result automatically. That explains why my example returns an error.

That makes sense, and yes, it fully explains the behaviour @s1na is seeing. With that, I think we can close this, because the original error does not seem to be an error in the go-ethereum codebase.