How to set Content-Length header

[vijayrajah]

I'm trying to call an API, that expects 'Content-Length' = 0 for a HTTP PATCH Request. There is no request body for this request.

It seems golang (from 1.8) does not allow to set Content-length with zero bytes. ( pls see -- golang/go#20257)

How do I set the header?
I have tried

restyClient.SetBody(http.NoBody).SetContentLength(true)

And

reqHeader := map[string]string {
		"Content-Length": "0",
}

None of these 2 ways set the content-length header.

[moorereason]

I'm not able to reproduce this issue. Can you provide a reproducible example?

package main

import (
        "fmt"

        "github.com/go-resty/resty/v2"
)

func main() {
        resp, err := resty.New().R().Patch("https://httpbin.org/anything")
        if err != nil {
                panic(err)
        }

        fmt.Printf("%s\n", resp.Body())
}

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept-Encoding": "gzip",
    "Content-Length": "0",
    "Host": "httpbin.org",
    "User-Agent": "go-resty/2.3.0 (https://github.com/go-resty/resty)",
    "X-Amzn-Trace-Id": "Root=1-5fce910b-2d8546fb35643ed93aab435b"
  },
  "json": null,
  "method": "PATCH",
  "url": "https://httpbin.org/anything"
}

[vijayrajah]

@moorereason Thanks for looking into this issue

I added debug.. so this is my test code now

func main(){
c1 := createHTTPClient("test-token").R()

	reqFlushQuery := map[string]string {
		"action": "flush",
		"position": "11111",
	}

	c1.SetQueryParams(reqFlushQuery)
	resp, err := c1.Patch("https://httpbin.org/anything")
	if err != nil {
		panic(err)
	}
	fmt.Printf("%s\n", resp.Body())
}

func createHTTPClient(bt string) *resty.Client {
	httpClient := resty.New()
	httpClient.SetDebug(true)
	//set timeout
	httpClient.SetTimeout(5 * time.Minute)

	//set retirees to 2
	httpClient.RetryCount = 2

	httpClient.SetAuthScheme("Bearer").SetAuthToken(bt)

	return httpClient
}

The http bin response indeed has content-length header. But the request does not have one

2020/12/08 08:42:42.174303 DEBUG RESTY 
==============================================================================
~~~ REQUEST ~~~
PATCH  /anything?action=flush&position=11111  HTTP/1.1
HOST   : httpbin.org
HEADERS:
	Authorization: Bearer test-token
	User-Agent: go-resty/2.3.0 (https://github.com/go-resty/resty)
BODY   :
***** NO CONTENT *****
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 200 OK
PROTO        : HTTP/2.0
RECEIVED AT  : 2020-12-08T08:42:42.1673033+05:30
TIME DURATION: 1.2824316s
HEADERS      :
	Access-Control-Allow-Credentials: true
	Access-Control-Allow-Origin: *
	Content-Length: 544
	Content-Type: application/json
	Date: Tue, 08 Dec 2020 03:12:42 GMT
	Server: gunicorn/19.9.0
BODY         :
{
   "args": {
      "action": "flush",
      "position": "11111"
   },
   "data": "",
   "files": {},
   "form": {},
   "headers": {
      "Accept-Encoding": "gzip",
      "Authorization": "Bearer test-token",
      "Content-Length": "0",
      "Host": "httpbin.org",
      "User-Agent": "go-resty/2.3.0 (https://github.com/go-resty/resty)",
      "X-Amzn-Trace-Id": "Root=1-5fceef2a-18ab2d1a6c6dc5e808d99102"
   },
   "json": null,
   "method": "PATCH",
   "origin": "x.x.x.x",
   "url": "https://httpbin.org/anything?action=flush&position=11111"
}

==============================================================================

as you can see the request does not have the content-length header

[vijayrajah]

one more interesting info.

if i set

SetContentLength(true)

// add header
reqHeader := map[string]string {
		"Content-Length": "0",
	}
request.SetHeaders(reqHeader)

resty debug shows the content-length header is being set

2020/12/08 09:02:06.212107 DEBUG RESTY 
==============================================================================
~~~ REQUEST ~~~
PATCH  /anything?action=flush&position=11111  HTTP/1.1
HOST   : httpbin.org
HEADERS:
	Authorization: Bearer test-token
	Content-Length: 0
	User-Agent: go-resty/2.3.0 (https://github.com/go-resty/resty)
	X-Ms-Version: 2019-12-12
BODY   :
***** NO CONTENT *****

But fidler trace does not show the header as part of this request....

PATCH https://httpbin.org/anything?action=flush&position=11111 HTTP/1.1
Host: httpbin.org
User-Agent: go-resty/2.3.0 (https://github.com/go-resty/resty)
Authorization: Bearer test-token
Accept-Encoding: gzip

[moorereason]

Resty debug likely won't include the Content-Length header unless you add the header yourself.

It looks to me like httpbin is receiving a Content-Length header (based upon your first reply). Can you cross-reference the httpbin output and Fiddler?

[vijayrajah]

@moorereason It seems this is inconsistent.. I do not see the Content-Length response from httpbin

here is the code I used

package main

import (
	"fmt"
	"time"

	"github.com/go-resty/resty/v2"
)

func main() {
	client := resty.New().SetTimeout(5 * time.Minute).SetAuthScheme("Bearer").SetAuthToken("bt")
	client.RetryCount = 3

	resp, err := client.R().Patch("http://httpbin.org/anything")
	if err != nil {
		panic(err)
	}

	fmt.Printf("%s\n", resp.Body())
}

Here is the response from httpbin

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Authorization": "Bearer bt", 
    "Host": "httpbin.org", 
    "User-Agent": "go-resty/2.3.0 (https://github.com/go-resty/resty)", 
    "X-Amzn-Trace-Id": "Root=1-5fd44c6c-014416c57be5712a6459e40e"
  }, 
  "json": null, 
  "method": "PATCH", 
  "origin": "<MYIP>", 
  "url": "http://httpbin.org/anything"
}

While this was running I did packet capture... Here is the packet capture... ( Using: tcpdump -nn -vv -A port 80 )

10:21:56.489359 IP (tos 0x0, ttl 64, id 34255, offset 0, flags [DF], proto TCP (6), length 212)
    192.168.10.176.46818 > 52.3.32.149.80: Flags [P.], cksum 0x20b7 (incorrect -> 0x0fcf), seq 1:161, ack 1, win 502, options [nop,nop,TS val 768094857 ecr 114680685], length 160: HTTP
E.....@.@..d..
.4. ....PW....,(..... ......
-.2....mPATCH /anything HTTP/1.1
Host: httpbin.org
User-Agent: go-resty/2.3.0 (https://github.com/go-resty/resty)
Authorization: Bearer bt
Accept-Encoding: gzip


10:21:56.712734 IP (tos 0x0, ttl 220, id 51969, offset 0, flags [DF], proto TCP (6), length 52)
    52.3.32.149.80 > 192.168.10.176.46818: Flags [.], cksum 0x864b (correct), seq 1, ack 161, win 110, options [nop,nop,TS val 114680741 ecr 768094857], length 0
E..4..@.....4. ...
..P...,(.W......n.K.....
....-.2.
10:21:56.714495 IP (tos 0x0, ttl 220, id 51970, offset 0, flags [DF], proto TCP (6), length 282)
    52.3.32.149.80 > 192.168.10.176.46818: Flags [P.], cksum 0x0026 (correct), seq 1:231, ack 161, win 110, options [nop,nop,TS val 114680741 ecr 768094857], length 230: HTTP, length: 230
	HTTP/1.1 200 OK
	Date: Sat, 12 Dec 2020 04:51:56 GMT
	Content-Type: application/json
	Content-Length: 429
	Connection: keep-alive
	Server: gunicorn/19.9.0
	Access-Control-Allow-Origin: *
	Access-Control-Allow-Credentials: true
	
E.....@.....4. ...
..P...,(.W......n.&.....
....-.2.HTTP/1.1 200 OK
Date: Sat, 12 Dec 2020 04:51:56 GMT
Content-Type: application/json
Content-Length: 429
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true


10:21:56.714523 IP (tos 0x0, ttl 64, id 34256, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.10.176.46818 > 52.3.32.149.80: Flags [.], cksum 0x2017 (incorrect -> 0x82fd), seq 161, ack 231, win 501, options [nop,nop,TS val 768095082 ecr 114680741], length 0
E..4..@.@.....
.4. ....PW....,)..... ......
-.3j....
10:21:56.714536 IP (tos 0x0, ttl 220, id 51971, offset 0, flags [DF], proto TCP (6), length 481)
    52.3.32.149.80 > 192.168.10.176.46818: Flags [P.], cksum 0xde76 (correct), seq 231:660, ack 161, win 110, options [nop,nop,TS val 114680741 ecr 768094857], length 429: HTTP
E.....@...."4. ...
..P...,).W......n.v.....
....-.2.{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Authorization": "Bearer bt", 
    "Host": "httpbin.org", 
    "User-Agent": "go-resty/2.3.0 (https://github.com/go-resty/resty)", 
    "X-Amzn-Trace-Id": "Root=1-5fd44c6c-014416c57be5712a6459e40e"
  }, 
  "json": null, 
  "method": "PATCH", 
  "origin": "<MYIP>", 
  "url": "http://httpbin.org/anything"
}

If it makes any difference, This test I performed in my personal PC (go 1.15.6 on Linux ). The previous test was from my Work PC( go 1.15.4 & Win 10)

Is there a way to Force 'Content-length' and not let golang 'swallow' it?

[vijayrajah]

One more info.. Courtesy of this SO question

i wrote this code

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
)

func main() {

	url := "http://httpbin.org/anything"
	method := "PATCH"

	client := &http.Client{}
	req, err := http.NewRequest(method, url, http.NoBody)
	if err != nil {
		panic(err)
	}

	req.TransferEncoding = []string{"identity"}
	req.Header.Set("Authorization", "Bearer my-token")

	res, err := client.Do(req)
	defer res.Body.Close()
	body, err := ioutil.ReadAll(res.Body)

	fmt.Println(string(body))
}

For this I DO see the content-Length header

Response:

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Authorization": "Bearer my-token", 
    "Content-Length": "0", 
    "Host": "httpbin.org", 
    "User-Agent": "Go-http-client/1.1", 
    "X-Amzn-Trace-Id": "Root=1-5fd463ac-40a22b297a343fc255e0e621"
  }, 
  "json": null, 
  "method": "PATCH", 
  "origin": "<MYIP>", 
  "url": "http://httpbin.org/anything"
}


Process finished with exit code 0

Packet capture

12:01:08.520958 IP (tos 0x0, ttl 64, id 21336, offset 0, flags [DF], proto TCP (6), length 205)
    192.168.10.176.37488 > 184.72.216.47.80: Flags [P.], cksum 0x5c90 (incorrect -> 0x123b), seq 1:154, ack 1, win 502, options [nop,nop,TS val 2495356391 ecr 304828402], length 153: HTTP
E...SX@.@.....
..H./.p.P.......*....\......
.....+O.PATCH /anything HTTP/1.1
Host: httpbin.org
User-Agent: Go-http-client/1.1
Content-Length: 0
Authorization: Bearer my-token
Accept-Encoding: gzip


12:01:08.724087 IP (tos 0x0, ttl 224, id 51983, offset 0, flags [DF], proto TCP (6), length 52)
    184.72.216.47.80 > 192.168.10.176.37488: Flags [.], cksum 0x5c08 (correct), seq 1, ack 154, win 110, options [nop,nop,TS val 304828453 ecr 2495356391], length 0
E..4..@...s..H./..
..P.p...*...q...n\......
.+P%....
12:01:08.725423 IP (tos 0x0, ttl 224, id 51984, offset 0, flags [DF], proto TCP (6), length 282)
    184.72.216.47.80 > 192.168.10.176.37488: Flags [P.], cksum 0xe2df (correct), seq 1:231, ack 154, win 110, options [nop,nop,TS val 304828453 ecr 2495356391], length 230: HTTP, length: 230
	HTTP/1.1 200 OK
	Date: Sat, 12 Dec 2020 06:31:08 GMT
	Content-Type: application/json
	Content-Length: 431
	Connection: keep-alive
	Server: gunicorn/19.9.0
	Access-Control-Allow-Origin: *
	Access-Control-Allow-Credentials: true
	
E.....@...r..H./..
..P.p...*...q...n.......
.+P%....HTTP/1.1 200 OK
Date: Sat, 12 Dec 2020 06:31:08 GMT
Content-Type: application/json
Content-Length: 431
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true


12:01:08.725430 IP (tos 0x0, ttl 64, id 21337, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.10.176.37488 > 184.72.216.47.80: Flags [.], cksum 0x5bf7 (incorrect -> 0x58ce), seq 154, ack 231, win 501, options [nop,nop,TS val 2495356596 ecr 304828453], length 0
E..4SY@.@.....
..H./.p.P...q........[......
.....+P%
12:01:08.725434 IP (tos 0x0, ttl 224, id 51985, offset 0, flags [DF], proto TCP (6), length 483)
    184.72.216.47.80 > 192.168.10.176.37488: Flags [P.], cksum 0xb3a9 (correct), seq 231:662, ack 154, win 110, options [nop,nop,TS val 304828453 ecr 2495356391], length 431: HTTP
E.....@...r2.H./..
..P.p.......q...n.......
.+P%....{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Authorization": "Bearer my-token", 
    "Content-Length": "0", 
    "Host": "httpbin.org", 
    "User-Agent": "Go-http-client/1.1", 
    "X-Amzn-Trace-Id": "Root=1-5fd463ac-40a22b297a343fc255e0e621"
  }, 
  "json": null, 
  "method": "PATCH", 
  "origin": "<MYIP>", 
  "url": "http://httpbin.org/anything"
}

Can we do similar thing in Resty?

[moorereason]

Interesting. If you hit the SSL URL (https://httpbin.org/anything), resty sends a Content-Length header. Non-SSL requests don't send the header. 😕 Hmm

[moorereason]

Findings

Ok. I'm not able to come up with a way to send a Content-Length header for non-http2 requests with the current release of resty (v2.3.0). My previous comment about SSL requests working was really that the client was using http2. If you disable http2 on the client (GODEBUG=http2client=0 go run main.go), the SSL request also fails to send the Content-Length header.

Work-around in master

With the most recent commit to master, the following will send a Content-Length header:

resp, err := resty.New().R().SetBody(http.NoBody).Patch("http://httpbin.org/anything")

However, that's not very intuitive and requires knowing a rather peculiar implementation detail with the http package.

Possible Fix

Given the following patch against master:

diff --git a/middleware.go b/middleware.go
index 50ff448..2b653fd 100644
--- a/middleware.go
+++ b/middleware.go
@@ -162,6 +162,12 @@ func createHTTPRequest(c *Client, r *Request) (err error) {
        if r.bodyBuf == nil {
                if reader, ok := r.Body.(io.Reader); ok {
                        r.RawRequest, err = http.NewRequest(r.Method, r.URL, reader)
+               } else if c.setContentLength || r.setContentLength {
+                       r.RawRequest, err = http.NewRequest(r.Method, r.URL, http.NoBody)
                } else {
                        r.RawRequest, err = http.NewRequest(r.Method, r.URL, nil)
                }

I'm able to get a Content-Length header on non-http2 requests when using only SetContentLength(true):

resp, err := resty.New().R().SetContentLength(true).Patch("http://httpbin.org/anything")

Because of what happens later on in createHTTPRequest, simply creating a NewRequest with nil doesn't maintain the body in a way that the http package can detect the zero-length body. I think this patch lets SetContentLength do it's job when we want it to without further complicating how we process the body.

@jeevatkm, what are your thoughts on this change? All existing tests pass.

[moorereason]

BTW, this is fixed upstream in Go 1.16beta1. See golang/go#40978. The draft release notes say:

The Client now sends an explicit Content-Length: 0 header in PATCH requests with empty bodies, matching the existing behavior of POST and PUT.

With Go 1.16beta1 and resty v2.3.0, I'm now seeing a Content-Length: 0 header with the following request:

resty.New().R().Patch("http://httpbin.org/anything")

Yea! 🎉

[vijayrajah]

@moorereason Thanks a lot. That is the exact API, I'm trying to use. :) (azure DFS Gen2 API)

one more reason to look forward for 1.16 ( as it also includes //go:embed)

[jeevatkm]

@moorereason @vijayrajah Nice interaction 👍

@moorereason your fix suggestion, I think we can go for it.

[jeevatkm]

@moorereason I have applied your suggestion will be part of v2.4.0 release.
@vijayrajah FYI