Description
What version of Go are you using (go version)?
$ go version
go version go1.11.5 linux/amd64
Does this issue reproduce with the latest release?
Yes (1.11.5), haven't tried the 1.12 RC, but the implementations of persistConnWriter and transferWriter.writeBody do not appear to have changed, which is where the issue lies.
What operating system and processor architecture are you using (go env)?
$ go env GOOS GOARCH
linux
amd64
What did you do?
Upload a file using PUT in a http.Request, by setting the file as Body:
fh, err := os.Open(fileName)
// ....
req, err := http.NewRequest("PUT", destURL, fh)
(Not that our implementations at HashiCorp usually wrap in go-retryablehttp, which is interface-compatible with the above. This is what the benchmarks below use. The above example is to illustrate the repro using the stdlib.)
This test utility was tested on live systems sending uploads to live services.
What did you expect to see?
An upload speed matching something more representative of the speed of the links the systems were connected to. fasthttp, which was also tested, gave upload speeds of about 50-70 MB/sec on live systems depending on the location of the client machine.
What did you see instead?
Upload speeds ranging from about 8MB/sec (lower latency systems) to as low as sub-1MB/sec (higher-latency systems).
Additional info/root cause
Investigation into the internals of the net/http transport and its behavior compared to fasthttp revealed that the path fasthttp currently takes allowed it to use sendfile to transfer the data instead of write calls. This was discovered by using various tracing tools (strace/bpftrace) during testing.
Further investigation into the net/http transport revealed that its writer is currently wrapped in persistConnWriter which does not implement io.ReaderFrom, which is what is necessary for an io.Copy to the transport's connection (ie: TCPConn) to ultimately fast-path to sendfile. See here. Further to that, the request body is wrapped in a transferBodyReader, and possibly an ioutil.nopCloser, obfuscating the reader to the point where TCPConn cannot discern properly whether or not the underlying reader is an *os.File.
Adding a trivial implementation for ReadFrom in persistConnWriter allows for the pathing to sendfile, in addition to significantly speeding up uploads when the reader is not an *os.File, or if the OS does not have a sendfile implementation (example: darwin), due to the use of genericReadFrom. Replacing transferBodyReader with a non-writer implementation - and providing additional unwrapping within writeBody to unwrap the underlying reader from the nopCloser if the wrapping exists - ensures that the proper reader makes its way to TCPConn.sendFile.
Example on a local machine using a simple net/http server reading the body into an ioutil.Discard, uploading a 1GB file of random data:
$ ./simpleput-write garbage.dat http://127.0.0.1:8080/
2019/02/24 09:39:16 [DEBUG] PUT http://127.0.0.1:8080/
2019/02/24 09:39:18 upload complete; duration: 1.564957672s; size: 1048576000 bytes; rate: 638.994918 MB/sec
$ ./simpleput-sendfile garbage.dat http://127.0.0.1:8080/
2019/02/24 09:39:39 [DEBUG] PUT http://127.0.0.1:8080/
2019/02/24 09:39:39 upload complete; duration: 244.542093ms; size: 1048576000 bytes; rate: 4089.275542 MB/sec