crypto/tls: linux/arm64 Go 1.8 performance is slow, max 12.5 MB/sec

[vielmetti]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8 linux/arm64

What operating system and processor architecture are you using (go env)?

96-core Cavium ThunderX, Packet type "2A" server

root@docker-build-test:~# go env
GOARCH="arm64"
GOBIN=""
GOEXE=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root"
GORACE=""
GOROOT="/usr/lib/go-1.8"
GOTOOLDIR="/usr/lib/go-1.8/pkg/tool/linux_arm64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build192910166=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

root@docker-build-test:~# go test crypto/tls -bench BenchmarkThroughput

What did you expect to see?

TLS performance on my 96-core ARMv8 server faster than my laptop.

What did you see instead?

root@docker-build-test:~# go test crypto/tls -bench BenchmarkThroughput

BenchmarkThroughput/MaxPacket/1MB-96                  10         128291858 ns/op    8.17 MB/s
BenchmarkThroughput/MaxPacket/2MB-96                   5         211866625 ns/op    9.90 MB/s
BenchmarkThroughput/MaxPacket/4MB-96                   3         378852259 ns/op   11.07 MB/s
BenchmarkThroughput/MaxPacket/8MB-96                   2         715603298 ns/op   11.72 MB/s
BenchmarkThroughput/MaxPacket/16MB-96                  1        1387017225 ns/op   12.10 MB/s
BenchmarkThroughput/MaxPacket/32MB-96                  1        2713806130 ns/op   12.36 MB/s
BenchmarkThroughput/MaxPacket/64MB-96                  1        5402023727 ns/op   12.42 MB/s
BenchmarkThroughput/DynamicPacket/1MB-96              10         128462369 ns/op    8.16 MB/s
BenchmarkThroughput/DynamicPacket/2MB-96               5         211779553 ns/op    9.90 MB/s
BenchmarkThroughput/DynamicPacket/4MB-96               3         378591737 ns/op   11.08 MB/s
BenchmarkThroughput/DynamicPacket/8MB-96               2         711548140 ns/op   11.79 MB/s
BenchmarkThroughput/DynamicPacket/16MB-96              1        1385720232 ns/op   12.11 MB/s
BenchmarkThroughput/DynamicPacket/32MB-96              1        2711156682 ns/op   12.38 MB/s
BenchmarkThroughput/DynamicPacket/64MB-96              1        5378659024 ns/op   12.48 MB/s
PASS
ok      crypto/tls      36.894s

[bradfitz]

TLS performance on my 96-core

It only uses 1 core per connection, FWIW, so the 96 core part is not related.

[vielmetti]

Thanks @bradfitz for the 1 core issue. Still, these are slower single cores than I'd expect at 2 Ghz, and the resulting performance hit causes tests in gnatsd to fail at nats-io/nats-server#466 .

[bradfitz]

There's no arm64 assembly implementations of any of the crypto stuff.

amd64 has:

./vendor/golang_org/x/crypto/chacha20poly1305/chacha20poly1305_amd64.s
./vendor/golang_org/x/crypto/curve25519/mul_amd64.s
./vendor/golang_org/x/crypto/curve25519/square_amd64.s
./vendor/golang_org/x/crypto/curve25519/cswap_amd64.s
./vendor/golang_org/x/crypto/curve25519/const_amd64.s
./vendor/golang_org/x/crypto/curve25519/ladderstep_amd64.s
./vendor/golang_org/x/crypto/curve25519/freeze_amd64.s
./vendor/golang_org/x/crypto/poly1305/sum_amd64.s
./crypto/aes/gcm_amd64.s
./crypto/aes/asm_amd64.s
./crypto/elliptic/p256_asm_amd64.s

There are some for 32-bit ARM, and some for s390x and ppc64, but nothing for arm64.

[vielmetti]

A related issue is minio/sha256-simd#7 and

Based on that I'll alert @williamweixiao @fwessels @Gaillard @ncw and @harshavardhana to this report.

[williamweixiao]

The poor performance won't surprise me since crypto has not been accelerated by hardware for arm64 and we have planned to optimize AES and others this year.

[vielmetti]

Following up on this for @williamweixiao - is there a project schedule or open trackable issue for crypto improvements on golang for AES and specifically for this library?

[bradfitz]

https://golang.org/wiki/Go-Release-Cycle documents the project schedule. This would need to happen in the first three (ideally two) months of the development window.

[williamweixiao]

We plan to optimize AES for arm64 in Go1.10. But there is some uncertainty for AES and other optimizations if our patch (CL41654) of go syntax extension for SIMD can't be merged as soon as the tree is open.

[vielmetti]

For reference CL41654 can be read here:

https://go-review.googlesource.com/c/41654/

[vielmetti]

It looks like CL41654 has been merged. @williamweixiao - should this be good for AES support as mentioned in CL64490 ?

[williamweixiao]

yes, CL41654 is the base for other crypto optimisations such as CL64490, CL61550 and CL61570. As for TLS, we are also optimising AES-GCM which will be submitted soon. But upstream seems to be busy with other things and we maybe need to raise these CLs priority for reviewing and merging.