Sourcegraph Docker container crashes on startup

[beyang]

Filing this as an issue in case we run into this with customers. @mrnugget reported:

I've just tried to start up our sourcegraph/server image (3.15.1 and insiders) on Ubuntu 20.04 and it crashes with:

runtime: mlock of signal stack failed: 12
runtime: increase the mlock limit (ulimit -l) or
runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
fatal error: mlock failed

Turns out there's an issue with Go 1.14 and buggy Linux kernel versions: golang/go#37436
The end result is this wiki page: https://github.com/golang/go/wiki/LinuxKernelSignalVectorBug

I had to use docker run --ulimit memlock=67108864 ... to get it to work. (It's --ulimit and not -ulimit as documented on the wiki page).

If I read the whole issue correctly, the kernel in Ubuntu 20.04 should be patched. It's 5.4.0, but some in the issue say that Ubuntu patched the kernel without bumping the version or something?

[dcoles]

@beyang I've also run into this issue when trying to use the latest Docker image with SourceGraph 3.15.1 on Ubuntu 19.10 and 20.04.

Since Ubuntu 20.04 is a LTS release, they pick a stable kernel and will back port critical fixes to this kernel. However this means that it break's Go's "fixed in 5.3.15, 5.4.2, and 5.5" check.

I've tested Ubuntu 20.04 (5.4.0-29-generic x86_64) using the bug test application provided and confirmed that this issue does not affect Ubuntu 20.04.

The check was made less restrictive in Go 1.15 and backported to Go 1.14.1 so Go will no longer aggressively abort on any possibly affected system.

It looks like the sourcegraph/server 3.15.1 was built against Go 1.14. Would it be possible to cut a new release that uses Go 1.14.1 or newer?

[dcoles]

The new 3.16.0 release is also affected (appears to still be built using Go 1.14).

[emidoots]

So sorry, this fell through the cracks on our end -- this was supposed to be fixed in Sourcegraph 3.16.0 for sure. Working on getting a build you can use ASAP, sorry for the trouble @dcoles !

[emidoots]

Hey @dcoles here is a fresh build of 3.16 built with Go 1.14.3, could you give it a try and confirm for me it fixes the issue for you?

docker run --publish 7080:7080 --publish 127.0.0.1:3370:3370 --rm --volume ~/.sourcegraph/config:/etc/sourcegraph --volume ~/.sourcegraph/data:/var/opt/sourcegraph sourcegraph/server:3.16-insiders@sha256:d74f7ed843cc842520c463f9869a4946a73f717ad8b0f0574acc0214236a3ece

If that resolved the problem for you, we'll publish this under a 3.16.1 tag in the next week or so (but it'd just be identical to the above, so it's safe to run the above in prod if you like).

[ryan-blunden]

here is a fresh build of 3.16 built with Go 1.14.3

@slimsag I just tested this on the AWS 20.04 community image and it worked

[emidoots]

Great!

[dcoles]

@slimsag I can also confirm it now works on 20.04! Thanks for the quick fix!

[emidoots]

3.16.1 which includes this patch has been released, thanks for all the feedback!