x/tools/godoc: memory grows exponentially if symlink loop exists in $GOPATH

[kevinburke]

Please answer these questions before submitting your issue. Thanks!

What did you do?

Compile and run godoc as godoc -http=:6060.

What did you expect to see?

I expected the program to use a reasonable amount of memory; on my machine it gets to about 200 MB and then remains constant.

What did you see instead?

The program's memory consumption grew in an unbounded way; up to 30 GB RAM before I killed it. In addition, a number of error output lines (thousands) were streamed to the console:

$ godoc -http=:6060
2017/07/17 22:51:20 ignoring symlink: stat /Users/kevin/src/chromium.googlesource.com/infra/infra/appengine_module/gae_event_mon/checkouts.py: no such file or directory
2017/07/17 22:51:20 ignoring symlink: stat /Users/kevin/src/chromium.googlesource.com/infra/infra/appengine_module/gae_event_mon/config.py: no such file or directory
2017/07/17 22:51:20 ignoring symlink: stat /Users/kevin/src/chromium.googlesource.com/infra/infra/appengine_module/gae_event_mon/httplib2_utils.py: no such file or direct

Does this issue reproduce with the latest release (go1.8.3)?

Yes

System details

go version devel +e9b9dfe3f7 Tue Jul 18 04:42:09 2017 +0000 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/kevin"
GORACE=""
GOROOT="/Users/kevin/go"
GOTOOLDIR="/Users/kevin/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/sf/fsn3_vgd0n98r0jb86bgp83r0000gn/T/go-build558191022=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOROOT/bin/go version: go version devel +e9b9dfe3f7 Tue Jul 18 04:42:09 2017 +0000 darwin/amd64
GOROOT/bin/go tool compile -V: compile version devel +e9b9dfe3f7 Tue Jul 18 04:42:09 2017 +0000
uname -v: Darwin Kernel Version 16.6.0: Fri Apr 14 16:21:16 PDT 2017; root:xnu-3789.60.24~6/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.12.5
BuildVersion:	16F73
lldb --version: lldb-370.0.42
  Swift-3.1

I've bisected the commits in x/tools and https://go-review.googlesource.com/c/45096/ is the first one where the program's memory begins growing.

[kevinburke]

Reading over that CL I don't see anything obvious causing the problem... maybe ReadDir is calling itself in a loop?

So I think the next step may be to back it out, or run the program with -benchmem on and see where the allocations are occurring.

[mvdan]

Is there a symlink loop anywhere?

[RLH]

What does GODEBUG=gctrace=1 ./myAppReproducer say. This will tell us if it is growth in the Go heap or growth somewhere else.

…

On Tue, Jul 18, 2017 at 2:12 AM, Daniel Martí ***@***.***> wrote: Is there a symlink loop anywhere? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#21061 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA7Wn8sa93HFBwAUCcjFHPWYJVJcAIl3ks5sPGjRgaJpZM4Oa4Nb> .

[kevinburke]

https://gist.github.com/kevinburke/d1e3c65ecbb486862669f6889a939598

[kevinburke]

I think there's a symlink loop somewhere because if I do this:

diff --git a/godoc/corpus.go b/godoc/corpus.go
index f2c7ebbf..61408745 100644
--- a/godoc/corpus.go
+++ b/godoc/corpus.go
@@ -147,7 +147,7 @@ func (c *Corpus) Init() error {
 }
 
 func (c *Corpus) initFSTree() error {
-	dir := c.newDirectory(pathpkg.Join("/", c.testDir), -1)
+	dir := c.newDirectory(pathpkg.Join("/", c.testDir), 30)
 	if dir == nil {
 		return errors.New("godoc: corpus fstree is nil")
 	}

The memory usage settles at ~500 MB.

That said, is a symlink loop that uncommon, and can we have godoc error (instead of growing RAM use infinitely) if it finds one?

[bradfitz]

Did https://go-review.googlesource.com/c/45096/ help or hurt?

[kevinburke]

Oh, interesting, these are pretty common Go libraries, I think.

gfind: File system loop detected; ‘./src/github.com/coreos/etcd/cmd/etcd’ is part of the same file system loop as ‘./src/github.com/coreos/etcd’.
gfind: File system loop detected; ‘./src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd’ is part of the same file system loop as ‘./src/github.com/coreos/etcd’.
gfind: File system loop detected; ‘./src/github.com/jteeuwen/go-bindata/testdata/symlinkRecursiveParent/symlinkTarget’ is part of the same file system loop as ‘./src/github.com/jteeuwen/go-bindata/testdata/symlinkRecursiveParent’.
gfind: File system loop detected; ‘./src/github.com/kubernetes/kops/vendor/github.com/jteeuwen/go-bindata/testdata/symlinkRecursiveParent/symlinkTarget’ is part of the same file system loop as ‘./src/github.com/kubernetes/kops/vendor/github.com/jteeuwen/go-bindata/testdata/symlinkRecursiveParent’.

[kevinburke]

That change hurt. Running godoc on the commit immediately before it, memory settles at 200MB. Recompiling with that change and it grows exponentially.

[kevinburke]

I can take a go at fixing this... does anyone have any ideas about a solution? The dumb one would be to create a map of symlinks, e.g. map[existing]new. Any time we come across a new symlink, if the file it points to is in the "existing" map, we have a loop, and should stop following those links, and/or panic. I don't think we should panic because there are legitimate reasons to have symlink loops in your file tree, see above.

Another (much easier to implement) solution would be to cap godoc directory recursion at, say, a depth of 20 or 30.

[bradfitz]

How can I repro?

For any given directory, we should only read it once, regardless of the path we got to it. Subsequent reads can just return empty results probably.

[kevinburke]

It should be enough to clone the most recent version of go-bindata into your GOPATH, then recompile godoc with the tip of x/tools master, and restart it, you should see the memory just grow and grow. go-bindata contains a recursive symlink in the testdata directory.

gfind: File system loop detected; ‘./src/github.com/jteeuwen/go-bindata/testdata/symlinkRecursiveParent/symlinkTarget’ is part of the same file system loop as ‘./src/github.com/jteeuwen/go-bindata/testdata/symlinkRecursiveParent’.

[ALTree]

I think this may be a dup of #17344. There are memprofiles too, there.

[kevinburke]

I've been running godoc locally for about a year now, I've had go-bindata (and the other symlink loops) locally for over a year and I only saw it eat all of my RAM in the last week or so. I can also make the problem go away by reverting the CL mentioned above and reinstalling so I think this is a recent issue.

On Wed, Jul 19, 2017 at 14:03 Alberto Donizetti ***@***.***> wrote: I think this may be a dup of #17344 <#17344>. There are memprofiles too, there. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#21061 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOSI65x2vTccRjIFD68W408-HsnIAEWks5sPm8CgaJpZM4Oa4Nb> .

--

…

-- Kevin Burke 925.271.7005 | kev.inburke.com