cmd/compile: limit the number of files open at the same time

[chenzhongtao]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.9rc2

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOHOSTARCH="amd64"
GOHOSTOS="linux"

What did you do?

src/cmd/compile/internal/gc/noder.go : func parseFiles(filenames []string) uint
If the length of filenames is large, it will open too many files at
the same time, it will cause EMFILE error. modify code to limit max
goroutines.

[dcheney-atlassian]

Hi,

Thanks for opening this issue. Thank you for describing where the bug occurs, but can you please tell us how you trigger it. Output from go build / go test / go install would be best.

[gopherbot]

Change https://golang.org/cl/57751 mentions this issue: cmd/compile: limit the number of files open at the same time

[chenzhongtao]

if your Go packages with enough source files more than ulimit -n, it can trigger it.
you can try by :

export GOROOT_BOOTSTRAP=/usr/lib/go-go1.9rc2
ulimit -n 32
cd /home/go/src
./all.bash

the output maybe like this:

bootstrap/cmd/compile/internal/ssa

src/bootstrap/cmd/compile/internal/ssa/decompose.go:0:0: open src/bootstrap/cmd/compile/internal/ssa/decompose.go: too many open files

this problem can be avioded if you modify max open files of system.

but in my compiling environment, my code in cifs filesystem(linux mount filesystem from windows), in this environment , concurrent open the file will lead to error: input/output error,

go build myproject

golang.org/x/sys/unix

../common/src/golang.org/x/sys/unix/sockcmsg_linux.go:0:0: open ../common/src/golang.org/x/sys/unix/sockcmsg_linux.go: input/output error

so I hope I can control the max open files by environment variable GOMAXPROCS.
the modify is :

sem := make(chan struct{}, runtime.GOMAXPROCS(0)) // Semaphore to guard against parsing too many source files simultaneously

export GOMAXPROCS=1
go build myproject // it can avoid error in my environment

[davecheney]

Can you please explain why you need to build the Go distribution with such a small ulimit.

[chenzhongtao]

small ulimit only for easy to trigger the error, but if Go packages with enough source files it will be a problem. And in special environment , concurrent open the file will lead to error: input/output error(EIO) , it lead to can not build code in such environment ,so I hope I can control the max open files by environment variable GOMAXPROCS.

[ianlancetaylor]

Closing in favor of #21621 since that one has a test case.