cmd/link, cmd/cgo: internal linking fails if CGO_CFLAGS enables LTO

[mwhudson]

What version of Go are you using (go version)?

$ go version
go version devel +9eef49cfa6 Mon Jan 4 17:59:30 2021 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes, although the error is a bit different.

What operating system and processor architecture are you using (go env)?

Ubuntu 20.04 or 20.10 (current devel series).

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mwhudson/.cache/go-build"
GOENV="/home/mwhudson/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/mwhudson/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mwhudson/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/mwhudson/src/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/mwhudson/src/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="devel +9eef49cfa6 Mon Jan 4 17:59:30 2021 +0000"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1258774523=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I ran a trivial os/user-using program with CFLAGS that caused gcc to create object files ready for link time optimization:

mwhudson@anduril:~/tmp$ cat user.go
package main

import (
	"fmt"
	"os/user"
)

func main() {
	u, _ := user.Current()
	fmt.Printf("username: %s\n", u.Name)
}
mwhudson@anduril:~/tmp$ CGO_CFLAGS='-g -flto -ffat-lto-objects' ~/src/go/bin/go run user.go 

What did you expect to see?

username: Michael Hudson-Doyle

What did you see instead?

Errors of the form "loadelf: /home/mwhudson/.cache/go-build/1c/1cfde03a90c420f612c840a344f1de135898b62010e420c7b14c3e353c13c3d9-d(_x001.o): 67693: sym#26: ignoring symbol in section 5 (type 0)" and then a panic:

# command-line-arguments
loadelf: /home/mwhudson/.cache/go-build/1c/1cfde03a90c420f612c840a344f1de135898b62010e420c7b14c3e353c13c3d9-d(_x001.o): 67693: sym#26: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/1c/1cfde03a90c420f612c840a344f1de135898b62010e420c7b14c3e353c13c3d9-d(_x002.o): 67704: sym#44: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/1c/1cfde03a90c420f612c840a344f1de135898b62010e420c7b14c3e353c13c3d9-d(_x003.o): 67712: sym#32: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/1c/1cfde03a90c420f612c840a344f1de135898b62010e420c7b14c3e353c13c3d9-d(_x004.o): 67719: sym#29: ignoring symbol in section 4 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x001.o): 67723: sym#24: ignoring symbol in section 4 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x002.o): 67730: sym#29: ignoring symbol in section 4 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x003.o): 67735: sym#26: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x004.o): 67741: sym#27: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x005.o): 67751: sym#38: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x006.o): 67759: sym#31: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x007.o): 67764: sym#27: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x008.o): 67769: sym#27: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x009.o): 67774: sym#26: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x010.o): 67779: sym#26: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x011.o): 67785: sym#27: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x012.o): 67790: sym#34: ignoring symbol in section 5 (type 0)
_cgo_callers: relocation target x_cgo_callers not defined
_cgo_init: relocation target x_cgo_init not defined
_cgo_mmap: relocation target x_cgo_mmap not defined
_cgo_munmap: relocation target x_cgo_munmap not defined
_cgo_notify_runtime_init_done: relocation target x_cgo_notify_runtime_init_done not defined
/home/mwhudson/src/go/pkg/tool/linux_amd64/link: too many errors
unexpected fault address 0x7f310460c000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7f310460c000 pc=0x46db13]
goroutine 129 [running]:

runtime.throw(0x6dac2e, 0x5)

/home/mwhudson/src/go/src/runtime/panic.go:1112 +0x72 fp=0xc00061fd18 sp=0xc00061fce8 pc=0x436e12

runtime.sigpanic()

/home/mwhudson/src/go/src/runtime/signal_unix.go:737 +0x268 fp=0xc00061fd50 sp=0xc00061fd18 pc=0x44e4a8

runtime.memmove(0x7f310460baa0, 0x7f3104eb29b1, 0xa6a)

/home/mwhudson/src/go/src/runtime/memmove_amd64.s:367 +0x433 fp=0xc00061fd58 sp=0xc00061fd50 pc=0x46db13

cmd/link/internal/ld.(*OutBuf).WriteSym(0xc00111e000, 0xc000128000, 0x3925, 0x2, 0xc0000b60c5, 0x1)

/home/mwhudson/src/go/src/cmd/link/internal/ld/outbuf.go:300 +0x2bc fp=0xc00061fde8 sp=0xc00061fd58 pc=0x5ea7fc

cmd/link/internal/ld.writeBlock(0xc00010e000, 0xc00111e000, 0xc000128000, 0xc000eae000, 0x5e2, 0x6a0, 0x40daa0, 0x9b4a5, 0xc0000b60c5, 0x1, ...)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:952 +0x1a9 fp=0xc00061ff10 sp=0xc00061fde8 pc=0x589149

cmd/link/internal/ld.writeBlocks.func1(0xc00010e000, 0xc001114020, 0xc0000f40e0, 0xc00111e000, 0xc000128000, 0xc000eae000, 0x5e2, 0x6a0, 0x401000, 0x9b4a5, ...)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:909 +0xb1 fp=0xc00061ff78 sp=0xc00061ff10 pc=0x608311

runtime.goexit()

/home/mwhudson/src/go/src/runtime/asm_amd64.s:1367 +0x1 fp=0xc00061ff80 sp=0xc00061ff78 pc=0x46c901

created by cmd/link/internal/ld.writeBlocks

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:908 +0x565
goroutine 1 [semacquire]:

sync.runtime_Semacquire(0xc0011a4038)

/home/mwhudson/src/go/src/runtime/sema.go:56 +0x45

sync.(*WaitGroup).Wait(0xc0011a4030)

/home/mwhudson/src/go/src/sync/waitgroup.go:130 +0x65

cmd/link/internal/ld.asmb(0xc00010e000)

/home/mwhudson/src/go/src/cmd/link/internal/ld/asmb.go:73 +0x235

cmd/link/internal/ld.Main(0x87d2e0, 0x20, 0x20, 0x1, 0x7, 0x10, 0x0, 0x0, 0x6e663e, 0x1b, ...)

/home/mwhudson/src/go/src/cmd/link/internal/ld/main.go:331 +0x140e

main.main()

/home/mwhudson/src/go/src/cmd/link/main.go:68 +0x258
goroutine 24 [semacquire]:

sync.runtime_Semacquire(0xc001230008)

/home/mwhudson/src/go/src/runtime/sema.go:56 +0x45

sync.(*WaitGroup).Wait(0xc001230000)

/home/mwhudson/src/go/src/sync/waitgroup.go:130 +0x65

cmd/link/internal/ld.writeBlocks(0xc00010e000, 0xc00119eb60, 0xc0000f40e0, 0xc000128000, 0xc0011c2218, 0x4e5, 0x4e5, 0x53f000, 0x15ca0, 0x8bd260, ...)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:924 +0x736

cmd/link/internal/ld.writeDatblkToOutBuf(...)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:1006

cmd/link/internal/ld.datblk(0xc00010e000, 0xc00119eb60, 0x53f000, 0x15ca0)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:994 +0xad

cmd/link/internal/ld.writeParallel.func1(0xc0011a4030, 0x6f23a8, 0xc00010e000, 0xc00119eb60, 0x53f000, 0x15ca0)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:988 +0x7a

created by cmd/link/internal/ld.writeParallel

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:986 +0x152
goroutine 23 [runnable]:

cmd/link/internal/loader.(*Loader).SubSym(...)

/home/mwhudson/src/go/src/cmd/link/internal/loader/loader.go:1740

cmd/link/internal/loader.(*Loader).AttrSubSymbol(...)

/home/mwhudson/src/go/src/cmd/link/internal/loader/loader.go:1161

cmd/link/internal/ld.writeBlocks(0xc00010e000, 0xc00119eaf0, 0xc0000f40e0, 0xc000128000, 0xc0011b6008, 0x1d27, 0x1d27, 0x49d000, 0xa1028, 0x8bd260, ...)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:858 +0x785

cmd/link/internal/ld.writeDatblkToOutBuf(...)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:1006

cmd/link/internal/ld.datblk(0xc00010e000, 0xc00119eaf0, 0x49d000, 0xa1028)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:994 +0xad

cmd/link/internal/ld.writeParallel.func1(0xc0011a4030, 0x6f23a8, 0xc00010e000, 0xc00119eaf0, 0x49d000, 0xa1028)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:988 +0x7a

created by cmd/link/internal/ld.writeParallel

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:986 +0x152
goroutine 21 [semacquire]:

sync.runtime_Semacquire(0xc001114028)

/home/mwhudson/src/go/src/runtime/sema.go:56 +0x45

sync.(*WaitGroup).Wait(0xc001114020)

/home/mwhudson/src/go/src/sync/waitgroup.go:130 +0x65

cmd/link/internal/ld.writeBlocks(0xc00010e000, 0xc00119ea10, 0xc0000f40e0, 0xc000128000, 0xc000eae000, 0x5e2, 0x6a0, 0x401000, 0x9b4a5, 0xc0000b60c5, ...)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:924 +0x736

cmd/link/internal/ld.CodeblkPad(...)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:834

cmd/link/internal/ld.asmb.func1(0xc00010e000, 0xc00119ea10, 0x401000, 0x9b4a5)

/home/mwhudson/src/go/src/cmd/link/internal/ld/asmb.go:39 +0xd3

cmd/link/internal/ld.writeParallel.func1(0xc0011a4030, 0x6f23a0, 0xc00010e000, 0xc00119ea10, 0x401000, 0x9b4a5)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:988 +0x7a

created by cmd/link/internal/ld.writeParallel

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:986 +0x152
goroutine 113 [runnable]:

os.Exit(0x2)

/home/mwhudson/src/go/src/os/proc.go:62 +0x73

cmd/link/internal/ld.Exit(0x2)

/home/mwhudson/src/go/src/cmd/link/internal/ld/util.go:31 +0x8a

cmd/link/internal/ld.Exitf(0x6dfa23, 0xf, 0x0, 0x0, 0x0)

/home/mwhudson/src/go/src/cmd/link/internal/ld/util.go:38 +0x106

cmd/link/internal/ld.afterErrorAction()

/home/mwhudson/src/go/src/cmd/link/internal/ld/util.go:49 +0x7a

cmd/link/internal/loader.(*ErrorReporter).Errorf(0xc00010e010, 0xa5e0, 0x6e8fde, 0x20, 0xc001225ae0, 0x1, 0x1)

/home/mwhudson/src/go/src/cmd/link/internal/loader/loader.go:2619 +0x143

cmd/link/internal/ld.(*ErrorReporter).errorUnresolved(0xc00010e010, 0xc000128000, 0xa5e0, 0xa5df)

/home/mwhudson/src/go/src/cmd/link/internal/ld/errors.go:63 +0x4e9

cmd/link/internal/ld.(*relocSymState).relocsym(0xc001225ea0, 0xa5e0, 0x7f310474b4c0, 0x8, 0x7b40)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:199 +0x338a

cmd/link/internal/ld.writeBlock(0xc00010e000, 0xc001232000, 0xc000128000, 0xc0011c2218, 0x4e5, 0x4e5, 0x54d4c0, 0x15ca0, 0x8bd260, 0x200, ...)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:953 +0x1e7

cmd/link/internal/ld.writeBlocks.func1(0xc00010e000, 0xc001230000, 0xc0000f40e0, 0xc001232000, 0xc000128000, 0xc0011c2218, 0x4e5, 0x4e5, 0x53f000, 0x15ca0, ...)

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:909 +0xb1

created by cmd/link/internal/ld.writeBlocks

/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:908 +0x565

[mwhudson]

So what's going on here is that these cflags make gcc include symbols in object files that cmd/link/internal/loadelf doesn't like, causing it to abandon parsing the file and so not find symbols it needs to find.

loadelf could be taught to ignore these symbols, I guess, or just skip all symbols it doesn't know how to handle, but really with my Ubuntu/Debian golang maintainer hat on, I'd rather there was some way of preventing the linker ever attempt internal linking if any cgo is used at all, at least during package builds. I don't want to disable internal linking entirely -- that would mean all go executables, even those not using cgo at all, would be dynamically linked, which seems a bit much of a change. I guess I can do this by patching internalpkg in cmd/link/internal/ld/lib.go to be empty but that also seems a bit much. I don't think there is an existing flag or environment variable that does this.

[ianlancetaylor]

We want to use internal linking for Go programs that do not explicitly use cgo. That includes Go programs that use the standard packages net and os/user, even though those packages do use cgo by default. So we do want internal linking to work for programs that use cgo, provided the only packages that use cgo are those in the standard library.

But we could consider using external linking by default if CGO_CFLAGS contains anything other than -g and -O options, or something along those lines.

[mwhudson]

That would work. Can the linker assume that CGO_CFLAGS in its environment is the one that was used to build the object files it is processing though? The cflags are recorded in the _cgo_flags but that's only used by the gccgo support in cmd/go currently, I'm not sure the linker can read them from anywhere?

[ianlancetaylor]

It should be the case that if the linker sees CGO_CFLAGS in the environment, then it was used to build the object files.

But what about flags specified with #cgo CFLAGS:? It seems to me that they ought to have an identical effect. And I don't know if the linker can see those at present.

[mwhudson]

It should be the case that if the linker sees CGO_CFLAGS in the environment, then it was used to build the object files.

It's certainly good enough for my use case!

But what about flags specified with #cgo CFLAGS:? It seems to me that they ought to have an identical effect. And I don't know if the linker can see those at present.

I wondered about that, but any use of cgo outside the standard library triggers external linking anyway so this isn't an issue (I think we can be sure no-one will accidentally add "#cgo CFLAGS: -flto -ffat-lto-objects" to runtime/cgo ...)

[ianlancetaylor]

Good point.

[gopherbot]

Change https://golang.org/cl/281314 mentions this issue: cmd/link: check CGO_CFLAGS for non -g/-O options before internal linking

[cherrymui]

I think only accepting -g and -O is too narrow. On some platforms there are some flags we use. For example, -march=armv6 is used when we build ARMv6 release. On Darwin we pass some flag to specify the macOS version. Listing all allowed flags seems complicated. Compared to this, maybe -linkmode=external-if-cgo is more desirable.

[gopherbot]

Change https://golang.org/cl/322614 mentions this issue: cmd/link, cmd/cgo: support -flto in CFLAGS