gollvm: unable to build etcd

[thanm]

What version of Go are you using (go version)?

$ go version
go version go1.12.2 gollvm LLVM 9.0.0svn linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

linux/amd64

What did you do?

As reported by yuanting@ict.ac.cn

$ git clone https://github.com/etcd-io/etcd.git
$ cd etcd/ && ./build

What did you expect to see?

Successful compilation

What did you see instead?

go build: when using gccgo toolchain, please pass linker flags using -gccgoflags, not -ldflags
# go.etcd.io/etcd/etcdserver/api/rafthttp
 #0 0x000055ecc4cfa34a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/home/yt/LLVMsvn/install/bin/llvm-goc+0xa6c34a)
 #1 0x000055ecc4cf8124 llvm::sys::RunSignalHandlers() (/home/yt/LLVMsvn/install/bin/llvm-goc+0xa6a124)
 #2 0x000055ecc4cf8262 SignalHandler(int) (/home/yt/LLVMsvn/install/bin/llvm-goc+0xa6a262)
 #3 0x00007fbbd49d5890 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12890)
 #4 0x000055ecc44f0940 Export::type_index(Type const*) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x262940)
 #5 0x000055ecc44f0c31 Export::write_type(Type const*) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x262c31)
 #6 0x000055ecc449f2e1 Named_type::do_export(Export*) const (/home/yt/LLVMsvn/install/bin/llvm-goc+0x2112e1)
 #7 0x000055ecc44f10ce Export::write_type_definition(Type const*, int) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x2630ce)
 #8 0x000055ecc44f1390 Export::write_types(int) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x263390)
 #9 0x000055ecc44fcea0 Export::export_globals(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, std::map, std::allocator >, Package*, std::less, std::allocator > >, std::allocator, std::allocator > const, Package*> > > const&, std::map, std::allocator >, Package*, std::less, std::allocator > >, std::allocator, std::allocator > const, Package*> > > const&, std::__cxx11::basic_string, std::allocator > const&, Import_init_set const&, Bindings const*) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x26eea0)
#10 0x000055ecc4457047 Gogo::do_exports() (/home/yt/LLVMsvn/install/bin/llvm-goc+0x1c9047)
#11 0x000055ecc4441a90 go_parse_input_files(char const**, unsigned int, bool, bool) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x1b3a90)
#12 0x000055ecc442ee69 gollvm::driver::CompileGoImpl::invokeFrontEnd() (/home/yt/LLVMsvn/install/bin/llvm-goc+0x1a0e69)
#13 0x000055ecc4436ee5 gollvm::driver::CompileGoImpl::performAction(gollvm::driver::Compilation&, gollvm::driver::Action const&, llvm::SmallVector const&, gollvm::driver::Artifact const&) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x1a8ee5)
#14 0x000055ecc442950f gollvm::driver::Driver::processAction(gollvm::driver::Action*, gollvm::driver::Compilation&, bool) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x19b50f)
#15 0x000055ecc44296af gollvm::driver::Driver::processActions(gollvm::driver::Compilation&) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x19b6af)
#16 0x000055ecc43cf403 main (/home/yt/LLVMsvn/install/bin/llvm-goc+0x141403)
#17 0x00007fbbd386db97 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:344:0
#18 0x000055ecc442130a _start (/home/yt/LLVMsvn/install/bin/llvm-goc+0x19330a)
Stack dump:
0.	Program arguments: /home/yt/LLVMsvn/install/bin/llvm-goc -c -O2 -g -m64 -fdebug-prefix-map=/tmp/go-build979597534=/tmp/go-build -gno-record-gcc-switches -fgo-pkgpath=go.etcd.io/etcd/etcdserver/api/rafthttp -o $WORK/b192/_go_.o -I $WORK/b192/_importcfgroot_ etcdserver/api/rafthttp/coder.go etcdserver/api/rafthttp/doc.go etcdserver/api/rafthttp/http.go etcdserver/api/rafthttp/metrics.go etcdserver/api/rafthttp/msg_codec.go etcdserver/api/rafthttp/msgappv2_codec.go etcdserver/api/rafthttp/peer.go etcdserver/api/rafthttp/peer_status.go etcdserver/api/rafthttp/pipeline.go etcdserver/api/rafthttp/probing_status.go etcdserver/api/rafthttp/remote.go etcdserver/api/rafthttp/snapshot_sender.go etcdserver/api/rafthttp/stream.go etcdserver/api/rafthttp/transport.go etcdserver/api/rafthttp/urlpick.go etcdserver/api/rafthttp/util.go

Needs investigation. There have been several changes and bugs in import/export lately; this looks like a new one.

[gopherbot]

Change https://golang.org/cl/185977 mentions this issue: compiler: fix bug in handling of unordered set during exporting

[thanm]

CL 185977 resolves the problem listed above, bit it looks like there is another problem hiding behind that one (I am investigating). Failure mode:

llvm-goc: /go-llvm-materialize.cpp:853: Bexpression* Llvm_backend::materializeComposite(Bexpression*): Assertion vals.size() == numElements' failed.
`

[gopherbot]

Change https://golang.org/cl/186697 mentions this issue: test: new testcase for gccgo bug

[shawndx]

Enclose a reduced case which could reproduce the issue of Llvm_backend::materializeComposite.

Given a struct type:

type FArg func(args []string) error
type Command struct {
  Name string
  Arg1 FArg
  Arg2 func(args []string) error
}

The named base of FArg is set to "func(args []string)error" which happens to be the type of field 'Arg2', converting the backend representation of the type of Arg1 has a side-effect which sets the btype of Arg2's type, such side-effect result in the post-processing of 'Arg2' being bypassed and remaining 'Command' in 'unresolved placeholder' status, so its opaque llvm type fails to be concretized. So far the issue could be reproduced via 'importing' type definitions only as the exporter seems to do type definitions sharing, perhaps there are other scenarios undiscovered

Tried a fix locally which adds place-holder pointer types created for function type to a list (similar to Type::placeholder_pointers, or they can share the same list) then lets finish_pointer_types takes care the concretizing, it could fix the crash of building etcd (a few errors of undefined symbol observed), wonder there might be a better solution from the perspective of type sharing.

[shawndx]

Seems that neither .go nor .txt is supported, enclosing the code here.

//t.go
package Args 

type FArg func(args []string) error

type Command struct {
  Name string
  Arg1 FArg 
  Arg2 func(args []string) error 
}

// main.go
package main

import (
	"fmt"
        "mypkg/Args"
)

var (
  cmd = &Args.Command {
    Name: "test",
  }
)

func main() {
  fmt.Println("test main")
}

/* put t.go under $GOPATH
    % go build mypkg/Args 
   % go build main.go

#4  0x0000555555ed47f0 in Llvm_backend::materializeComposite (this=0x55555d054350, comExpr=0x55555d07e710) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm-materialize.cpp:860
#5  0x0000555555edb8e1 in MaterializeVisitor::visitNodePost (this=0x7fffffffd5a0, node=0x55555d07e710) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm-materialize.cpp:1967
#6  0x0000555555ee04c8 in UpdatingNodeWalker<MaterializeVisitor>::walk (this=0x7fffffffd528, node=0x55555d07e710) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm-bnode.h:514
#7  0x0000555555ede344 in update_walk_nodes<MaterializeVisitor> (root=0x55555d07e710, vis=...) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm-bnode.h:530
#8  0x0000555555ed8df5 in Llvm_backend::materialize (this=0x55555d054350, expr=0x55555d07e710, lvalueContext=VE_rvalue) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm-materialize.cpp:2037
#9  0x0000555555de582f in Llvm_backend::makeInitStatement (this=0x55555d054350, bfunction=0x55555d0ce0e0, var=0x55555d07e5e0, init=0x55555d07e710) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm.cpp:1759
#10 0x0000555555de572a in Llvm_backend::init_statement (this=0x55555d054350, bfunction=0x55555d0ce0e0, var=0x55555d07e5e0, init=0x55555d07e710) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm.cpp:1742
#11 0x0000555555de8109 in Llvm_backend::temporary_variable (this=0x55555d054350, function=0x55555d0ce0e0, bblock=0x55555d0a0c00, btype=0x55555d0a74b0, binit=0x55555d07e710, is_address_taken=true, location=...,
    pstatement=0x7fffffffd800) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm.cpp:2305
#12 0x0000555555e813c6 in Heap_expression::do_get_backend (this=0x55555d0642f0, context=0x7fffffffdc30) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/expressions.cc:16780
#13 0x0000555555e4d5c1 in Expression::get_backend (this=0x55555d0642f0, context=0x7fffffffdc30) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/expressions.cc:543
#14 0x0000555555e5523b in Type_conversion_expression::do_get_backend (this=0x55555d0cd1f0, context=0x7fffffffdc30) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/expressions.cc:3960
#15 0x0000555555e4d5c1 in Expression::get_backend (this=0x55555d0cd1f0, context=0x7fffffffdc30) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/expressions.cc:543
#16 0x0000555555d6e0d6 in Temporary_statement::do_get_backend (this=0x55555d0cd300, context=0x7fffffffdc30) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/statements.cc:628
#17 0x0000555555d6cb25 in Statement::get_backend (this=0x55555d0cd300, context=0x7fffffffdc30) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/statements.cc:185
#18 0x0000555555cf945d in Block::get_backend (this=0x55555d0cd290, context=0x7fffffffdd00) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/gogo.cc:7065
#19 0x0000555555cfbb11 in Variable::get_init_block (this=0x55555d063d80, gogo=0x55555d05cdd0, function=0x55555d0cddf0, var_decl=0x55555d0a2bf0) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/gogo.cc:7821
#20 0x0000555555ce7eaa in Gogo::write_globals (this=0x55555d05cdd0) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/gogo.cc:1687
#21 0x0000555555cdfac4 in go_write_globals () at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/go.cc:194
#22 0x0000555555ccba93 in gollvm::driver::CompileGoImpl::invokeFrontEnd (this=0x55555d001d90) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/driver/CompileGo.cpp:811
#23 0x0000555555cc89c7 in gollvm::driver::CompileGoImpl::performAction (this=0x55555d001d90, compilation=..., jobAction=..., inputArtifacts=..., output=...)
    at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/driver/CompileGo.cpp:192
#24 0x0000555555ccc8fe in gollvm::driver::CompileGo::performAction (this=0x55555d039dc0, compilation=..., jobAction=..., inputArtifacts=..., output=...) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/driver/CompileGo.cpp:1022
#25 0x0000555555cbcbe5 in gollvm::driver::Driver::processAction (this=0x7fffffffe2d0, act=0x55555d03a3b0, compilation=..., lastAct=false) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/driver/Driver.cpp:546
#26 0x0000555555cbcda7 in gollvm::driver::Driver::processActions (this=0x7fffffffe2d0, compilation=...) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/driver/Driver.cpp:568

*/

[thanm]

Thanks @shawn-xdji for the analysis and the reduced test case. I agree with your analysis, the problem is that the FE is manufacturing a placeholder but then never finalizing it. There was a similar problem a while back fixed in https://go-review.googlesource.com/c/gofrontend/+/51131, I think the fix for this bug is to generalize/extend that change. I will send a CL.

[gopherbot]

Change https://golang.org/cl/191743 mentions this issue: test: new testcase for gollvm bug

[gopherbot]

Change https://golang.org/cl/191744 mentions this issue: compiler: generalize cleanup of unresolved placeholder pointer types