cmd/link: statically compiling windows/amd64 binary on linux fails on 1.5

[jessfraz]

Previously on go 1.4.3 this used to work:

export CC=x86_64-w64-mingw32-gcc
export CGO_ENABLED=1
export GOOS=windows
export GOARCH=amd64

go build -o bundles/1.9.0-dev/cross/windows/amd64/docker-1.9.0-dev.exe -a -tags 'netgo static_build sqlite_omit_load_extension apparmor selinux daemon journald btrfs_noversion libdm_no_deferred_remove' -installsuffix netgo -ldflags '-w -extldflags "-static" -extld=x86_64-w64-mingw32-gcc' ./docker

Now when trying to run the same thing I get:

# github.com/docker/docker/docker
/usr/local/go/pkg/tool/linux_amd64/link: running x86_64-w64-mingw32-gcc failed: exit status 1
/usr/bin/x86_64-w64-mingw32-ld: i386 architecture of input file `/tmp/go-link-623415307/000000.o' is incompatible with i386:x86-64 output
collect2: error: ld returned 1 exit status

That's a super gross command I know, but I can make a small go program and try to compile statically for windows on linux and see the same thing.

[jessfraz]

Also this is go 1.5.1

[davecheney]

Two questions

1. Isn't cross compilation always statically linked, ie, no cgo by default unless you work really hard at it
2. The error message seems to be saying your trying to link a 32 bit .o using a 64 bit linker, the mingw32 would agree

[jessfraz]

well its weird because the x86_64-w64 in the prefix is supposed to mean 64bit, its poorly named trying to find link to what I read when I originally wrote the binary compile script

[jessfraz]

so from what I can tell for 32bit i686-w64-mingw32-gcc is used and for 64bit x86_64-w64-mingw32-gcc is used.

[mwhudson]

Still, /tmp/go-link-623415307/000000.o, which I think is a file produced by cgo, seems to be 32 bit when it needs to be 64-bit. Can you run go build -x and stick the output in a gist?

[jessfraz]

will do! also this is the docs for mingw http://sourceforge.net/p/mingw-w64/wiki2/Cross%20Win32%20and%20Win64%20compiler/#building-binutils-cross-toolchain

[jessfraz]

https://gist.github.com/anonymous/b6c4010d380d4a80f0f8

[jessfraz]

FWIW this is the output with go 1.4.3 https://gist.github.com/anonymous/5c991873a89319fc958f

[mwhudson]

Hm, well one thing that's iirc changed is that there is external linking support for windows now. Can you try adding -v to -ldflags? And maybe -linkmode=internal too?

[davecheney]

Hm, well one thing that's iirc changed is that there is external linking support for windows now. Can you try adding -v to -ldflags? And maybe -linkmode=internal too?

That would explain it. The previous 1.4 behaviour is "schmeh, i'll ignore your suggestion for linking and do it myself".

[jessfraz]

yay that works omg thanks for the speedy help \o/

[mwhudson]

This still looks like a bug in something or other, fwiw: external linking should work when cross compiling.

[davecheney]

@jfrazelle could you run file /tmp/go-link-623415307/000000.o or similar (not the microsoft tools) to solve the argument about which arch this binary is? There might be something bone headed in the go tool that is passing -m32 or something when it should not.

[jessfraz]

totally will do!

On Tue, Oct 27, 2015 at 6:52 PM, Dave Cheney [email protected]
wrote:

@jfrazelle https://github.com/jfrazelle could you run file
/tmp/go-link-623415307/000000.o or similar (not the microsoft tools) to
solve the argument about which arch this binary is? There might be
something bone headed in the go tool that is passing -m32 or something
when it should not.

—
Reply to this email directly or view it on GitHub
#13070 (comment).

Jessie Frazelle
4096R / D4C4 DD60 0D66 F65A 8EFC 511E 18F3 685C 0022 BFF3
pgp.mit.edu http://pgp.mit.edu/pks/lookup?op=get&search=0x18F3685C0022BFF3

[jessfraz]

this is probably a dumb question but it is deleting that file before i can run file on it... is there something i need to pass??

[davecheney]

-work

On Wed, 28 Oct 2015, 13:13 Jess Frazelle [email protected] wrote:

this is probably a dumb question but it is deleting that file before i can
run file on it... is there something i need to pass??

—
Reply to this email directly or view it on GitHub
#13070 (comment).

[jessfraz]

wow i swear i knew that :shame:

[jessfraz]

so its still deleting it but keeping the go-build-xxxx dir instead of the link dir :(

[mwhudson]

add -tmpdir /home/goobjs or something like that to -ldflags (after creating the directory!)

[jessfraz]

---> Making bundle: binary (in bundles/1.9.0-dev/binary)
Building: bundles/1.9.0-dev/binary/docker-1.9.0-dev.exe
# github.com/docker/docker/docker
/usr/local/go/pkg/tool/linux_amd64/link: running x86_64-w64-mingw32-gcc failed: exit status 1
/usr/bin/x86_64-w64-mingw32-ld: i386 architecture of input file `/butts/000000.o' is incompatible with i386:x86-64 output
collect2: error: ld returned 1 exit status

root@dfd7314e5304:/go/src/github.com/docker/docker# ^C
root@dfd7314e5304:/go/src/github.com/docker/docker# file /butts/000000.o
/butts/000000.o: Targa image data - Map 1 x

[davecheney]

Ain't nobody gonna have time to link that!

On Wed, 28 Oct 2015, 13:40 Jess Frazelle [email protected] wrote:

---> Making bundle: binary (in bundles/1.9.0-dev/binary)
Building: bundles/1.9.0-dev/binary/docker-1.9.0-dev.exe

github.com/docker/docker/docker

/usr/local/go/pkg/tool/linux_amd64/link http://github.com/docker/docker/docker/usr/local/go/pkg/tool/linux_amd64/link: running x86_64-w64-mingw32-gcc failed: exit status 1
/usr/bin/x86_64-w64-mingw32-ld: i386 architecture of input file `/butts/000000.o' is incompatible with i386:x86-64 output
collect2: error: ld returned 1 exit status

root@dfd7314e5304:/go/src/github.com/docker/docker# ^C
root@dfd7314e5304:/go/src/github.com/docker/docker# file /butts/000000.o
/butts/000000.o: Targa image data - Map 1 x

—
Reply to this email directly or view it on GitHub
#13070 (comment).

[jessfraz]

Lol I was thinking I have no idea what this means

[davecheney]

I'm wondering if that .o is corrupt.

On Wed, 28 Oct 2015, 14:21 Jess Frazelle [email protected] wrote:

Lol I was thinking I have no idea what this means

—
Reply to this email directly or view it on GitHub
#13070 (comment).

[minux]

Could you please make that 00000.o available somewhere so that i can take a look? Thanks.

[jessfraz]

uploaded to https://s3.j3ss.co/tmp/linker-artifacts.tar.gz

On Wed, Oct 28, 2015 at 12:14 PM, Minux Ma [email protected] wrote:

Could you please make that 00000.o available somewhere so that i can take a
look? Thanks.

—
Reply to this email directly or view it on GitHub
#13070 (comment).

Jessie Frazelle
4096R / D4C4 DD60 0D66 F65A 8EFC 511E 18F3 685C 0022 BFF3
pgp.mit.edu http://pgp.mit.edu/pks/lookup?op=get&search=0x18F3685C0022BFF3

[alexbrainman]

Your object file has 1 section (.rsrc) with some resources (icons, bitmaps and similar) in it:

C:\dev\go\src>objdump -x u:\butts\000000.o

u:\butts\000000.o:     file format pe-i386
u:\butts\000000.o
architecture: i386, flags 0x00000039:
HAS_RELOC, HAS_DEBUG, HAS_SYMS, HAS_LOCALS
start address 0x00000000

Characteristics 0x104
        line numbers stripped
        32 bit words

Time/Date               Thu Jan 01 11:00:00 1970
Magic                   0000
MajorLinkerVersion      0
MinorLinkerVersion      0
SizeOfCode              00000000
SizeOfInitializedData   00000000
SizeOfUninitializedData 00000000
AddressOfEntryPoint     00000000
BaseOfCode              00000000
BaseOfData              00000000
ImageBase               00000000
SectionAlignment        00000000
FileAlignment           00000000
MajorOSystemVersion     0
MinorOSystemVersion     0
MajorImageVersion       0
MinorImageVersion       0
MajorSubsystemVersion   0
MinorSubsystemVersion   0
Win32Version            00000000
SizeOfImage             00000000
SizeOfHeaders           00000000
CheckSum                00000000
Subsystem               00000000        (unspecified)
DllCharacteristics      00000000
SizeOfStackReserve      00000000
SizeOfStackCommit       00000000
SizeOfHeapReserve       00000000
SizeOfHeapCommit        00000000
LoaderFlags             00000000
NumberOfRvaAndSizes     00000000

The Data Directory
Entry 0 00000000 00000000 Export Directory [.edata (or where ever we found it)]
Entry 1 00000000 00000000 Import Directory [parts of .idata]
Entry 2 00000000 00000000 Resource Directory [.rsrc]
Entry 3 00000000 00000000 Exception Directory [.pdata]
Entry 4 00000000 00000000 Security Directory
Entry 5 00000000 00000000 Base Relocation Directory [.reloc]
Entry 6 00000000 00000000 Debug Directory
Entry 7 00000000 00000000 Description Directory
Entry 8 00000000 00000000 Special Directory
Entry 9 00000000 00000000 Thread Storage Directory [.tls]
Entry a 00000000 00000000 Load Configuration Directory
Entry b 00000000 00000000 Bound Import Directory
Entry c 00000000 00000000 Import Address Table Directory
Entry d 00000000 00000000 Delay Import Directory
Entry e 00000000 00000000 CLR Runtime Header
Entry f 00000000 00000000 Reserved

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .rsrc         0005aaa7  00000000  00000000  0000003c  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
SYMBOL TABLE:
[  0](sec  1)(fl 0x00)(ty   0)(scl   3) (nx 0) 0x00000000 .rsrc


RELOCATION RECORDS FOR [.rsrc]:
OFFSET   TYPE              VALUE
00000158 rva32             .rsrc
00000168 rva32             .rsrc
00000178 rva32             .rsrc
00000188 rva32             .rsrc
00000198 rva32             .rsrc
000001a8 rva32             .rsrc
000001b8 rva32             .rsrc
000001c8 rva32             .rsrc



C:\dev\go\src>

Alex

[alexbrainman]

If I use simple "go build" command on a simple cgo program, I can see completely different output{

C:\dev\src\a>type foo.go
package main

import "C"

func main() {
        println("hello")
}

C:\dev\src\a>go build -ldflags="-tmpdir=c:\tmp\aaa" foo.go

C:\dev\src\a>dir c:\tmp\aaa
 Volume in drive C has no label.
 Volume Serial Number is F483-CD2E

 Directory of c:\tmp\aaa

29/10/2015  09:28 AM    <DIR>          .
29/10/2015  09:28 AM    <DIR>          ..
29/10/2015  09:29 AM             2,107 000000.o
29/10/2015  09:29 AM            11,723 000001.o
29/10/2015  09:29 AM           884,460 go.o
               3 File(s)        898,290 bytes
               2 Dir(s)  100,695,478,272 bytes free

C:\dev\src\a>objdump -h c:\tmp\aaa\000000.o

c:\tmp\aaa\000000.o:     file format pe-i386

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         00000000  00000000  00000000  00000000  2**2
                  ALLOC, LOAD, READONLY, CODE
  1 .data         00000000  00000000  00000000  00000000  2**2
                  ALLOC, LOAD, DATA
  2 .rdata$zzz    00000028  00000000  00000000  0000017c  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .bss          00000000  00000000  00000000  00000000  2**2
                  ALLOC
  4 .debug_aranges 00000030  00000000  00000000  000001a4  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
  5 .debug_info   000002a5  00000000  00000000  000001d4  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
  6 .debug_abbrev 00000032  00000000  00000000  00000479  2**0
                  CONTENTS, READONLY, DEBUGGING
  7 .debug_line   0000003a  00000000  00000000  000004ab  2**0
                  CONTENTS, READONLY, DEBUGGING
  8 .debug_str    00000000  00000000  00000000  00000000  2**0
                  READONLY, DEBUGGING

C:\dev\src\a>objdump -h c:\tmp\aaa\000001.o

c:\tmp\aaa\000001.o:     file format pe-i386

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         000001f0  00000000  00000000  000001f4  2**4
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  1 .data         00000000  00000000  00000000  00000000  2**2
                  ALLOC, LOAD, DATA
  2 .rdata        00000084  00000000  00000000  000003e4  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .rdata$zzz    00000064  00000000  00000000  00000468  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .eh_frame     00000140  00000000  00000000  000004cc  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
  5 .bss          00000000  00000000  00000000  00000000  2**2
                  ALLOC
  6 .debug_aranges 000000b0  00000000  00000000  0000060c  2**3
                  CONTENTS, RELOC, READONLY, DEBUGGING
  7 .debug_info   00000f0c  00000000  00000000  000006bc  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
  8 .debug_abbrev 00000499  00000000  00000000  000015c8  2**0
                  CONTENTS, READONLY, DEBUGGING
  9 .debug_line   000002f6  00000000  00000000  00001a61  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
 10 .debug_str    00000018  00000000  00000000  00001d57  2**0
                  CONTENTS, READONLY, DEBUGGING
 11 .debug_loc    0000006f  00000000  00000000  00001d6f  2**0
                  CONTENTS, READONLY, DEBUGGING

C:\dev\src\a>

That is all on Windows with whatever version of mingw I have.

Alex

[jessfraz]

Ya sorry the rsrc tool has an issue filed I guess akavel/rsrc#11

[jessfraz]

We embed an image we can probably remove it ... windows... cute whale but idk if worth the trouble ;)

[lowenna]

@jfrazelle Ah, we can't remove the use of akavel/rsrc as it's also the thing that manifests docker.exe for Windows 10, otherwise version APIs don't work.

(PS We hit the same thing - I emailed golang-nuts: https://groups.google.com/forum/#!topic/golang-nuts/pZB_QQQk2Z4)

[alexbrainman]

@jhowardmsft go1.5 started using external linker on windows. I suspect some functionality we had with internal linker is broken. For example we cannot include dwarf information anymore (see issue #10776). It sounds like we don't link *.syso files properly either.

What others said, you can use -linkmode=internal ldflag when building executable. This should revert your build process to what 1.4 use to do.

Alex

[lowenna]

@alexbrainman Yes, thank you so much. That's great - I can compile again using 1.5.1 with that workaround on Windows.

[alexbrainman]

I am pretty sure the problem is akavel/rsrc/issues/11 (hopefully @akavel will fix the issue).

Alex

[minux]

The rsrc tool seems to use an old trick: the internal linker doesn't check the file architecture, so it's possible to link 32-bit COFF object files into 64-bit programs. so I think the rsrc tool only creates a 32-bit syso file to embed the resources, and it worked fine with internal linking (and prior to Go 1.5, Go on windows only supports internal linking). This is safe as the COFF file contains only resource, not any code. Now that Go 1.5 supports external linking, and it uses external linking for any code that uses cgo, this problem manifests itself. Before akavel/rsrc#11 is fixed, I suggest you use a proper tool to generate two syso files, one for 32-bit and one for 64-bit and bundle them with docker.

[akavel]

If I change rsrc to make 2 .syso files, one for 32-bit and one for 64-bit ARCH, (how) can I then tell the go tool which one of them it should choose for linking? Would naming them foo_amd64.syso and foo_386.syso work? I understand this is not assumed to be a regression; given that, has it now become impossible to have a project with a COFF .syso file be go-gettable on any ARCH? or if it is, how? I'd be grateful for any help here!

[lowenna]

I was thinking about this over the weekend. Building both .syso files would be problematic for linking as obviously only one file is needed. Perhaps an rsrc --arch=amd64 .... option?

[akavel]

@jhowardmsft Yeah, that's what I'll try to do as a quick workaround (?) but the problem is, such a project becomes then somewhat strongly tied to exactly 1 architecture (amd64 in this case), making it significantly more annoying to build on e.g. 386, or potentially some, uhm, Windows RT? or some other arm stuff in the future? Raspberry Pi maybe?

As another workaround, I'm thinking if maybe doing something like shown below might maybe work (? untested), but boy is it ugly:

$ find amd64 386 -t f
amd64/rsrc.syso
amd64/dummy.go
386/rsrc.syso
386/dummy.go
$ cat rsrc_amd64.go
import _ "myproj/amd64"
$ cat rsrc_386.go
import _ "myproj/386"

[lowenna]

Oh yuck! ;) Yeah, that is kind of horrible. I am wondering if that would work unless rsrc.syso had an rsrc_ARCH.syso name. Does go use that as a build tag on an _ARCH postfix for syso files? I'm really not sure if it does or not.

[mwhudson]

I think the GOOS_GOARCH naming convention works for syso files too (it's
how the race detector works)

On 3 November 2015 at 07:03, Mateusz Czapliński [email protected]
wrote:

If I change rsrc to make 2 .syso files, one for 32-bit and one for 64-bit
ARCH, (how) can I then tell the go tool which one of them it should choose
for linking? Would naming them foo_amd64.syso and foo_386.syso work?
I understand this is not assumed to be a regression, so given that, did it
now become impossible to have a project with a COFF .syso file be
go-gettable on any ARCH? or if it is, how? I'd be grateful for any help
here!

—
Reply to this email directly or view it on GitHub
#13070 (comment).

[minux]

Yeah, build two files rsrc_windows_amd64.syso rsrc_windows_386.syso The linker will pick the right one to link automatically.

[akavel]

@mwhudson @minux Ok, that's great then, thanks! Now just to know if a quick .Machine=pe.IMAGE_FILE_MACHINE_AMD64 will be enough, or some more serious investigation will be needed...