Cross-compiling with cgo for ARMv7 fails

[gjstrange]

We are using Go on a ARM335x/Linux board very similar to the BeagleBone black, but the cross compile fails if it involves mixed C and Go code and cgo.

Background: Using go version 1.5.1. Host is Ubunutu 15.04 running in a VM on a Mac. Target settings:
GOOS= linux
GOARCH=arm
GOARM=7

CC, CXX, etc. are also set to the cross compiler gcc tools.

The error reported is:

`# runtime/cgo
In file included from /opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi/usr/include/features.h:389:0,
from /opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi/usr/include/errno.h:28,
from /usr/local/go/src/runtime/cgo/cgo.go:50:
/opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi/usr/include/gnu/stubs.h:7:29: fatal error: gnu/stubs-soft.h: No such file or directory

include <gnu/stubs-soft.h>

                         ^

compilation terminated.`

Some research showed that the problem stemmed from incorrect compiler flags. The ARMv7 processor (TI AM3359 Cortex A8 in this case), uses hardware hardware floating point, so the following flags need to be set:

"-mthumb-interwork", "-mfloat-abi=hard", "-mfpu=neon"

It also should link against /lib/ld-linux-armhf.so.3, but links against /lib/ld-linux.so.3 instead.

We have a personal branch of Go now that fixes the problem, though we are new to Go, so the change may not be well done.

[ianlancetaylor]

You should be able to address this without modifying Go by setting the CC environment variable when running the go tool. I don't know of any simple way that the Go distribution can get this right for all the possible ARM cases.

[gjstrange]

We are setting the CC environment variable. Go strips off all of the
parameters from it and only uses the compiler name. We have CC set as

arm-phytec-linux-gnueabi-gcc -march=armv7-a -marm -mthumb-interwork
-mfloat-abi=hard -mfpu=neon -mtune=cortex-a8
--sysroot=/opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi

Here is the output when Go compiles What we get when Go compiles cgo (go
build -x test.go):

WORK=/tmp/go-build861803713
mkdir -p $WORK/runtime/cgo/_obj/
mkdir -p $WORK/runtime/
cd /usr/local/go/src/runtime/cgo
CGO_LDFLAGS="-g" "-O2" "-lpthread" /usr/local/go/pkg/tool/linux_amd64/cgo
-objdir $WORK/runtime/cgo/_obj/ -importpath runtime/cgo
-import_runtime_cgo=false -import_syscall=false -- -I
$WORK/runtime/cgo/_obj/ -Wall -Werror cgo.go
arm-phytec-linux-gnueabi-gcc -I . -fPIC -marm -pthread -fmessage-length=0
-print-libgcc-file-name
arm-phytec-linux-gnueabi-gcc -I . -fPIC -marm -pthread -fmessage-length=0
-I $WORK/runtime/cgo/_obj/ -g -O2 -Wall -Werror -o
$WORK/runtime/cgo/_obj/_cgo_main.o -c $WORK/runtime/cgo/_obj/_cgo_main.c
arm-phytec-linux-gnueabi-gcc -I . -fPIC -marm -pthread -fmessage-length=0
-I $WORK/runtime/cgo/_obj/ -g -O2 -Wall -Werror -o
$WORK/runtime/cgo/_obj/_cgo_export.o -c $WORK/runtime/cgo/_obj/_cgo_export.c
arm-phytec-linux-gnueabi-gcc -I . -fPIC -marm -pthread -fmessage-length=0
-I $WORK/runtime/cgo/_obj/ -g -O2 -Wall -Werror -o
$WORK/runtime/cgo/_obj/cgo.cgo2.o -c $WORK/runtime/cgo/_obj/cgo.cgo2.c

runtime/cgo

In file included from
/opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi/usr/include/features.h:389:0,
from
/opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi/usr/include/errno.h:28,
from /usr/local/go/src/runtime/cgo/cgo.go:50:
/opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi/usr/include/gnu/stubs.h:7:29:
fatal error: gnu/stubs-soft.h: No such file or directory

include <gnu/stubs-soft.h>

                         ^

compilation terminated.

Go used the compiler we specified, but replaced all of the command line
arguments with it's own. Is there something else we should be doing?

Greg Strange
Software Engineering Program Lead
Synapse Product Development

mail 1511 6th Ave Suite 400, Seattle, WA 98101
direct 206-832-1269 ext. 3505 <206-832-1269,3505> | office 206-381-0898 |
mobile 206-240-5605
greg@synapse.com | http://www.synapse.com

This email and any files transmitted with it are confidential. Unauthorized
publication, use or dissemination of this email is prohibited.
Please consider the environment before printing.

On Tue, Nov 24, 2015 at 10:31 AM, Ian Lance Taylor <notifications@github.com

wrote:

You should be able to address this without modifying Go by setting the CC
environment variable when running the go tool. I don't know of any simple
way that the Go distribution can get this right for all the possible ARM
cases.

—
Reply to this email directly or view it on GitHub
#13377 (comment).

[ianlancetaylor]

You may also need to set CGO_CFLAGS for the compiler flags.

Actually, though, the behaviour you are describing sounds like a bug. Or else a misunderstanding on my part.

[gjstrange]

Thanks. You are right. Setting CGO_CFLAGS corrects the compiler flag
problem. There is just one problem remaining...

The resulting code does not run because it is trying to use the wrong
dynamic linker. It is trying to use /lib/ld-linux.so.3 which does not exist
on our target. For the ARMv7 it should be using /lib/ld-linux-armhf.so.3. I
can skirt the issue in one of two ways. I can hack the code in
src/cmd/link/internal/arm/obj.go to point to /lib/ld-linux-armhf.so.3 or I
can create a symlink between /lib/ld-linux.so.3 on the target to
/lib/ld-linux-armhf.so.3. Either allows the program to run. Is there more
correct way to get Go to use the correct loader?

Greg Strange
Software Engineering Program Lead
Synapse Product Development

mail 1511 6th Ave Suite 400, Seattle, WA 98101
direct 206-832-1269 ext. 3505 <206-832-1269,3505> | office 206-381-0898 |
mobile 206-240-5605
greg@synapse.com | http://www.synapse.com

This email and any files transmitted with it are confidential. Unauthorized
publication, use or dissemination of this email is prohibited.
Please consider the environment before printing.

On Tue, Nov 24, 2015 at 11:26 AM, Ian Lance Taylor <notifications@github.com

wrote:

You may also need to set CGO_CFLAGS for the compiler flags.

Actually, though, the behaviour you are describing sounds like a bug. Or
else a misunderstanding on my part.

—
Reply to this email directly or view it on GitHub
#13377 (comment).

[mwhudson]

I'd recommend using external linking, if you are not using that already (add -ldflags="-linkmode=external -v" to your go build invocation).

[ianlancetaylor]

When you run your compiler directly, does it use the correct dynamic linker?

Go is not going to override that.

As Michael says, make sure you are using external linking, although that should be the default since you are using cgo.

[gjstrange]

Yes, when I compile directly (C code only), the resulting program uses
/lib/ld-linux-armhf.so.3 and runs correctly.

Setting linkmode=external seems to make no difference.

Greg Strange
Software Engineering Program Lead
Synapse Product Development

mail 1511 6th Ave Suite 400, Seattle, WA 98101
direct 206-832-1269 ext. 3505 <206-832-1269,3505> | office 206-381-0898 |
mobile 206-240-5605
greg@synapse.com | http://www.synapse.com

This email and any files transmitted with it are confidential. Unauthorized
publication, use or dissemination of this email is prohibited.
Please consider the environment before printing.

On Tue, Nov 24, 2015 at 1:40 PM, Ian Lance Taylor notifications@github.com
wrote:

When you run your compiler directly, does it use the correct dynamic
linker?

Go is not going to override that.

As Michael says, make sure you are using external linking, although that
should be the default since you are using cgo.

—
Reply to this email directly or view it on GitHub
#13377 (comment).

[mwhudson]

That's pretty strange. Can you add -x to the build command and stick the
output in a gist?

On 25 November 2015 at 11:01, gjstrange notifications@github.com wrote:

Yes, when I compile directly (C code only), the resulting program uses
/lib/ld-linux-armhf.so.3 and runs correctly.

Setting linkmode=external seems to make no difference.

Greg Strange
Software Engineering Program Lead
Synapse Product Development

mail 1511 6th Ave Suite 400, Seattle, WA 98101
direct 206-832-1269 ext. 3505 <206-832-1269,3505> | office 206-381-0898 |
mobile 206-240-5605
greg@synapse.com | http://www.synapse.com

This email and any files transmitted with it are confidential. Unauthorized
publication, use or dissemination of this email is prohibited.
Please consider the environment before printing.

On Tue, Nov 24, 2015 at 1:40 PM, Ian Lance Taylor <
notifications@github.com>
wrote:

When you run your compiler directly, does it use the correct dynamic
linker?

Go is not going to override that.

As Michael says, make sure you are using external linking, although that
should be the default since you are using cgo.

—
Reply to this email directly or view it on GitHub
#13377 (comment).

—
Reply to this email directly or view it on GitHub
#13377 (comment).

[minux]

The problem is probably that when invoking the external linker, cmd/link failed to to specify its float abi is hard, so gcc uses the normal dynamic interpreter.

[gjstrange]

Yea sure. Here it is

https://gist.github.com/gjstrange/5243afe5b16dc8836db8

Greg Strange
Software Engineering Program Lead
Synapse Product Development

mail 1511 6th Ave Suite 400, Seattle, WA 98101
direct 206-832-1269 ext. 3505 <206-832-1269,3505> | office 206-381-0898 |
mobile 206-240-5605
greg@synapse.com | http://www.synapse.com

This email and any files transmitted with it are confidential. Unauthorized
publication, use or dissemination of this email is prohibited.
Please consider the environment before printing.

On Tue, Nov 24, 2015 at 2:12 PM, Michael Hudson-Doyle <
notifications@github.com> wrote:

That's pretty strange. Can you add -x to the build command and stick the
output in a gist?

On 25 November 2015 at 11:01, gjstrange notifications@github.com wrote:

Yes, when I compile directly (C code only), the resulting program uses
/lib/ld-linux-armhf.so.3 and runs correctly.

Setting linkmode=external seems to make no difference.

Greg Strange
Software Engineering Program Lead
Synapse Product Development

mail 1511 6th Ave Suite 400, Seattle, WA 98101
direct 206-832-1269 ext. 3505 <206-832-1269,3505> | office 206-381-0898
|
mobile 206-240-5605
greg@synapse.com | http://www.synapse.com

This email and any files transmitted with it are confidential.
Unauthorized
publication, use or dissemination of this email is prohibited.
Please consider the environment before printing.

On Tue, Nov 24, 2015 at 1:40 PM, Ian Lance Taylor <
notifications@github.com>
wrote:

When you run your compiler directly, does it use the correct dynamic
linker?

Go is not going to override that.

As Michael says, make sure you are using external linking, although
that
should be the default since you are using cgo.

—
Reply to this email directly or view it on GitHub
#13377 (comment).

—
Reply to this email directly or view it on GitHub
#13377 (comment).

—
Reply to this email directly or view it on GitHub
#13377 (comment).

[davecheney]

Shouldn't gcc, if compiled against hard float use this as a default? Maybe
this can be fixed with -ldflags

Did someone ask for the output of gcc -v to see what it's default are ?

On Wed, 25 Nov 2015, 09:41 Minux Ma notifications@github.com wrote:

The problem is probably that when invoking the external linker, cmd/link
failed to to specify its float abi is hard, so gcc uses the normal dynamic
interpreter.

—
Reply to this email directly or view it on GitHub
#13377 (comment).

[mwhudson]

On 25 November 2015 at 11:45, gjstrange notifications@github.com wrote:

Yea sure. Here it is

https://gist.github.com/gjstrange/5243afe5b16dc8836db8

This looks like the same sort of "only using the first word of $CC" problem
as was alluded to earlier. You can probably fix this by adding
"-ldflags=-extldflags=-march=armv7-a -marm -mthumb-interwork
-mfloat-abi=hard -mfpu=neon -mtune=cortex-a8
--sysroot=/opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi"
or similar to the go build command but that's pretty ugly.

I think I've seen this behaviour too, but it wasn't important to me at the
time so I didn't chase. But Ian seemed of the opinion that this was a bug.

[gjstrange]

Actually I put the linker flags into CGO_LDFLAGS environment variable and
that fixed the problem. Don't know if you want to pursue this any further,
but this solution works for us. Thanks for all your help.

Greg Strange
Software Engineering Program Lead
Synapse Product Development

mail 1511 6th Ave Suite 400, Seattle, WA 98101
direct 206-832-1269 ext. 3505 <206-832-1269,3505> | office 206-381-0898 |
mobile 206-240-5605
greg@synapse.com | http://www.synapse.com

This email and any files transmitted with it are confidential. Unauthorized
publication, use or dissemination of this email is prohibited.
Please consider the environment before printing.

On Tue, Nov 24, 2015 at 2:58 PM, Michael Hudson-Doyle <
notifications@github.com> wrote:

On 25 November 2015 at 11:45, gjstrange notifications@github.com wrote:

Yea sure. Here it is

https://gist.github.com/gjstrange/5243afe5b16dc8836db8

This looks like the same sort of "only using the first word of $CC" problem
as was alluded to earlier. You can probably fix this by adding
"-ldflags=-extldflags=-march=armv7-a -marm -mthumb-interwork
-mfloat-abi=hard -mfpu=neon -mtune=cortex-a8

--sysroot=/opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi"
or similar to the go build command but that's pretty ugly.

I think I've seen this behaviour too, but it wasn't important to me at the
time so I didn't chase. But Ian seemed of the opinion that this was a bug.

—
Reply to this email directly or view it on GitHub
#13377 (comment).

[minux]

Another workaround is to create a gcc wrapper that contains the necessary fp abi flags, and set that to $CC.