Cross-compiling with cgo for ARMv7 fails #13377

gjstrange · 2015-11-24T00:59:32Z

We are using Go on a ARM335x/Linux board very similar to the BeagleBone black, but the cross compile fails if it involves mixed C and Go code and cgo.

Background: Using go version 1.5.1. Host is Ubunutu 15.04 running in a VM on a Mac. Target settings:
GOOS= linux
GOARCH=arm
GOARM=7

CC, CXX, etc. are also set to the cross compiler gcc tools.

The error reported is:

`# runtime/cgo
In file included from /opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi/usr/include/features.h:389:0,
from /opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi/usr/include/errno.h:28,
from /usr/local/go/src/runtime/cgo/cgo.go:50:
/opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi/usr/include/gnu/stubs.h:7:29: fatal error: gnu/stubs-soft.h: No such file or directory

include <gnu/stubs-soft.h>

compilation terminated.`

Some research showed that the problem stemmed from incorrect compiler flags. The ARMv7 processor (TI AM3359 Cortex A8 in this case), uses hardware hardware floating point, so the following flags need to be set:

"-mthumb-interwork", "-mfloat-abi=hard", "-mfpu=neon"

It also should link against /lib/ld-linux-armhf.so.3, but links against /lib/ld-linux.so.3 instead.

We have a personal branch of Go now that fixes the problem, though we are new to Go, so the change may not be well done.

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2015-11-24T18:30:50Z

You should be able to address this without modifying Go by setting the CC environment variable when running the go tool. I don't know of any simple way that the Go distribution can get this right for all the possible ARM cases.

gjstrange · 2015-11-24T19:08:14Z

We are setting the CC environment variable. Go strips off all of the
parameters from it and only uses the compiler name. We have CC set as

arm-phytec-linux-gnueabi-gcc -march=armv7-a -marm -mthumb-interwork
-mfloat-abi=hard -mfpu=neon -mtune=cortex-a8
--sysroot=/opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi

Here is the output when Go compiles What we get when Go compiles cgo (go
build -x test.go):

WORK=/tmp/go-build861803713
mkdir -p $WORK/runtime/cgo/_obj/
mkdir -p $WORK/runtime/
cd /usr/local/go/src/runtime/cgo
CGO_LDFLAGS="-g" "-O2" "-lpthread" /usr/local/go/pkg/tool/linux_amd64/cgo
-objdir $WORK/runtime/cgo/_obj/ -importpath runtime/cgo
-import_runtime_cgo=false -import_syscall=false -- -I
$WORK/runtime/cgo/_obj/ -Wall -Werror cgo.go
arm-phytec-linux-gnueabi-gcc -I . -fPIC -marm -pthread -fmessage-length=0
-print-libgcc-file-name
arm-phytec-linux-gnueabi-gcc -I . -fPIC -marm -pthread -fmessage-length=0
-I $WORK/runtime/cgo/_obj/ -g -O2 -Wall -Werror -o
$WORK/runtime/cgo/_obj/_cgo_main.o -c $WORK/runtime/cgo/_obj/_cgo_main.c
arm-phytec-linux-gnueabi-gcc -I . -fPIC -marm -pthread -fmessage-length=0
-I $WORK/runtime/cgo/_obj/ -g -O2 -Wall -Werror -o
$WORK/runtime/cgo/_obj/_cgo_export.o -c $WORK/runtime/cgo/_obj/_cgo_export.c
arm-phytec-linux-gnueabi-gcc -I . -fPIC -marm -pthread -fmessage-length=0
-I $WORK/runtime/cgo/_obj/ -g -O2 -Wall -Werror -o
$WORK/runtime/cgo/_obj/cgo.cgo2.o -c $WORK/runtime/cgo/_obj/cgo.cgo2.c

runtime/cgo

In file included from
/opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi/usr/include/features.h:389:0,
from
/opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi/usr/include/errno.h:28,
from /usr/local/go/src/runtime/cgo/cgo.go:50:
/opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi/usr/include/gnu/stubs.h:7:29:
fatal error: gnu/stubs-soft.h: No such file or directory

include <gnu/stubs-soft.h>

compilation terminated.

Go used the compiler we specified, but replaced all of the command line
arguments with it's own. Is there something else we should be doing?

Greg Strange
Software Engineering Program Lead
Synapse Product Development

mail 1511 6th Ave Suite 400, Seattle, WA 98101
direct 206-832-1269 ext. 3505 <206-832-1269,3505> | office 206-381-0898 |
mobile 206-240-5605
[email protected] | http://www.synapse.com

This email and any files transmitted with it are confidential. Unauthorized
publication, use or dissemination of this email is prohibited.
Please consider the environment before printing.

On Tue, Nov 24, 2015 at 10:31 AM, Ian Lance Taylor <[email protected]

wrote:

You should be able to address this without modifying Go by setting the CC
environment variable when running the go tool. I don't know of any simple
way that the Go distribution can get this right for all the possible ARM
cases.

—
Reply to this email directly or view it on GitHub
#13377 (comment).

ianlancetaylor · 2015-11-24T19:25:50Z

You may also need to set CGO_CFLAGS for the compiler flags.

Actually, though, the behaviour you are describing sounds like a bug. Or else a misunderstanding on my part.

gjstrange · 2015-11-24T20:56:58Z

Thanks. You are right. Setting CGO_CFLAGS corrects the compiler flag
problem. There is just one problem remaining...

The resulting code does not run because it is trying to use the wrong
dynamic linker. It is trying to use /lib/ld-linux.so.3 which does not exist
on our target. For the ARMv7 it should be using /lib/ld-linux-armhf.so.3. I
can skirt the issue in one of two ways. I can hack the code in
src/cmd/link/internal/arm/obj.go to point to /lib/ld-linux-armhf.so.3 or I
can create a symlink between /lib/ld-linux.so.3 on the target to
/lib/ld-linux-armhf.so.3. Either allows the program to run. Is there more
correct way to get Go to use the correct loader?

Greg Strange
Software Engineering Program Lead
Synapse Product Development

mail 1511 6th Ave Suite 400, Seattle, WA 98101
direct 206-832-1269 ext. 3505 <206-832-1269,3505> | office 206-381-0898 |
mobile 206-240-5605
[email protected] | http://www.synapse.com

This email and any files transmitted with it are confidential. Unauthorized
publication, use or dissemination of this email is prohibited.
Please consider the environment before printing.

On Tue, Nov 24, 2015 at 11:26 AM, Ian Lance Taylor <[email protected]

wrote:

You may also need to set CGO_CFLAGS for the compiler flags.

Actually, though, the behaviour you are describing sounds like a bug. Or
else a misunderstanding on my part.

—
Reply to this email directly or view it on GitHub
#13377 (comment).

mwhudson · 2015-11-24T21:01:09Z

I'd recommend using external linking, if you are not using that already (add -ldflags="-linkmode=external -v" to your go build invocation).

ianlancetaylor · 2015-11-24T21:38:47Z

When you run your compiler directly, does it use the correct dynamic linker?

Go is not going to override that.

As Michael says, make sure you are using external linking, although that should be the default since you are using cgo.

gjstrange · 2015-11-24T22:00:54Z

Yes, when I compile directly (C code only), the resulting program uses
/lib/ld-linux-armhf.so.3 and runs correctly.

Setting linkmode=external seems to make no difference.

Greg Strange
Software Engineering Program Lead
Synapse Product Development

mail 1511 6th Ave Suite 400, Seattle, WA 98101
direct 206-832-1269 ext. 3505 <206-832-1269,3505> | office 206-381-0898 |
mobile 206-240-5605
[email protected] | http://www.synapse.com

This email and any files transmitted with it are confidential. Unauthorized
publication, use or dissemination of this email is prohibited.
Please consider the environment before printing.

On Tue, Nov 24, 2015 at 1:40 PM, Ian Lance Taylor [email protected]
wrote:

When you run your compiler directly, does it use the correct dynamic
linker?

Go is not going to override that.

As Michael says, make sure you are using external linking, although that
should be the default since you are using cgo.

—
Reply to this email directly or view it on GitHub
#13377 (comment).

mwhudson · 2015-11-24T22:11:59Z

That's pretty strange. Can you add -x to the build command and stick the
output in a gist?

On 25 November 2015 at 11:01, gjstrange [email protected] wrote:

Yes, when I compile directly (C code only), the resulting program uses
/lib/ld-linux-armhf.so.3 and runs correctly.

Setting linkmode=external seems to make no difference.

Greg Strange
Software Engineering Program Lead
Synapse Product Development

mail 1511 6th Ave Suite 400, Seattle, WA 98101
direct 206-832-1269 ext. 3505 <206-832-1269,3505> | office 206-381-0898 |
mobile 206-240-5605
[email protected] | http://www.synapse.com

This email and any files transmitted with it are confidential. Unauthorized
publication, use or dissemination of this email is prohibited.
Please consider the environment before printing.

On Tue, Nov 24, 2015 at 1:40 PM, Ian Lance Taylor <
[email protected]>
wrote:

When you run your compiler directly, does it use the correct dynamic
linker?

Go is not going to override that.

As Michael says, make sure you are using external linking, although that
should be the default since you are using cgo.

—
Reply to this email directly or view it on GitHub
#13377 (comment).

—
Reply to this email directly or view it on GitHub
#13377 (comment).

minux · 2015-11-24T22:41:27Z

The problem is probably that when invoking the external linker, cmd/link failed to to specify its float abi is hard, so gcc uses the normal dynamic interpreter.

gjstrange · 2015-11-24T22:45:32Z

Yea sure. Here it is

https://gist.github.com/gjstrange/5243afe5b16dc8836db8

Greg Strange
Software Engineering Program Lead
Synapse Product Development

mail 1511 6th Ave Suite 400, Seattle, WA 98101
direct 206-832-1269 ext. 3505 <206-832-1269,3505> | office 206-381-0898 |
mobile 206-240-5605
[email protected] | http://www.synapse.com

This email and any files transmitted with it are confidential. Unauthorized
publication, use or dissemination of this email is prohibited.
Please consider the environment before printing.

On Tue, Nov 24, 2015 at 2:12 PM, Michael Hudson-Doyle <
[email protected]> wrote:

That's pretty strange. Can you add -x to the build command and stick the
output in a gist?

On 25 November 2015 at 11:01, gjstrange [email protected] wrote:

Yes, when I compile directly (C code only), the resulting program uses
/lib/ld-linux-armhf.so.3 and runs correctly.

Setting linkmode=external seems to make no difference.

Greg Strange
Software Engineering Program Lead
Synapse Product Development

mail 1511 6th Ave Suite 400, Seattle, WA 98101
direct 206-832-1269 ext. 3505 <206-832-1269,3505> | office 206-381-0898
|
mobile 206-240-5605
[email protected] | http://www.synapse.com

This email and any files transmitted with it are confidential.
Unauthorized
publication, use or dissemination of this email is prohibited.
Please consider the environment before printing.

On Tue, Nov 24, 2015 at 1:40 PM, Ian Lance Taylor <
[email protected]>
wrote:

When you run your compiler directly, does it use the correct dynamic
linker?

Go is not going to override that.

As Michael says, make sure you are using external linking, although
that
should be the default since you are using cgo.

—
Reply to this email directly or view it on GitHub
#13377 (comment).

—
Reply to this email directly or view it on GitHub
#13377 (comment).

—
Reply to this email directly or view it on GitHub
#13377 (comment).

davecheney · 2015-11-24T22:47:57Z

Shouldn't gcc, if compiled against hard float use this as a default? Maybe
this can be fixed with -ldflags

Did someone ask for the output of gcc -v to see what it's default are ?

On Wed, 25 Nov 2015, 09:41 Minux Ma [email protected] wrote:

The problem is probably that when invoking the external linker, cmd/link
failed to to specify its float abi is hard, so gcc uses the normal dynamic
interpreter.

—
Reply to this email directly or view it on GitHub
#13377 (comment).

mwhudson · 2015-11-24T22:57:22Z

On 25 November 2015 at 11:45, gjstrange [email protected] wrote:

Yea sure. Here it is

https://gist.github.com/gjstrange/5243afe5b16dc8836db8

This looks like the same sort of "only using the first word of $CC" problem
as was alluded to earlier. You can probably fix this by adding
"-ldflags=-extldflags=-march=armv7-a -marm -mthumb-interwork
-mfloat-abi=hard -mfpu=neon -mtune=cortex-a8
--sysroot=/opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi"
or similar to the go build command but that's pretty ugly.

I think I've seen this behaviour too, but it wasn't important to me at the
time so I didn't chase. But Ian seemed of the opinion that this was a bug.

gjstrange · 2015-11-24T23:09:52Z

Actually I put the linker flags into CGO_LDFLAGS environment variable and
that fixed the problem. Don't know if you want to pursue this any further,
but this solution works for us. Thanks for all your help.

Greg Strange
Software Engineering Program Lead
Synapse Product Development

mail 1511 6th Ave Suite 400, Seattle, WA 98101
direct 206-832-1269 ext. 3505 <206-832-1269,3505> | office 206-381-0898 |
mobile 206-240-5605
[email protected] | http://www.synapse.com

This email and any files transmitted with it are confidential. Unauthorized
publication, use or dissemination of this email is prohibited.
Please consider the environment before printing.

On Tue, Nov 24, 2015 at 2:58 PM, Michael Hudson-Doyle <
[email protected]> wrote:

On 25 November 2015 at 11:45, gjstrange [email protected] wrote:

Yea sure. Here it is

https://gist.github.com/gjstrange/5243afe5b16dc8836db8

This looks like the same sort of "only using the first word of $CC" problem
as was alluded to earlier. You can probably fix this by adding
"-ldflags=-extldflags=-march=armv7-a -marm -mthumb-interwork
-mfloat-abi=hard -mfpu=neon -mtune=cortex-a8

--sysroot=/opt/yogurt/AM335x-PD15.1.1/sysroots/cortexa8t2hf-vfp-neon-phytec-linux-gnueabi"
or similar to the go build command but that's pretty ugly.

I think I've seen this behaviour too, but it wasn't important to me at the
time so I didn't chase. But Ian seemed of the opinion that this was a bug.

—
Reply to this email directly or view it on GitHub
#13377 (comment).

minux · 2015-11-25T00:47:03Z

Another workaround is to create a gcc wrapper that contains the necessary fp abi flags, and set that to $CC.

davmaz · 2016-08-04T21:14:38Z

Do you happen to have a small example that shows how to do the workaround you mention?

minux · 2016-08-04T22:51:33Z

See $GOROOT/misc/ios/clangwrap.sh for an example script that is used as $CC during iOS go build. It's for iOS, but it should give you the idea.

nevelshute · 2017-05-21T14:20:53Z

This problem was also seen when build BusyBox from a Yocto toolchain using the SDK installed externally.

The problem stemmed from the fact that the BusyBox build used the $CROSS_COMPILE and the $CFLAGS environment variables and di not access the $CC variable which had the switches: -march=armv7-a -mfloat-abi=hard -mfpu=neon -mtune=cortex-a9 --sysroot=$SDKTARGETSYSROOT

With busybox we could add the additional CFLAGS switches via the menuconfig or we could modify the Environment.sh script and added the flags to the $CFLAGS variable.

Hope this helps someone as we could not find an answer!

Nev

ianlancetaylor closed this as completed Nov 24, 2015

AxbB36 mentioned this issue Apr 26, 2016

cmd/go: go env discards all but the first word of $CC #15457

Closed

golang locked and limited conversation to collaborators May 21, 2018

gopherbot added the FrozenDueToAge label May 21, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cross-compiling with cgo for ARMv7 fails #13377

Cross-compiling with cgo for ARMv7 fails #13377

gjstrange commented Nov 24, 2015

ianlancetaylor commented Nov 24, 2015

gjstrange commented Nov 24, 2015

ianlancetaylor commented Nov 24, 2015

gjstrange commented Nov 24, 2015

mwhudson commented Nov 24, 2015 via email

ianlancetaylor commented Nov 24, 2015

gjstrange commented Nov 24, 2015

mwhudson commented Nov 24, 2015

minux commented Nov 24, 2015 via email

gjstrange commented Nov 24, 2015

davecheney commented Nov 24, 2015

mwhudson commented Nov 24, 2015

gjstrange commented Nov 24, 2015

minux commented Nov 25, 2015 via email

davmaz commented Aug 4, 2016

minux commented Aug 4, 2016 via email

nevelshute commented May 21, 2017

Cross-compiling with cgo for ARMv7 fails #13377

Cross-compiling with cgo for ARMv7 fails #13377

Comments

gjstrange commented Nov 24, 2015

include <gnu/stubs-soft.h>

ianlancetaylor commented Nov 24, 2015

gjstrange commented Nov 24, 2015

runtime/cgo

include <gnu/stubs-soft.h>

ianlancetaylor commented Nov 24, 2015

gjstrange commented Nov 24, 2015

mwhudson commented Nov 24, 2015 via email

ianlancetaylor commented Nov 24, 2015

gjstrange commented Nov 24, 2015

mwhudson commented Nov 24, 2015

minux commented Nov 24, 2015 via email

gjstrange commented Nov 24, 2015

davecheney commented Nov 24, 2015

mwhudson commented Nov 24, 2015

gjstrange commented Nov 24, 2015

minux commented Nov 25, 2015 via email

davmaz commented Aug 4, 2016

minux commented Aug 4, 2016 via email

nevelshute commented May 21, 2017