4.0.0 can not be built with LTO

[dvzrv]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Hi! I'm currently trying to build 4.0.0 as a package for Arch Linux. We have switched to LTO by default (can be switched off per package) over the past months. Unfortunately podman does not seem to be compatible with that:

# github.com/containers/podman/v4/cmd/rootlessport
_cgo_callers: relocation target x_cgo_callers not defined
_cgo_init: relocation target x_cgo_init not defined
_cgo_mmap: relocation target x_cgo_mmap not defined
_cgo_munmap: relocation target x_cgo_munmap not defined
_cgo_notify_runtime_init_done: relocation target x_cgo_notify_runtime_init_done not defined
_cgo_sigaction: relocation target x_cgo_sigaction not defined
_cgo_thread_start: relocation target x_cgo_thread_start not defined
net._cgo_3c1cec0c9a4e_C2func_getaddrinfo: relocation target _cgo_3c1cec0c9a4e_C2func_getaddrinfo not defined
net._cgo_3c1cec0c9a4e_Cfunc_freeaddrinfo: relocation target _cgo_3c1cec0c9a4e_Cfunc_freeaddrinfo not defined
net._cgo_3c1cec0c9a4e_Cfunc_gai_strerror: relocation target _cgo_3c1cec0c9a4e_Cfunc_gai_strerror not defined
runtime._cgo_setenv: relocation target x_cgo_setenv not defined
runtime._cgo_unsetenv: relocation target x_cgo_unsetenv not defined
runtime.cgo_yield: relocation target _cgo_yield not defined
.rela: relocation target _cgo_3c1cec0c9a4e_C2func_getaddrinfo not defined
.rela: relocation target _cgo_3c1cec0c9a4e_Cfunc_freeaddrinfo not defined
.rela: relocation target _cgo_3c1cec0c9a4e_Cfunc_gai_strerror not defined
.rela: relocation target _cgo_yield not defined
.rela: relocation target x_cgo_setenv not defined
.rela: relocation target x_cgo_unsetenv not defined
.rela: relocation target x_cgo_init not defined
.rela: relocation target x_cgo_thread_start not defined
/usr/lib/go/pkg/tool/linux_amd64/link: too many errors
unexpected fault address 0x66a19906f000
fatal error: fault

Full build log: podman-4.0.0-build.log

Steps to reproduce the issue:

  export BUILDTAGS='apparmor seccomp systemd'
  export CGO_CPPFLAGS="${CPPFLAGS}"
  export CGO_CFLAGS="${CFLAGS}"
  export CGO_CXXFLAGS="${CXXFLAGS}"
  export CGO_LDFLAGS="${LDFLAGS}"
  export GOFLAGS="-buildmode=pie -trimpath"

  make EXTRA_LDFLAGS='-s -w -linkmode=external'

Here CFLAGS, CPPFLAGS, CXXFLAGS and LDFLAGS represent our distribution wide build flags (through which LTO is introduced into the build).

Describe the results you received:

The build fails.

Describe the results you expected:

The build succeeds

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

4.0.0

Output of podman info --debug:

n/a

Package info (e.g. output of rpm -q podman or apt list podman):

n/a

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.): Package build context for Arch Linux

[baude]

@dvzrv thanks for the report ... what are the expectations upon this team for this one? Do you plan to submit a patch for LTO or is the expectation that we do this?

[mheon]

Also - what Go version is Arch shipping, so we can replicate the build setup?

[baude]

or make a container image for us and push it somewhere (even better)

[dvzrv]

@baude @mheon thanks for the replies!
My go skills sadly lack to do this for this project. However, I can provide you with what you are asking for:

We are using go 1.17.7, gcc 11.2.0 and glibc 2.35 at the moment (see the full build log for all dependency versions).
Our distribution wide build flags can be found here: https://github.com/archlinux/devtools/blob/a17c9c6a703901bff12779ab3b5bebde5194c617/makepkg-x86_64.conf#L43-L47
Please note that -flto is currently inserted into the build flags via an option that is on by default.

You can use the official Arch Linux container (archlinux:latest) to build in that environment.

Additionally, you may also use the official tooling that we use for packaging (sudo pacman -S devtools), which would require you to also install the tooling to download the sources (sudo pacman -S asp).
You can then download the sources:

asp checkout podman
cd podman/trunk

In there you will find this PKGBUILD, in which this line switches off LTO (you can remove it if you want to).

To build the package you only have to invoke extra-x86_64-build in the directory with the PKGBUILD.

[rhatdan]

This issue seems to be answered, closing. Reopen if I am mistaken.

[baude]

@vrothberg @giuseppe either of you dare tackle this?

[vrothberg]

Can you share a reproducer using our Makefile? The CGO_ variables are not shown in the build log.

[giuseppe]

extra-x86_64-build failed with a bunch of errors so I've tried to run these steps in a archlinux:latest container and it worked:

pacman -Sy apparmor btrfs-progs device-mapper go go-md2man git gpgme libseccomp systemd go gcc pkg-config
export CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions \
        -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security \
        -fstack-clash-protection -fcf-protection"
export CXXFLAGS="$CFLAGS -Wp,-D_GLIBCXX_ASSERTIONS"
export LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now"

export BUILDTAGS='apparmor seccomp systemd'
export CGO_CPPFLAGS="${CPPFLAGS}"
export CGO_CFLAGS="${CFLAGS}"
export CGO_CXXFLAGS="${CXXFLAGS}"
export CGO_LDFLAGS="${LDFLAGS}"
export GOFLAGS="-buildmode=pie -trimpath"

make EXTRA_LDFLAGS='-s -w -linkmode=external'

what more do I miss to reproduce the error?

[giuseppe]

just realized I miss -flto, trying again..

[giuseppe]

I was able to reproduce locally adding -flto (works also on Fedora):

#!/bin/sh

export CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions \
        -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security \
        -fstack-clash-protection -fcf-protection  -flto"
export CXXFLAGS="$CFLAGS -Wp,-D_GLIBCXX_ASSERTIONS"
export LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto"

export BUILDTAGS='apparmor seccomp systemd'
export CGO_CPPFLAGS="${CPPFLAGS}"
export CGO_CFLAGS="${CFLAGS}"
export CGO_CXXFLAGS="${CXXFLAGS}"
export CGO_LDFLAGS="${LDFLAGS}"
export GOFLAGS="-buildmode=pie -trimpath"

make EXTRA_LDFLAGS='-s -w -linkmode=external'

Go doesn't seem to support lto: golang/go#43505

I am not sure there is anything we can do from Podman, except forcing -fno-lto in the Makefile.

[dvzrv]

I am not sure there is anything we can do from Podman, except forcing -fno-lto in the Makefile.

Hm, I guess that would be an option. I can only compare it to other go packages that I build, which do not require me to explicitly switch of LTO (e.g. kubernetes).