setup.py: GCC detection is broken when cross-compiling with a German locale

[AlexGrund]

BPO	38472
Nosy	@doko42, @vstinner, @miss-islington, @iritkatriel
PRs	bpo-38472: setup.py uses LC_ALL=C to check the C compiler #30929 [3.10] bpo-38472: setup.py uses LC_ALL=C to check the C compiler (GH-30929) #30931 [3.9] bpo-38472: setup.py uses LC_ALL=C to check the C compiler (GH-30929) #30932

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-10-14.12:28:34.957>
labels = ['build', '3.9']
title = 'setup.py: GCC detection is broken when cross-compiling with a German locale'
updated_at = <Date 2022-01-26.23:50:38.165>
user = 'https://bugs.python.org/AlexGrund'

bugs.python.org fields:

activity = <Date 2022-01-26.23:50:38.165>
actor = 'miss-islington'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2019-10-14.12:28:34.957>
creator = 'Alex Grund'
dependencies = []
files = []
hgrepos = []
issue_num = 38472
keywords = ['patch']
message_count = 11.0
messages = ['354629', '355072', '355074', '410831', '411388', '411801', '411805', '411806', '411807', '411812', '411813']
nosy_count = 5.0
nosy_names = ['doko', 'vstinner', 'miss-islington', 'Alex Grund', 'iritkatriel']
pr_nums = ['30929', '30931', '30932']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue38472'
versions = ['Python 3.9']

[AlexGrund]

setup.py runs <CC> -E -v - </dev/null 1>/dev/null to figure out include and library paths from the compiler in the function add_gcc_paths.

However sample output from the compiler is:

Es werden eingebaute Spezifikationen verwendet.
COLLECT_GCC=g++
Ziel: x86_64-pc-linux-gnu
Konfiguriert mit: ../configure --enable-languages=c,c++,fortran --enable-lto --enable-checking=release --disable-multilib --enable-shared=yes --enable-static=yes --enable-threads=posix --enable-plugins --enable-gold=default --enable-ld --with-plugin-ld=ld.gold --prefix=/sw/installed/GCCcore/9.1.0 --with-local-prefix=/sw/installed/GCCcore/9.1.0 --enable-bootstrap --with-isl=/dev/shm/easybuild-build/GCCcore/9.1.0/dummy-/gcc-9.1.0/stage2_stuff
Thread-Modell: posix
gcc-Version 9.1.0 (GCC)
COLLECT_GCC_OPTIONS='-E' '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64'
/software/haswell/GCCcore/9.1.0/bin/../libexec/gcc/x86_64-pc-linux-gnu/9.1.0/cc1 -E -quiet -v -iprefix /software/haswell/GCCcore/9.1.0/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/ - -mtune=generic -march=x86-64
nicht vorhandenes Verzeichnis »/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/include« wird ignoriert
doppeltes Verzeichnis »/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/../../lib/gcc/x86_64-pc-linux-gnu/9.1.0/include« wird ignoriert
doppeltes Verzeichnis »/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/../../lib/gcc/x86_64-pc-linux-gnu/9.1.0/include-fixed« wird ignoriert
nicht vorhandenes Verzeichnis »/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/../../lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/include« wird ignoriert
doppeltes Verzeichnis »/sw/installed/GCCcore/9.1.0/include« wird ignoriert
da es ein Nicht-Systemverzeichnis ist, das ein Systemverzeichnis dupliziert
Suche für »#include "..."« beginnt hier:
Suche für »#include <...>« beginnt hier:
/sw/installed/binutils/2.32-GCCcore-9.1.0/include
/sw/installed/zlib/1.2.11-GCCcore-9.1.0/include
/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/include
/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/include-fixed
/sw/installed/GCCcore/9.1.0/include
/usr/include
Ende der Suchliste.
COMPILER_PATH=/software/haswell/GCCcore/9.1.0/bin/../libexec/gcc/x86_64-pc-linux-gnu/9.1.0/:/software/haswell/GCCcore/9.1.0/bin/../libexec/gcc/
LIBRARY_PATH=/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/:/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/:/sw/installed/GCCcore/9.1.0/lib64/../lib64/:/sw/installed/GCCcore/9.1.0/lib/../lib64/:/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/sw/installed/binutils/2.32-GCCcore-9.1.0/lib/:/sw/installed/zlib/1.2.11-GCCcore-9.1.0/lib/:/sw/installed/GCCcore/9.1.0/lib64/:/sw/installed/GCCcore/9.1.0/lib/:/software/haswell/GCCcore/9.1.0/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-E' '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64'

So the correct matcher would be "gcc-Version", maybe in addition to "gcc version"

Due to this the setup fails to build e.g. bzip2 modules as the include and lib paths are passed in via CPATH and LIBRARY_PATH only (module system on HPC system)

[AlexGrund]

This seems to be a locale issue. So a solution would be to use LC_ALL=C before invoking the compiler (or the cross-platform equivalent)

[vstinner]

What are you trying to do? Compile Python? Cross-compile Python?

setup.py runs <CC> -E -v - </dev/null 1>/dev/null to figure out include and library paths from the compiler in the function add_gcc_paths.

Are you talking about the add_cross_compiling_paths() function which is only called when _PYTHON_HOST_PLATFORM environment variable is set?

So GCC says "gcc-Version 9.1.0 (GCC) " with your locale, but does it write "gcc version 9.1.0 ..." with LC_ALL=C?

[iritkatriel]

Alex, is this still an issue with current python versions (>= 3.9)? And can you followup on Victor's questions?

[AlexGrund]

Yes this is still an issue.

I'm trying to compile Python on an HPC system which uses modules (see e.g. LMod).

Yes with LC_ALL=C it does write "gcc version 9.1.0 ..."

[vstinner]

Ok, I reproduced the issue. I wrote PR 30929 to fix it.

The issue only occurs when cross-compiling Python with GCC and a German locale. I can reproduce the issue on Fedora 35:

$ LC_ALL=de_DE gcc -E -v
Es werden eingebaute Spezifikationen verwendet.
COLLECT_GCC=gcc
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Ziel: x86_64-redhat-linux
Konfiguriert mit: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,objc,obj-c++,ada,go,d,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl=/builddir/build/BUILD/gcc-11.2.1-20211203/obj-x86_64-redhat-linux/isl-install --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux --with-build-config=bootstrap-lto --enable-link-serialization=1
Thread-Modell: posix
Unterst�tzte LTO-Kompressionsalgorithmen: zlib zstd
gcc-Version 11.2.1 20211203 (Red Hat 11.2.1-7) (GCC) 

The last line starts with "gcc-Version 11.2.1". Whereas the last line starts with "gcc version 11.2.1" with the C locale:

$ LC_ALL=C gcc -E -v
Using built-in specs.
COLLECT_GCC=gcc
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,objc,obj-c++,ada,go,d,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl=/builddir/build/BUILD/gcc-11.2.1-20211203/obj-x86_64-redhat-linux/isl-install --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux --with-build-config=bootstrap-lto --enable-link-serialization=1
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.2.1 20211203 (Red Hat 11.2.1-7) (GCC)

[vstinner]

FYI distutils.unixccompiler has a private _is_gcc() function:

    def _is_gcc(self, compiler_name):
        # clang uses same syntax for rpath as gcc
        return any(name in compiler_name for name in ("gcc", "g++", "clang"))

It's called with:

    compiler = os.path.basename(sysconfig.get_config_var("CC"))

For example, on my Fedora 35, compiler is the string: 'gcc -pthread'. So _is_gcc() returns True.

[vstinner]

New changeset a9503ac by Victor Stinner in branch 'main':
bpo-38472: setup.py uses LC_ALL=C to check the C compiler (GH-30929)
a9503ac