Missing absolute include paths with Remote Execution

[ffortier]

This is a new issue with toolchains_llvm v1.5.0. I have created this project to reproduce it: https://github.com/ffortier/issue-llvm-rbe

When using a remote executor (buildbuddy in this example), I get the following error:

ERROR: /home/runner/work/issue-llvm-rbe/issue-llvm-rbe/v1.5.0/app/BUILD.bazel:3:10: Compiling app/main.c failed: absolute path inclusion(s) found in rule '//app:app':
the source file 'app/main.c' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain):
  '/buildbuddy-execroot/external/toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/stddef.h'
  '/buildbuddy-execroot/external/toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/__stddef_size_t.h'
  '/buildbuddy-execroot/external/toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/__stddef_null.h'
  '/buildbuddy-execroot/external/toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/stdarg.h'
  '/buildbuddy-execroot/external/toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/__stdarg___gnuc_va_list.h'
Target //app:app failed to build

This use to work fine with v1.4.0. This is a screenshot of the workflows:

[fmeum]

CC @sluongng

[fmeum]

This is caused by #518 (comment), but I haven't understood why yet.

[ffortier]

Probably not related to this issue, but when I was looking at the cc_wrapper.sh file edited in the MR above, I found a potential issue here:

toolchains_llvm/toolchain/cc_wrapper.sh.tpl

Line 112 in 0bdeae2

if [[ ${!i} == @* && -r "${i:1}" ]]; then

It was there before and is still there, but we do the following

if [[ ${!i} == @* && -r "${i:1}" ]]; then

i is correctly deref in the first condition, but not in the second, so this if will always be false. It is correctly defer below in the while loop.

should be if [[ "${!i}" == @* && -r "${!i:1}" ]]; then

[ffortier]

I've had this kind of issue in the past when working on my own toolchain for another project and it was due to the dependency file generated by clang that bazel uses to lookup the correct libraries. So I compare a linux build, without the remote build execution, just to see the difference

# v1.4.0
bazel-out/k8-fastbuild/bin/app/_objs/app/main.pic.o: app/main.c \
  external/+_repo_rules+bullseye_amd64/usr/include/stdio.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/libc-header-start.h \
  external/+_repo_rules+bullseye_amd64/usr/include/features.h \
  external/+_repo_rules+bullseye_amd64/usr/include/stdc-predef.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/sys/cdefs.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/wordsize.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/long-double.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/gnu/stubs.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/gnu/stubs-64.h \
  external/toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/stddef.h \
  external/toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/__stddef_size_t.h \
  external/toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/__stddef_null.h \
  external/toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/stdarg.h \
  external/toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/__stdarg___gnuc_va_list.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/types.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/timesize.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/typesizes.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/time64.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/types/__fpos_t.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/types/__mbstate_t.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/types/__fpos64_t.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/types/__FILE.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/types/FILE.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/types/struct_FILE.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/stdio_lim.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/sys_errlist.h

# v1.5.0
bazel-out/k8-fastbuild/bin/app/_objs/app/main.pic.o: app/main.c \
  external/+_repo_rules+bullseye_amd64/usr/include/stdio.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/libc-header-start.h \
  external/+_repo_rules+bullseye_amd64/usr/include/features.h \
  external/+_repo_rules+bullseye_amd64/usr/include/stdc-predef.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/sys/cdefs.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/wordsize.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/long-double.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/gnu/stubs.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/gnu/stubs-64.h \
  external/toolchains_llvm++llvm+llvm_toolchain/bin/../../toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/stddef.h \
  external/toolchains_llvm++llvm+llvm_toolchain/bin/../../toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/__stddef_size_t.h \
  external/toolchains_llvm++llvm+llvm_toolchain/bin/../../toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/__stddef_null.h \
  external/toolchains_llvm++llvm+llvm_toolchain/bin/../../toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/stdarg.h \
  external/toolchains_llvm++llvm+llvm_toolchain/bin/../../toolchains_llvm++llvm+llvm_toolchain_llvm/lib/clang/20/include/__stdarg___gnuc_va_list.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/types.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/timesize.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/typesizes.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/time64.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/types/__fpos_t.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/types/__mbstate_t.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/types/__fpos64_t.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/types/__FILE.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/types/FILE.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/types/struct_FILE.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/stdio_lim.h \
  external/+_repo_rules+bullseye_amd64/usr/include/x86_64-linux-gnu/bits/sys_errlist.h

The latter has extra toolchains_llvm++llvm+llvm_toolchain/bin/../../ in the toolchain includes. This change seems to be related to the MR. Unfortunately, I can't get the dependency file from buildbuddy, but maybe this path is interpreted differently.

[fmeum]

Thanks for catching the separate bug as well as the diff - I think I now know what to do.