"Never resolved function from blockaddress" linking xfs.ko

[arndb]

Building randconfig arm64 kernels occasionally results in this error with LTO:

$ ld.lld -EL -maarch64elf -mllvm -import-instr-limit=5 -r -o xfs.lto.o --whole-archive xfs.o
ld.lld: error: Never resolved function from blockaddress (Producer: 'LLVM11.0.1' Reader: 'LLVM 11.0.1')

It turns out that this only happens in configurations that include xfs, but it also does happen for xfs as a loadable module, which makes it much easier to reproduce.

The attached archive contains a copy of the object files that go into an affected xfs.ko kernel module, with a oneline script for reproducing the problem.

xfs-lld-never-resolved-function.tar.gz
https://lore.kernel.org/lkml/CAK8P3a1Xfpt7QLkvxjtXKcgzcWkS8g9bmxD687+rqjTafTzKrg@mail.gmail.com/

[nickdesaulniers]

cc @samitolvanen

[arndb]

This appears to fix all instances I found. I have no idea what is special about these two files, but I found this list experimentally. The other files in libxfs are not affected, but disabling LTO in only one of the two files is not sufficient.

I also found this only happens with CONFIG_DYNAMIC_FTRACE.

commit 18ed03a2163b8668553b9f63255a1697326b68ef
Author: Arnd Bergmann <arnd@arndb.de>
Date:   Fri Dec 18 00:15:29 2020 +0100

    xfs: work around clang lto issue
    
    Something causes the link process to fail:
    
    ld.lld: error: Never resolved function from blockaddress (Producer: 'LLVM11.0.1' Reader: 'LLVM 11.0.1')
    
    Signed-off-by: Arnd Bergmann <arnd@arndb.de>

diff --git a/fs/xfs/Makefile b/fs/xfs/Makefile
index 04611a1068b4..df03efdb731e 100644
--- a/fs/xfs/Makefile
+++ b/fs/xfs/Makefile
@@ -12,6 +12,9 @@ obj-$(CONFIG_XFS_FS)		+= xfs.o
 # this one should be compiled first, as the tracing macros can easily blow up
 xfs-y				+= xfs_trace.o
 
+CFLAGS_REMOVE_libxfs/xfs_bmap.o		+= $(CC_FLAGS_LTO)
+CFLAGS_REMOVE_libxfs/xfs_inode_fork.o	+= $(CC_FLAGS_LTO)
+
 # build the libxfs code first
 xfs-y				+= $(addprefix libxfs/, \
 				   xfs_ag.o \

[bwendling]

The xfs_bmap.o module is mangled. If you look at the disassembly, you have this function:

define internal fastcc void @trace_xfs_read_extent(...
 ...
29:                                               ; preds = %20
  %30 = inttoptr i64 %27 to %struct.tracepoint_func*
  %31 = getelementptr inbounds %struct.tracepoint_func, %struct.tracepoint_func* %30, i64 0, i32 1
  %32 = load i8*, i8** %31, align 8
  %33 = call i32 @__traceiter_xfs_read_extent(i8* %32, %struct.xfs_inode* %0, %struct.xfs_iext_cursor* %1, i32 %2, i64 ptrtoint (i8* blockaddress(@xfs_iread_bmbt_block, %73) to i64)) #11
  br label %34

Note blockaddress(@xfs_iread_bmbt_block, %73) is referencing a block in another function.

[nickdesaulniers]

Note blockaddress(@xfs_iread_bmbt_block, %73) is referencing a block in another function.

This is one thing I've always disliked (or perhaps simply misunderstood) about LLVM's blockaddress Constant. In C, labels are scoped to functions, but in LLVM IR, they're not necessarily (the two operands are the Function and BasicBlock). I haven't yet seen such a case where a Function contains a blockaddress whose Function operand was a different Function, but perhaps this is a case I should sit down and look at further.

[nickdesaulniers]

I wasn't able to reproduce with the following configs:

$ grep -e DYNAMIC_FTRACE -e FUNCTION_TRACER -e LTO_CLANG_THIN -e XFS_FS -n .config
787:CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y
791:CONFIG_LTO_CLANG_THIN=y
8069:CONFIG_XFS_FS=y
8197:# CONFIG_VXFS_FS is not set
8919:CONFIG_HAVE_FUNCTION_TRACER=y
8921:CONFIG_HAVE_DYNAMIC_FTRACE=y
8922:CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
8935:CONFIG_FUNCTION_TRACER=y
8937:CONFIG_DYNAMIC_FTRACE=y
8938:CONFIG_DYNAMIC_FTRACE_WITH_REGS=y

@arndb was this with thin LTO or full lto? Can you post the config if it's still reproducible?

Non-reproducible with full LTO either, though I did get:

incomplete ORC unwind tables in file: vmlinux
Failed to sort kernel tables

[samitolvanen]

I can reproduce this with full LTO at least:

$ grep -E '(LTO_|XFS)' kernel-build/.config
CONFIG_LTO_CLANG=y
CONFIG_ARCH_SUPPORTS_LTO_CLANG=y
CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y
CONFIG_HAS_LTO_CLANG=y
# CONFIG_LTO_NONE is not set
CONFIG_LTO_CLANG_FULL=y
# CONFIG_LTO_CLANG_THIN is not set
CONFIG_XFS_FS=m
CONFIG_XFS_SUPPORT_V4=y
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
CONFIG_XFS_ONLINE_SCRUB=y
CONFIG_XFS_ONLINE_REPAIR=y
CONFIG_XFS_DEBUG=y
CONFIG_XFS_ASSERT_FATAL=y
CONFIG_VXFS_FS=m

[nickdesaulniers]

I was able to repro this just now building ARCH=i386 allmodconfig built with full LTO while testing this. I had to disable FTRACE, COMPILE_TEST, and GCOV.

ld.lld: error: Never resolved function from blockaddress (Producer: 'LLVM13.0.0git' Reader: 'LLVM 13.0.0git')
  LTO [M] lib/test_xarray.lto.o
make[1]: *** [scripts/Makefile.modpost:134: fs/xfs/xfs.lto.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:1882: modules] Error 2

[nickdesaulniers]

I gave this a shot today, but wasn't able to repro with ToT LLVM (~clang-13) for aarch64+full LTO (or thin LTO) + XFS_FS. Strange.

[nickdesaulniers]

I'm able to repro on i386 on next (see #1395 for config). (I see now that Arnd mentioned CONFIG_DYNAMIC_FTRACE; I should re-test with that).