BOLT instrumentation fails on aarch64 both on linux_x86 host and Mac_M2 (compatible?)

[cirt-os]

I have made attempts to perform bolt optimization on an aarch64 binary，but come accross with such many problems.

1. First I did bolt on on linux_intel-x86, and failed

$ ./build/bin/llvm-bolt test2 -instrument -o t2_instr
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: aarch64
BOLT-INFO: BOLT version: <unknown>
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x200000, offset 0x200000
BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: disabling -align-macro-fusion on non-x86 platform
./build/bin/llvm-bolt: 'test2': No relocation for first DT_FINI_ARRAY slot.

And the error comes from:

Error RewriteInstance::discoverRtFiniAddress() {
  ...
  if (const Relocation *Reloc = FiniArraySection->getDynamicRelocationAt(0)) {
    BC->FiniFunctionAddress = Reloc->Addend;
    return Error::success();
  }

  if (const Relocation *Reloc = FiniArraySection->getRelocationAt(0)) {
    BC->FiniFunctionAddress = Reloc->Value;
    return Error::success();
  }

  return createStringError(std::errc::not_supported,
                           "No relocation for first DT_FINI_ARRAY slot");
}

I checked that the Offset of fini_function in this binary did not start with 0x0 but with 0x8. Interesting, but why would threre be a deviation of offset calculation.

DynamicRelocations: Type size:213 ,Type:1027, Offset:0x8, Addend:0x1460, Value:0x0
DynamicRelocations: Type size:213 ,Type:1027, Offset:0x10, Addend:0x1454, Value:0x0

2. Then I changed the offset manually and recompiled llvm-bolt，And failed again:

BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: aarch64
BOLT-INFO: BOLT version: <unknown>
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x200000, offset 0x200000
BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: disabling -align-macro-fusion on non-x86 platform
BOLT-INFO: number of removed linker-inserted veneers: 0
BOLT-INFO: 0 out of 9 functions in the binary (0.0%) have non-empty execution profile
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 5
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 8
BOLT-INSTRUMENTER: Number of function descriptors: 8
BOLT-INSTRUMENTER: Number of branch counters: 90
BOLT-INSTRUMENTER: Number of ST leaf node counters: 0
BOLT-INSTRUMENTER: Number of direct call counters: 40
BOLT-INSTRUMENTER: Total number of counters: 130
BOLT-INSTRUMENTER: Total size of counters: 1040 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 922 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 4224 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file t1.fdata
BOLT-INFO: Starting stub-insertion pass
BOLT-INFO: Inserted 0 stubs in the hot area and 0 stubs in the cold area. Shared 0 times, iterated 1 times.
BOLT-INFO: padding code to 0x600000 to accommodate hot text
BOLT-ERROR: linking object with arch x86_64 into context with arch aarch64

Bolt cross-compiled on x86 machine seems not supported. The library libbolt_rt_instr.a seems to be generated dynamicly while performing

cmake -G Ninja ../llvm-project/llvm -DLLVM_TARGETS_TO_BUILD="X86;AArch64" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS="bolt"

It seems that, libbolt_rt_instr.a was built on x86, and can't be linked with aarch64 binary, too bad.

3. At last, I used a Mac with M2 chip based on ARM, hoping that it would be compatible with aarch64, and then had a try on MacOs, but unfortunately it turned out to be like this:

BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: aarch64
BOLT-INFO: BOLT version: 058e4454e8965f0d96ecbec61395d5a788c37f7c
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x200000, offset 0x200000
BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: disabling -align-macro-fusion on non-x86 platform
BOLT-INFO: number of removed linker-inserted veneers: 0
BOLT-INFO: 0 out of 9 functions in the binary (0.0%) have non-empty execution profile
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 5
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 8
BOLT-INSTRUMENTER: Number of function descriptors: 8
BOLT-INSTRUMENTER: Number of branch counters: 30
BOLT-INSTRUMENTER: Number of ST leaf node counters: 18
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 48
BOLT-INSTRUMENTER: Total size of counters: 384 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 922 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 4368 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /tmp/prof.fdata
BOLT-INFO: Starting stub-insertion pass
BOLT-INFO: Inserted 0 stubs in the hot area and 0 stubs in the cold area. Shared 0 times, iterated 1 times.
BOLT-INFO: padding code to 0x600000 to accommodate hot text
BOLT-ERROR: library not found: ./build/lib/libbolt_rt_instr.a

So confusing，libbolt_rt_instr.a was not generated. Moreover, the directory build/tools/bolt/bolt_rt-bins did not exist as well.

Over all, are these known problem? Did i miss something, or are there any way to fix them. Thanks.

[yota9]

Hello! One of the problem is probably related to CPU matching in osx cmake which should be fixed with #100866 .
The other problem I see is that osx intr library is compiled with -target x86_64-apple-darwin19.6.0 flag. I'm not sure why 19.6.9 is used and it seems that it would be an issue to compile the library with this flag on arm machine. Maybe @alexander-shaposhnikov might help here to pass proper target on arm machine :) By my self I suggest you to use linux virtual machine on you M2 mac, it should be fine "out of the box"