LLJIT running error "Segmentation fault: 11", while linking c++ standard libraries.

[lanistor]

I was using custom c++ functions, and it includes c++ standard libraries.
While i using c++ standard libraries, error occured:

./build/main;
Stack dump:
0.      Program arguments: ./build/main 
0  main                     0x000000010c4f7d8c llvm::sys::PrintStackTrace(llvm::raw_ostream&) + 60
1  main                     0x000000010c4f8349 PrintStackTraceSignalHandler(void*) + 25
2  main                     0x000000010c4f5dc6 llvm::sys::RunSignalHandlers() + 118
3  main                     0x000000010c4fbf0c SignalHandler(int) + 252
4  libsystem_platform.dylib 0x00007fff6af3642d _sigtramp + 29
5  libsystem_platform.dylib 000000000000000000 _sigtramp + 18446603338721827824
6  libsystem_platform.dylib 0x00000001162a8213 _sigtramp + 18446603343388679683
7  libsystem_platform.dylib 0x00000001162a80d1 _sigtramp + 18446603343388679361
8  libsystem_platform.dylib 0x00000001162a801a _sigtramp + 18446603343388679178
9  libsystem_platform.dylib 0x00000001162a7c2d _sigtramp + 18446603343388678173
10 libsystem_platform.dylib 0x00000001162a79ed _sigtramp + 18446603343388677597
11 libsystem_platform.dylib 0x00000001162a7147 _sigtramp + 18446603343388675383
12 libsystem_platform.dylib 0x00000001162a7032 _sigtramp + 18446603343388675106
13 main                     0x000000010c2f63d6 main + 1830
14 libdyld.dylib            0x00007fff6ad3d7fd start + 1
make: *** [start] Segmentation fault: 11

Codes:

main.cpp

#include "llvm/ExecutionEngine/JITLink/JITLinkMemoryManager.h"
#include "llvm/ExecutionEngine/JITSymbol.h"
#include "llvm/ExecutionEngine/Orc/LLJIT.h"
#include "llvm/ExecutionEngine/Orc/ObjectLinkingLayer.h"
#include "llvm/ExecutionEngine/Orc/ThreadSafeModule.h"
#include "llvm/IR/Verifier.h"
#include "llvm/Support/InitLLVM.h"
#include "llvm/Support/MemoryBuffer.h"
#include "llvm/Support/TargetSelect.h"
#include <fstream>
#include <iostream>

using namespace std;
using namespace llvm;
using namespace llvm::orc;

int main(int argc, char *argv[]) {

  InitLLVM X(argc, argv);
  InitializeNativeTarget();
  InitializeNativeTargetAsmPrinter();
  ThreadSafeContext context(std::make_unique<LLVMContext>());

  ExitOnError ExitOnErr;

  auto JTMB = ExitOnErr(JITTargetMachineBuilder::detectHost());
  JTMB.setCodeModel(CodeModel::Small);

  auto jit =
      ExitOnErr(LLJITBuilder()
                    .setJITTargetMachineBuilder(std::move(JTMB))
                    .create());

  jit->getMainJITDylib().addGenerator(
      ExitOnErr(orc::DynamicLibrarySearchGenerator::GetForCurrentProcess(
          jit->getDataLayout().getGlobalPrefix())));

  char        ffi_file[] = "build/ffi";
  llvm::Error err =
      jit->addObjectFile(ExitOnErr(errorOrToExpected(MemoryBuffer::getFileAsStream(ffi_file))));

  if (err) {
    cout << "error addObjectFile" << endl;
    return 1;
  };

  char func_name[]      = "add";
  auto func_add         = ExitOnErr(jit->lookup(func_name));
  int (*func)(int, int) = (int (*)(int, int))func_add.getAddress();
  int result            = func(111, 222);

  cout << "result: " << result << endl;

  return 0;
}

ffi.cpp, which will build and output build/ffifile.

#include <vector>

extern "C" int add(int a, int b) {
  std::vector<int> vc;
  vc.push_back(1);
  return a + b;
}

When vc.push_back(1); was included in ffi.cpp, error occoured, when delete this line , it runs ok.

Here is my environment

OS: MacOSX 10.15.2
LLVM: 10
Clang: 11

Here is my cmake file

cmake_minimum_required(VERSION 3.15.0)

project(jitdemo)

SET (CMAKE_C_COMPILER /usr/bin/clang)
SET (CMAKE_CXX_COMPILER /usr/bin/clang++)

SET ( CMAKE_BUILD_TYPE Debug )

find_package(LLVM REQUIRED CONFIG)

message(STATUS ">>Found LLVM ${LLVM_PACKAGE_VERSION}")
message(STATUS ">>Using LLVMConfig.cmake in: ${LLVM_DIR}")

add_compile_options(-std=c++17)
add_compile_options(-stdlib=libc++)
add_definitions(${LLVM_DEFINITIONS})

include_directories(${LLVM_INCLUDE_DIRS})
include_directories(${PROJECT_SOURCE_DIR})

add_subdirectory(link)

llvm_map_components_to_libnames(llvm_libs support core orcjit irreader nativecodegen)

add_executable(main main.cpp)
target_link_libraries(main link ${llvm_libs})

[androm3da]

Why does it fail? When you step through your program, do you get to func_add.getAddress()? What does it return?

[dwblaikie]

Probably best to build with debug info (-g) and with assertions enabled (including the LLVM libraries you're using - they should be built with both as well) - that's likely to produce a more informative error message.

[lanistor]

@dwblaikie LLVM was built by cmake -G "Ninja" -DLLVM_ENABLE_PROJECTS="clang;" -DCMAKE_BUILD_TYPE=Debug -DLLVM_ENABLE_ASSERTIONS=Yes ../llvm, this will include assertions.
And how to use -g?
This is all the error info now, there is no more.
Will it be the compatibility problem between clang-11 and llvm-10? Or the bug of MacOSX 10.15

[donhinton]

CMAKE will add -g when CMAKE_BUILD_TYPE=Debug. Run ninja -v to see what is actually passed to each target. hth... don

…

On Thu, Jan 2, 2020 at 3:29 AM vifird ***@***.***> wrote: @dwblaikie <https://github.com/dwblaikie> LLVM was built by cmake -G "Ninja" -DLLVM_ENABLE_PROJECTS="clang;" -DCMAKE_BUILD_TYPE=Debug -DLLVM_ENABLE_ASSERTIONS=Yes ../llvm, this will include assertions. And how to use -g? This is all the error info now, there is no more. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#84?email_source=notifications&email_token=AAQVGNXOK7BZ5XJBMUARCIDQ3XF3HA5CNFSM4KBKJVLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH6EWFA#issuecomment-570182420>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQVGNS3ZKV4IAN62I7MTCDQ3XF3HANCNFSM4KBKJVLA> .

[lhames]

Hi Vifird. I am not able to reproduce this locally. How are you building build/ffi?

[lanistor]

@lhames Thanks for concerning. It's built by clang++ -std=c++17 -stdlib=libc++ ffi.cpp -c -o build/ffi.

[lhames]

Are you building build/ffi with the same libc++ that you’re using to build LLVM? Or are you using the built clang++ / libc++ to build build/ffi?

It’s possible that this error is due an ABI incompatibility between libc++ versions.

Side note:

-c -o build/ffi

By convention relocatable objects should have a “.o” extension, so “-c -o build/ffi.o”. You won’t fundamentally break anything by not using the suffix (to my knowledge), but it might be confusing for some tools, or when others look at your project. :)

[lanistor]

@lhames Thanks a lot. I didn't install llvm's libc++, and i think they ( build/main & build/ffi.o ) are using the same libc++, see below output:
build/ffi was built by clang++ ffi.cpp -o build/ffi, without -c, intent to check it's dynamic linking libraries. (The normal build/ffi.o used in LLJIT is not built by this, it's built by clang++ ffi.cpp -c -o build/ffi.o)

➜  jit git:(master) ✗ otool -L build/main            
build/main:
        /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
        /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 800.7.0)
➜  jit git:(master) ✗ otool -L build/ffi
build/ffi:
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 800.7.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)
➜

I had changed my clang to llvm's clang(version 10), and here are my environment variables:
~/.bash_profile

# LLVM
export PATH="/Users/vifird/C/compile/llvm-project/build/bin:$PATH"
export CC="/Users/vifird/C/compile/llvm-project/build/bin/clang"
export CXX="/Users/vifird/C/compile/llvm-project/build/bin/clang++"
export LDFLAGS="-L/Users/vifird/C/compile/llvm-project/build/lib"
export CPPFLAGS="-I/Users/vifird/C/compile/llvm-project/build/include:/Users/vifird/C/compile/llvm-project/llvm/include"
export CPLUS_INCLUDE_PATH="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks"

ffi.cpp

#include <iostream>
#include <vector>

extern "C" int add(int a, int b) {
  std::vector<int> vc;
  vc.push_back(1);
  return a + b;
}

int main() {
  printf(">>main run\n");
  return add(1, 2);
}

[lanistor]

@lhames I had create a demo project to show this error in minimum code: https://github.com/vifird/jit. And i had tested it in two new MacBooks(MacOSX 10.15), those both displayed this error, so i think it may not be a problem of libc++.

[lhames]

@vifird: Is build/ffi an executable? You shouldn’t be able to load that into the JIT at all. But if it’s a relocatable object it shouldn’t show anything listed for otool -L.

I would be inclined to get hold of the preprocessed source for both main and ffi and compare the definition of their vector classes: The most likely explanation here is a difference in the vector definition being used by main and ffi.

I’m out on vacation this week so I may not get a chance to reply further. Please ping me next week though and I can take another look.