Getting TVM working with VE

[saudet]

I would like to use TVM to compile models like BERT for VE, but I'm encountering errors from LLVM like these:

LLVM ERROR: Cannot select: 0x10f7dfc8: v16f32 = fadd 0xd33cfa8, 0x99008b8
  0xd33cfa8: v16f32,ch = load<(load 64 from %ir.uglygep1718, !tbaa !247)> 0x2a31c68, 0x99002a0, undef:i64
    0x99002a0: i64 = add 0x10f7e308, 0x9901008
      0x10f7e308: i64,ch = CopyFromReg 0x2a31c68, Register:i64 %12
        0x9900b90: i64 = Register %12
      0x9901008: i64,ch = CopyFromReg 0x2a31c68, Register:i64 %17
        0xd33d280: i64 = Register %17
    0x9900ac0: i64 = undef
  0x99008b8: v16f32 = fadd 0x99001d0, 0x10f7e988
    0x99001d0: v16f32,ch = load<(load 64 from %ir.uglygep56, !tbaa !250)> 0x2a31c68, 0x10f7dc88, undef:i64
      0x10f7dc88: i64 = add 0x9900578, 0x9901008
        0x9900578: i64,ch = CopyFromReg 0x2a31c68, Register:i64 %16
          0x14856630: i64 = Register %16
        0x9901008: i64,ch = CopyFromReg 0x2a31c68, Register:i64 %17
          0xd33d280: i64 = Register %17
      0x9900ac0: i64 = undef
    0x10f7e988: v16f32,ch = load<(load 64 from %ir.uglygep2, !tbaa !253)> 0x2a31c68, 0x10f7dc20, undef:i64
      0x10f7dc20: i64 = add 0x9900648, 0x9901008
        0x9900648: i64,ch = CopyFromReg 0x2a31c68, Register:i64 %15
          0x9900e68: i64 = Register %15
        0x9901008: i64,ch = CopyFromReg 0x2a31c68, Register:i64 %17
          0xd33d280: i64 = Register %17
      0x9900ac0: i64 = undef
In function: __tvm_parallel_lambda
Aborted

What does that mean? Where should I start to debug this?

I build https://github.com/sx-aurora-dev/llvm-project this way on CentOS 7:

cmake ../llvm
make

And https://github.com/apache/incubator-tvm this way:

cmake -DUSE_LLVM=/path/to/llvm-project/build/bin/llvm-config ..

For the default BERT model described here on this blog post:

• https://medium.com/apache-mxnet/speed-up-your-bert-inference-by-3x-on-cpus-using-apache-tvm-9cf7776cd7f8

Using this script:

• https://gist.github.com/icemelon9/860d3d2c9566d6f69fa8112840dd95c1#file-optimize_bert-py
  But replacing everything that comes after target = "llvm -mcpu=skylake-avx512 -libs=cblas" with:

with tvm.transform.PassContext(opt_level=3, required_pass=["FastMath"]):
    compiled_lib = relay.build(mod, "llvm -mtriple=ve-linux", params=params)
compiled_lib.export_library("libbertve.so")

Everything works fine with the x86 target of LLVM, on the same machine with the same binaries, so it's something specific to the VE target. Any help would be greatly appreciated! Thanks in advance

[saudet]

/cc @mikishin @efocht

[mikishin]

Dear Samuel-san,
I am trying to reach the LLVM developer. If I have any update I will let you know ASAP.

[kaz7]

Thank you for letting us know the problem. The develop branch supports only limited vector instructions at the moment, and it causes your problem. @simoll, do you have any information?

[saudet]

@kaz7 Interesting, it does seem to go through when disabling vectorized operations this way:

with tvm.transform.PassContext(opt_level=3, config={"tir.disable_vectorize": True}, required_pass=["FastMath"]):
    compiled_lib = relay.build(mod, "llvm -mtriple=ve-linux", params=params)
compiled_lib.export_library("libbertve.so")

Thanks for the input!

[saudet]

Now I'm getting a Segmentation fault when trying to run the generated code though :(
@kaz7 Any recommendations about how to debug this?

[kaz7]

This is remedy if you don't need vectorization nor vector intrinsic instructions. You can disable vectorizations completely by changing VESubtarget.cpp like below.

diff --git a/llvm/lib/Target/VE/VESubtarget.cpp b/llvm/lib/Target/VE/VESubtarget.cpp
index 23a338a..ab0b730 100644
--- a/llvm/lib/Target/VE/VESubtarget.cpp
+++ b/llvm/lib/Target/VE/VESubtarget.cpp
@@ -28,7 +28,7 @@ void VESubtarget::anchor() {}
 VESubtarget &VESubtarget::initializeSubtargetDependencies(StringRef CPU,
                                                           StringRef FS) {
   // Default feature settings
-  EnableVPU = true;
+  EnableVPU = false;

   // Determine default and user specified characteristics
   std::string CPUName = std::string(CPU);

[saudet]

@kaz7 Thanks for the information. I'm getting Segmentation fault at runtime even with VPU disabled though.

[kaz7]

Sorry for hear that. :-(

[saudet]

Please let me know how I can help debug this, for example, is there a way to enable an info dump of some kind?

[simoll]

Hi @saudet ,

Could you try the hpce/develop branch? That one has vectorization and vector instruction selection - you will get the same SegFault as in the develop branch at runtime but i'd like to hear of any compiler crashes with that branch.
When the SegFault issue is fixed in develop we can merge that into hpce/develop and talk about vector code :)

[saudet]

@simoll Thanks for chipping in! I tried the hpce/develop branch, but it segfaults when importing the model, so something before anything related to the VE target (or the x86 one for that matter) is broken. Would it be possible to update the branch with new commits from develop?

[saudet]

Ok, I got something working. The cause of the Segmentation fault at runtime was with NCC, apparently problems with thread_local that were fixed with 3.1.0. The next issue that I faced was an undefined symbol: __ve_grow_stack, which I was able to work around by compiling and linking manually with grow_stack.S via NCC. (Is this intended? Or is this a codegen bug?) Now, I'm able to run the BERT model correctly, on all 8 threads, but it's running about 100x slower than expected. This is without vectorization, which I'm assuming would speed that up by a factor of maybe 10x, so what could be the cause for the remaining 10x? In any case, I plan to keep investigating this, but please let me know if you think of something! Thanks

[saudet]

Ah, here's the missing piece: BLAS. I was able to link it with blas_openmp from NLC 2.1.0, and that made it over 10x faster. It's now down to about 400 ms for a batch of 1 sequence of length 128 (and about 7000 ms for a batch of 32), which is still about 10x slower than what we'd expect, but I'm assuming that vectorized operations are the final piece of the puzzle...

[jam7]

I'm glad to hear that problems is fixed. Regarding to __ve_grow_stack, it is used in generated code by llvm for VE. If you use clang to link a target program, the particular library is automatically linked. If you use ncc, ncc doesn't know it. Please link $LLVM_INSTALL_PATH/lib/clang/12.0.0/lib/linux/libclang_rt.builtins-ve.a manually. We have plan to inline those code in the future to avoid this problem, but it is not finished atm.

[saudet]

I tried with the hpce/develop branch again (currently at llvm-ve-rv-1.8.0), and I'm not getting a Segmentation fault anymore when importing/exporting the model with TVM, so that's been fixed! It's also happily generating something for TVM's vectorized operations too! 👍 However, I am getting a Segmentation fault at runtime now, even when TVM's vectorizer is disabled.
@simoll Would you have some recommendations about what I should be looking at first?