Including new instructions from LLVM

[rmnattas]

Newer architecture with new instructions gets included to LLVM and LLVM-MC by the manufacturer.
I'm finding that Capstone does not support the new instructions, some are few years old.

Wondering what's the process of updating LLVM-MC and including the new instructions to capstone?

[kabeor]

In the past, the process is here. However, we are also exploring ways to automate, #1803 and #1831 is the work, but haven't finished yet.

[rmnattas]

Thanks, I'll try to apply it locally and see how it works.

[Rot127]

The new update mechanism @kabeor mentioned will take a while (around 1-2months I guess). LLVM changed quite a lot on there side and manually updating is really work intensive.
I currently work on ARM and will update the other archs afterwards (not x86 though. This will come after the feature is done). Which one do you need the most?

[rmnattas]

I noticed that while trying to apply the script. I'm interested in POWER's ISA3.1 (P10) 8-byte prefix-instructions.

I'm working slowly on it, but still face a bunch of errors between newer LLVM codebase and Capstone's.
What I did so far is:

• Updating both capstone/suite/synctool/tablegen/{PPC,include}
• Updating capstone/include/capstone/ppc.h with the new instructions (PPC_INS_*)
• Removing QReg's (PPC_QF*) from arch/PowerPC/PPCDisassembler.c (because the new .inc files don't have them), is it replaced by something else?
• And still facing a bunch of other errors.

[Rot127]

Removing QReg's (PPC_QF*) from arch/PowerPC/PPCDisassembler.c (because the new .inc files don't have them), is it replaced by something else?

No. PPC QPX is a deprecated CPU feature in LLVM (since LLVM-12 I think(?)). But we still need it in CS since it have to disassemble binaries with those instructions.

I wrote a little guide how you can add the deprecated features to the .td files of LLVM. Generated TableGen files will then contain these deprecated instructions, registers etc. (see: https://github.com/Rot127/capstone/blob/auto-sync/suite/auto-sync/LLVM_Deprecated_Features.md).

As a reference, here is the PPCDeprecated.td file I wrote a few weeks ago:

PPCDeprecated.zip

And still facing a bunch of other errors.

This is also why we try to automate the updating process as much as possible. It is just a too time consuming task.

[rmnattas]

I see that your deprecated file has those registers, but how do I use it or put it exactly? I'm still getting the undefined errors.

[Rot127]

Have you followed this guide?
You only have to do step 4. So if you want to include those QPX registers, you add the following to the end of PPCRegisterInfo.td

#define CAPSTONE_DEPR_REGS
include "PPCDeprecated.td"

Do the same for features, InstrInfo etc. .td files (with the corresponding #CAPSTONE_DEPR_<XXX> macros of cause).

I make a note to update this guide. It is pretty confusingly written I must admit.
Let me know if you have more questions.

[rmnattas]

My bad on that step, thanks. I'm wondering about if it's possible to include llvm code during CS compilation.

I have fixed all compiler error's so far [1] but now I'm getting link errors for new functions that the .inc files use from llvm code that are not in PPCDisassembler.c [2]. I tried briefly to copy them into that file but it started a small chain of dependencies, is there a better way to include the functions (and it's dependencies, like Reg list array etc) other than copying and modifying each?

[1]

update tablegen/PPC
update tablegen/include
keep PPCInstrQPX.td ./
add PPCDependency include at the bottom of ~RegisterInfo.td
add PPCDependency include at the bottom of ~PPCInstrQPX.td
run scripts 
./include/capstone/ppc.h instruction list
copy .inc to arch/....
printBranchOperand add arg in ./arch/PowerPC/PPCInstPrinter.c , uint64_t Address
uint64_t Address = 0; in printInstruction bc printBranchOperand new arg (in arch/PowerPC/PPCGenAsmWriter.inc)
printCustomAliasOperand add arg in arch/PowerPC/PPCInstPrinter.c , uint64_t Address (there's a call, used 0 as the new arg value for now)
include AliasPatternCond ( modified from ./suite/synctools/tablegen/include/llvm/MC/MCInstPrinter.h) [1]
include AliasPattern ( mod from MCInstPrinter.h) [2]
include PatternsForOpcode ( mod from MCInstPrinter.h) [3]
#include "./PPCGenSubtargetInfo.inc" in arch/PowerPC/PPCGenAsmWriter.inc
copy DecoderTableQPX32 and DecoderTablePS32 from old arch/PowerPC/PPCGenDisassemblerTables.inc
remove std_pair<const char *, uint64_t> PPCInstPrinter_getMnemonic from PPCGenAsmWriter.inc, not used...

[2]

undefined reference to `DecodeACCRCRegisterClass'
undefined reference to `decodeCondBrTarget'
undefined reference to `decodeDirectBrTarget'
undefined reference to `DecodeG8pRCRegisterClass'
undefined reference to `decodeImmZeroOperand'
undefined reference to `decodeMemRI34Operands'
undefined reference to `decodeMemRI34PCRelOperands'
undefined reference to `decodeMemRIHashOperands'
undefined reference to `decodeVSRpEvenOperands'
undefined reference to `DecodeVSRpRCRegisterClass'
undefined reference to `printImmZeroOperand'
undefined reference to `printMemRegImm34'
undefined reference to `printMemRegImm34PCRel'
undefined reference to `printMemRegImmHash'
undefined reference to `printS34ImmOperand'
undefined reference to `set_sme_index'
undefined reference to `getMnemonic'
undefined reference to `AArch64InstPrinterValidateMCOperand'
These mostly defined in:
./suite/synctools/tablegen/PPC/Disassembler/PPCDisassembler.cpp
./suite/synctools/tablegen/PPC/MCTargetDesc/PPCInstPrinter.cpp

[Rot127]

I'm wondering about if it's possible to include llvm code during CS compilation.

As you've seen the LLVM code is written in C++ and CS in C. So what would be needed in this case is some kind of adapter code to allow CS to call the LLVM disassembler functions. But this means a partially rewrite of CS, so it is time consuming (but maybe worth it?)

is there a better way to include the functions [...] other than copying and modifying each?

No, this part gets automated with the auto-sync feature as well. The LLVM Disassembler code will simply be translated to C by Python scripts.
If you want to extend those scripts feel free to take a look at them (please tell e before hand so we don't implement the same features).
You can find them here:
https://github.com/Rot127/capstone/tree/auto-sync/suite/auto-sync/CppTranslator

[Rot127]

@rmnattas The refactoring is mostly done and PPC should now support the new instructions.
I haven't added testing for it (this will be done after AArch64 is ready).

But if you find time you are very welcome to test it and give feedback: #2013

[rmnattas]

Thank you, I'm assuming its in your auto-sync-ppc branch. I see that its still fairly active so I might be cloning it at a wrong point.

Had a quick look and trying to generate the tablegen with LLVM 16.0 ran into:

PPC/PPCInstrInfo.td:2568:17: error: Variable not defined: 'int_ppc_mtctr'

---

Another thing that I don't think I mentioned is that llvm doesn't seem to include prefix instructions by default yet, even though they been out in production for a while.

On the March released LLVM 16.0 you need -mcpu=pwr10 or -mattr=prefix-instrs for llvm-mc to recognize them:

$ echo "0x04,0x00,0x00,0x00,0xf4,0xa4,0x00,0x08" | llvm-mc -mcpu=pwr10 --disassemble -triple=ppc64
	.text
	pstd 5, 8(4), 0
$ echo "0x04,0x00,0x00,0x00,0xf4,0xa4,0x00,0x08" | llvm-mc --disassemble -triple=ppc64
	.text
<stdin>:1:1: warning: invalid instruction encoding
0x04,0x00,0x00,0x00,0xf4,0xa4,0x00,0x08
^
<stdin>:1:21: warning: invalid instruction encoding
0x04,0x00,0x00,0x00,0xf4,0xa4,0x00,0x08

Wondering if these options would be needed in the gen-tablegen scripts to include these instructions.

[Rot127]

I see that its still fairly active so I might be cloning it at a wrong point.

We decided to finish up AArch64 before testing and fixing bugs in PPC. This is why you will still see some work there.

Had a quick look and trying to generate the tablegen with LLVM 16.0 ran into:

You used llvm-capstone branch auto-sync?
You can also try to use the more modern branch for AArch64.

Wondering if these options would be needed in the gen-tablegen scripts to include these instructions.

No. The table generation always generates the decoder and printer for all instructions of a target.
The disassembler then checks dynamically for enabled features (mcpu=pwr10 etc.) during disassembly to decide if the bytes are a valid instruction or not.

Capstone enables all features by default, if it doesn't lead to collisions how to interpret bytes. The bytes in your example don't get disassembled in Capstone. But I'll look into it.