Skip to content

Commit 6a84922

Browse files
Shahab VahediKernel Patches Daemon
Shahab Vahedi
authored and
Kernel Patches Daemon
committed
ARC: Add eBPF JIT support
This will add eBPF JIT support to the 32-bit ARCv2 processors. The implementation is qualified by running the BPF tests on a Synopsys HSDK board with "ARC HS38 v2.1c at 500 MHz" as the 4-core CPU. The test_bpf.ko reports 2-10 fold improvements in execution time of its tests. For instance: test_bpf: #33 tcpdump port 22 jited:0 704 1766 2104 PASS test_bpf: #33 tcpdump port 22 jited:1 120 224 260 PASS test_bpf: #141 ALU_DIV_X: 4294967295 / 4294967295 = 1 jited:0 238 PASS test_bpf: #141 ALU_DIV_X: 4294967295 / 4294967295 = 1 jited:1 23 PASS test_bpf: #776 JMP32_JGE_K: all ... magnitudes jited:0 2034681 PASS test_bpf: #776 JMP32_JGE_K: all ... magnitudes jited:1 1020022 PASS Deployment and structure ------------------------ The related codes are added to "arch/arc/net": - bpf_jit.h -- The interface that a back-end translator must provide - bpf_jit_core.c -- Knows how to handle the input eBPF byte stream - bpf_jit_arcv2.c -- The back-end code that knows the translation logic The bpf_int_jit_compile() at the end of bpf_jit_core.c is the entrance to the whole process. Normally, the translation is done in one pass, namely the "normal pass". In case some relocations are not known during this pass, some data (arc_jit_data) is allocated for the next pass to come. This possible next (and last) pass is called the "extra pass". 1. Normal pass # The necessary pass 1a. Dry run # Get the whole JIT length, epilogue offset, etc. 1b. Emit phase # Allocate memory and start emitting instructions 2. Extra pass # Only needed if there are relocations to be fixed 2a. Patch relocations Support status -------------- The JIT compiler supports BPF instructions up to "cpu=v4". However, it does not yet provide support for: - Tail calls - Atomic operations - 64-bit division/remainder - BPF_PROBE_MEM* (exception table) The result of "test_bpf" test suite on an HSDK board is: hsdk-lnx# insmod test_bpf.ko test_suite=test_bpf test_bpf: Summary: 863 PASSED, 186 FAILED, [851/851 JIT'ed] All the failing test cases are due to the ones that were not JIT'ed. Categorically, they can be represented as: .-----------.------------.-------------. | test type | opcodes | # of cases | |-----------+------------+-------------| | atomic | 0xC3, 0xDB | 149 | | div64 | 0x37, 0x3F | 22 | | mod64 | 0x97, 0x9F | 15 | `-----------^------------+-------------| | (total) 186 | `-------------' Setup: build config ------------------- The following configs must be set to have a working JIT test: CONFIG_BPF_JIT=y CONFIG_BPF_JIT_ALWAYS_ON=y CONFIG_TEST_BPF=m The following options are not necessary for the tests module, but are good to have: CONFIG_DEBUG_INFO=y # prerequisite for below CONFIG_DEBUG_INFO_BTF=y # so bpftool can generate vmlinux.h CONFIG_FTRACE=y # CONFIG_BPF_SYSCALL=y # all these options lead to CONFIG_KPROBE_EVENTS=y # having CONFIG_BPF_EVENTS=y CONFIG_PERF_EVENTS=y # Some BPF programs provide data through /sys/kernel/debug: CONFIG_DEBUG_FS=y arc# mount -t debugfs debugfs /sys/kernel/debug Setup: elfutils --------------- The libdw.{so,a} library that is used by pahole for processing the final binary must come from elfutils 0.189 or newer. The support for ARCv2 [1] has been added since that version. [1] https://sourceware.org/git/?p=elfutils.git;a=commit;h=de3d46b3e7 Setup: pahole ------------- The line below in linux/scripts/Makefile.btf must be commented out: pahole-flags-$(call test-ge, $(pahole-ver), 121) += --btf_gen_floats Or else, the build will fail: $ make V=1 ... BTF .btf.vmlinux.bin.o pahole -J --btf_gen_floats \ -j --lang_exclude=rust \ --skip_encoding_btf_inconsistent_proto \ --btf_gen_optimized .tmp_vmlinux.btf Complex, interval and imaginary float types are not supported Encountered error while encoding BTF. ... BTFIDS vmlinux ./tools/bpf/resolve_btfids/resolve_btfids vmlinux libbpf: failed to find '.BTF' ELF section in vmlinux FAILED: load BTF from vmlinux: No data available This is due to the fact that the ARC toolchains generate "complex float" DIE entries in libgcc and at the moment, pahole can't handle such entries. Running the tests ----------------- host$ scp /bld/linux/lib/test_bpf.ko arc: arc # sysctl net.core.bpf_jit_enable=1 arc # insmod test_bpf.ko test_suite=test_bpf ... test_bpf: #1048 Staggered jumps: JMP32_JSLE_X jited:1 697811 PASS test_bpf: Summary: 863 PASSED, 186 FAILED, [851/851 JIT'ed] Acknowledgments --------------- - Claudiu Zissulescu for his unwavering support - Yuriy Kolerov for testing and troubleshooting - Vladimir Isaev for the pahole workaround - Sergey Matyukevich for paving the road by adding the interpreter support Signed-off-by: Shahab Vahedi <[email protected]>
1 parent 459d932 commit 6a84922

File tree

9 files changed

+4611
-2
lines changed

9 files changed

+4611
-2
lines changed

‎Documentation/admin-guide/sysctl/net.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ two flavors of JITs, the newer eBPF JIT currently supported on:
7272
- riscv64
7373
- riscv32
7474
- loongarch64
75+
- arc
7576

7677
And the older cBPF JIT supported on the following archs:
7778

‎Documentation/networking/filter.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -513,7 +513,7 @@ JIT compiler
513513
------------
514514

515515
The Linux kernel has a built-in BPF JIT compiler for x86_64, SPARC,
516-
PowerPC, ARM, ARM64, MIPS, RISC-V and s390 and can be enabled through
516+
PowerPC, ARM, ARM64, MIPS, RISC-V, s390, and ARC and can be enabled through
517517
CONFIG_BPF_JIT. The JIT compiler is transparently invoked for each
518518
attached filter from user space or for internal kernel users if it has
519519
been previously enabled by root::
@@ -650,7 +650,7 @@ before a conversion to the new layout is being done behind the scenes!
650650

651651
Currently, the classic BPF format is being used for JITing on most
652652
32-bit architectures, whereas x86-64, aarch64, s390x, powerpc64,
653-
sparc64, arm32, riscv64, riscv32, loongarch64 perform JIT compilation
653+
sparc64, arm32, riscv64, riscv32, loongarch64, arc perform JIT compilation
654654
from eBPF instruction set.
655655

656656
Testing

‎MAINTAINERS

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3712,6 +3712,12 @@ S: Maintained
37123712
F: Documentation/devicetree/bindings/iio/imu/bosch,bmi323.yaml
37133713
F: drivers/iio/imu/bmi323/
37143714

3715+
BPF JIT for ARC
3716+
M: Shahab Vahedi <[email protected]>
3717+
L: [email protected]
3718+
S: Maintained
3719+
F: arch/arc/net/
3720+
37153721
BPF JIT for ARM
37163722
M: Russell King <[email protected]>
37173723
M: Puranjay Mohan <[email protected]>

‎arch/arc/Kbuild

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# SPDX-License-Identifier: GPL-2.0
22
obj-y += kernel/
33
obj-y += mm/
4+
obj-y += net/
45

56
# for cleaning
67
subdir- += boot

‎arch/arc/Kconfig

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ config ARC
5252
select PCI_SYSCALL if PCI
5353
select HAVE_ARCH_JUMP_LABEL if ISA_ARCV2 && !CPU_ENDIAN_BE32
5454
select TRACE_IRQFLAGS_SUPPORT
55+
select HAVE_EBPF_JIT if ISA_ARCV2
5556

5657
config LOCKDEP_SUPPORT
5758
def_bool y

‎arch/arc/net/Makefile

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# SPDX-License-Identifier: GPL-2.0-only
2+
3+
ifeq ($(CONFIG_ISA_ARCV2),y)
4+
obj-$(CONFIG_BPF_JIT) += bpf_jit_core.o
5+
obj-$(CONFIG_BPF_JIT) += bpf_jit_arcv2.o
6+
endif

‎arch/arc/net/bpf_jit.h

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
/* SPDX-License-Identifier: GPL-2.0 */
2+
/*
3+
* The interface that a back-end should provide to bpf_jit_core.c.
4+
*
5+
* Copyright (c) 2024 Synopsys Inc.
6+
* Author: Shahab Vahedi <[email protected]>
7+
*/
8+
9+
#ifndef _ARC_BPF_JIT_H
10+
#define _ARC_BPF_JIT_H
11+
12+
#include <linux/bpf.h>
13+
#include <linux/filter.h>
14+
15+
/* Print debug info and assert. */
16+
//#define ARC_BPF_JIT_DEBUG
17+
18+
/* Determine the address type of the target. */
19+
#ifdef CONFIG_ISA_ARCV2
20+
#define ARC_ADDR u32
21+
#endif
22+
23+
/*
24+
* For the translation of some BPF instructions, a temporary register
25+
* might be needed for some interim data.
26+
*/
27+
#define JIT_REG_TMP MAX_BPF_JIT_REG
28+
29+
/*
30+
* Buffer access: If buffer "b" is not NULL, advance by "n" bytes.
31+
*
32+
* This macro must be used in any place that potentially requires a
33+
* "buf + len". This way, we make sure that the "buf" argument for
34+
* the underlying "arc_*(buf, ...)" ends up as NULL instead of something
35+
* like "0+4" or "0+8", etc. Those "arc_*()" functions check their "buf"
36+
* value to decide if instructions should be emitted or not.
37+
*/
38+
#define BUF(b, n) (((b) != NULL) ? ((b) + (n)) : (b))
39+
40+
/************** Functions that the back-end must provide **************/
41+
/* Extension for 32-bit operations. */
42+
inline u8 zext(u8 *buf, u8 rd);
43+
/***** Moves *****/
44+
u8 mov_r32(u8 *buf, u8 rd, u8 rs, u8 sign_ext);
45+
u8 mov_r32_i32(u8 *buf, u8 reg, s32 imm);
46+
u8 mov_r64(u8 *buf, u8 rd, u8 rs, u8 sign_ext);
47+
u8 mov_r64_i32(u8 *buf, u8 reg, s32 imm);
48+
u8 mov_r64_i64(u8 *buf, u8 reg, u32 lo, u32 hi);
49+
/***** Loads and stores *****/
50+
u8 load_r(u8 *buf, u8 rd, u8 rs, s16 off, u8 size, bool sign_ext);
51+
u8 store_r(u8 *buf, u8 rd, u8 rs, s16 off, u8 size);
52+
u8 store_i(u8 *buf, s32 imm, u8 rd, s16 off, u8 size);
53+
/***** Addition *****/
54+
u8 add_r32(u8 *buf, u8 rd, u8 rs);
55+
u8 add_r32_i32(u8 *buf, u8 rd, s32 imm);
56+
u8 add_r64(u8 *buf, u8 rd, u8 rs);
57+
u8 add_r64_i32(u8 *buf, u8 rd, s32 imm);
58+
/***** Subtraction *****/
59+
u8 sub_r32(u8 *buf, u8 rd, u8 rs);
60+
u8 sub_r32_i32(u8 *buf, u8 rd, s32 imm);
61+
u8 sub_r64(u8 *buf, u8 rd, u8 rs);
62+
u8 sub_r64_i32(u8 *buf, u8 rd, s32 imm);
63+
/***** Multiplication *****/
64+
u8 mul_r32(u8 *buf, u8 rd, u8 rs);
65+
u8 mul_r32_i32(u8 *buf, u8 rd, s32 imm);
66+
u8 mul_r64(u8 *buf, u8 rd, u8 rs);
67+
u8 mul_r64_i32(u8 *buf, u8 rd, s32 imm);
68+
/***** Division *****/
69+
u8 div_r32(u8 *buf, u8 rd, u8 rs, bool sign_ext);
70+
u8 div_r32_i32(u8 *buf, u8 rd, s32 imm, bool sign_ext);
71+
/***** Remainder *****/
72+
u8 mod_r32(u8 *buf, u8 rd, u8 rs, bool sign_ext);
73+
u8 mod_r32_i32(u8 *buf, u8 rd, s32 imm, bool sign_ext);
74+
/***** Bitwise AND *****/
75+
u8 and_r32(u8 *buf, u8 rd, u8 rs);
76+
u8 and_r32_i32(u8 *buf, u8 rd, s32 imm);
77+
u8 and_r64(u8 *buf, u8 rd, u8 rs);
78+
u8 and_r64_i32(u8 *buf, u8 rd, s32 imm);
79+
/***** Bitwise OR *****/
80+
u8 or_r32(u8 *buf, u8 rd, u8 rs);
81+
u8 or_r32_i32(u8 *buf, u8 rd, s32 imm);
82+
u8 or_r64(u8 *buf, u8 rd, u8 rs);
83+
u8 or_r64_i32(u8 *buf, u8 rd, s32 imm);
84+
/***** Bitwise XOR *****/
85+
u8 xor_r32(u8 *buf, u8 rd, u8 rs);
86+
u8 xor_r32_i32(u8 *buf, u8 rd, s32 imm);
87+
u8 xor_r64(u8 *buf, u8 rd, u8 rs);
88+
u8 xor_r64_i32(u8 *buf, u8 rd, s32 imm);
89+
/***** Bitwise Negate *****/
90+
u8 neg_r32(u8 *buf, u8 r);
91+
u8 neg_r64(u8 *buf, u8 r);
92+
/***** Bitwise left shift *****/
93+
u8 lsh_r32(u8 *buf, u8 rd, u8 rs);
94+
u8 lsh_r32_i32(u8 *buf, u8 rd, u8 imm);
95+
u8 lsh_r64(u8 *buf, u8 rd, u8 rs);
96+
u8 lsh_r64_i32(u8 *buf, u8 rd, s32 imm);
97+
/***** Bitwise right shift (logical) *****/
98+
u8 rsh_r32(u8 *buf, u8 rd, u8 rs);
99+
u8 rsh_r32_i32(u8 *buf, u8 rd, u8 imm);
100+
u8 rsh_r64(u8 *buf, u8 rd, u8 rs);
101+
u8 rsh_r64_i32(u8 *buf, u8 rd, s32 imm);
102+
/***** Bitwise right shift (arithmetic) *****/
103+
u8 arsh_r32(u8 *buf, u8 rd, u8 rs);
104+
u8 arsh_r32_i32(u8 *buf, u8 rd, u8 imm);
105+
u8 arsh_r64(u8 *buf, u8 rd, u8 rs);
106+
u8 arsh_r64_i32(u8 *buf, u8 rd, s32 imm);
107+
/***** Frame related *****/
108+
u32 mask_for_used_regs(u8 bpf_reg, bool is_call);
109+
u8 arc_prologue(u8 *buf, u32 usage, u16 frame_size);
110+
u8 arc_epilogue(u8 *buf, u32 usage, u16 frame_size);
111+
/***** Jumps *****/
112+
/*
113+
* Different sorts of conditions (ARC enum as opposed to BPF_*).
114+
*
115+
* Do not change the order of enums here. ARC_CC_SLE+1 is used
116+
* to determine the number of JCCs.
117+
*/
118+
enum ARC_CC {
119+
ARC_CC_UGT = 0, /* unsigned > */
120+
ARC_CC_UGE, /* unsigned >= */
121+
ARC_CC_ULT, /* unsigned < */
122+
ARC_CC_ULE, /* unsigned <= */
123+
ARC_CC_SGT, /* signed > */
124+
ARC_CC_SGE, /* signed >= */
125+
ARC_CC_SLT, /* signed < */
126+
ARC_CC_SLE, /* signed <= */
127+
ARC_CC_AL, /* always */
128+
ARC_CC_EQ, /* == */
129+
ARC_CC_NE, /* != */
130+
ARC_CC_SET, /* test */
131+
ARC_CC_LAST
132+
};
133+
134+
/*
135+
* A few notes:
136+
*
137+
* - check_jmp_*() are prerequisites before calling the gen_jmp_*().
138+
* They return "true" if the jump is possible and "false" otherwise.
139+
*
140+
* - The notion of "*_off" is to emphasize that these parameters are
141+
* merely offsets in the JIT stream and not absolute addresses. One
142+
* can look at them as addresses if the JIT code would start from
143+
* address 0x0000_0000. Nonetheless, since the buffer address for the
144+
* JIT is on a word-aligned address, this works and actually makes
145+
* things simpler (offsets are in the range of u32 which is more than
146+
* enough).
147+
*/
148+
bool check_jmp_32(u32 curr_off, u32 targ_off, u8 cond);
149+
bool check_jmp_64(u32 curr_off, u32 targ_off, u8 cond);
150+
u8 gen_jmp_32(u8 *buf, u8 rd, u8 rs, u8 cond, u32 c_off, u32 t_off);
151+
u8 gen_jmp_64(u8 *buf, u8 rd, u8 rs, u8 cond, u32 c_off, u32 t_off);
152+
/***** Miscellaneous *****/
153+
u8 gen_func_call(u8 *buf, ARC_ADDR func_addr, bool external_func);
154+
u8 arc_to_bpf_return(u8 *buf);
155+
/*
156+
* - Perform byte swaps on "rd" based on the "size".
157+
* - If "force" is set, do it unconditionally. Otherwise, consider the
158+
* desired "endian"ness and the host endianness.
159+
* - For data "size"s up to 32 bits, perform a zero-extension if asked
160+
* by the "do_zext" boolean.
161+
*/
162+
u8 gen_swap(u8 *buf, u8 rd, u8 size, u8 endian, bool force, bool do_zext);
163+
164+
#endif /* _ARC_BPF_JIT_H */

0 commit comments

Comments
 (0)