Skip to content

Commit db22b13

Browse files
author
Alexei Starovoitov
committed
Merge branch 'replace-config_dmabuf_sysfs_stats-with-bpf'
T.J. Mercier says: ==================== Replace CONFIG_DMABUF_SYSFS_STATS with BPF Until CONFIG_DMABUF_SYSFS_STATS was added [1] it was only possible to perform per-buffer accounting with debugfs which is not suitable for production environments. Eventually we discovered the overhead with per-buffer sysfs file creation/removal was significantly impacting allocation and free times, and exacerbated kernfs lock contention. [2] dma_buf_stats_setup() is responsible for 39% of single-page buffer creation duration, or 74% of single-page dma_buf_export() duration when stressing dmabuf allocations and frees. I prototyped a change from per-buffer to per-exporter statistics with a RCU protected list of exporter allocations that accommodates most (but not all) of our use-cases and avoids almost all of the sysfs overhead. While that adds less overhead than per-buffer sysfs, and less even than the maintenance of the dmabuf debugfs_list, it's still *additional* overhead on top of the debugfs_list and doesn't give us per-buffer info. This series uses the existing dmabuf debugfs_list to implement a BPF dmabuf iterator, which adds no overhead to buffer allocation/free and provides per-buffer info. The list has been moved outside of CONFIG_DEBUG_FS scope so that it is always populated. The BPF program loaded by userspace that extracts per-buffer information gets to define its own interface which avoids the lack of ABI stability with debugfs. This will allow us to replace our use of CONFIG_DMABUF_SYSFS_STATS, and the plan is to remove it from the kernel after the next longterm stable release. [1] https://lore.kernel.org/linux-media/[email protected] [2] https://lore.kernel.org/all/[email protected] v1: https://lore.kernel.org/all/[email protected] v1 -> v2: Make the DMA buffer list independent of CONFIG_DEBUG_FS per Christian König Add CONFIG_DMA_SHARED_BUFFER check to kernel/bpf/Makefile per kernel test robot Use BTF_ID_LIST_SINGLE instead of BTF_ID_LIST_GLOBAL_SINGLE per Song Liu Fixup comment style, mixing code/declarations, and use ASSERT_OK_FD in selftest per Song Liu Add BPF_ITER_RESCHED feature to bpf_dmabuf_reg_info per Alexei Starovoitov Add open-coded iterator and selftest per Alexei Starovoitov Add a second test buffer from the system dmabuf heap to selftests Use the BPF program we'll use in production for selftest per Alexei Starovoitov https://r.android.com/c/platform/system/bpfprogs/+/3616123/2/dmabufIter.c https://r.android.com/c/platform/system/memory/libmeminfo/+/3614259/1/libdmabufinfo/dmabuf_bpf_stats.cpp v2: https://lore.kernel.org/all/[email protected] v2 -> v3: Rebase onto bpf-next/master Move get_next_dmabuf() into drivers/dma-buf/dma-buf.c, along with the new get_first_dmabuf(). This avoids having to expose the dmabuf list and mutex to the rest of the kernel, and keeps the dmabuf mutex operations near each other in the same file. (Christian König) Add Christian's RB to dma-buf: Rename debugfs symbols Drop RFC: dma-buf: Remove DMA-BUF statistics v3: https://lore.kernel.org/all/[email protected] v3 -> v4: Fix selftest BPF program comment style (not kdoc) per Alexei Starovoitov Fix dma-buf.c kdoc comment style per Alexei Starovoitov Rename get_first_dmabuf / get_next_dmabuf to dma_buf_iter_begin / dma_buf_iter_next per Christian König Add Christian's RB to bpf: Add dmabuf iterator v4: https://lore.kernel.org/all/[email protected] v4 -> v5: Add Christian's Acks to all patches Add Song Liu's Acks Move BTF_ID_LIST_SINGLE and DEFINE_BPF_ITER_FUNC closer to usage per Song Liu Fix open-coded iterator comment style per Song Liu Move iterator termination check to its own subtest per Song Liu Rework selftest buffer creation per Song Liu Fix spacing in sanitize_string per BPF CI v5: https://lore.kernel.org/all/[email protected] v5 -> v6: Song Liu: Init test buffer FDs to -1 Zero-init udmabuf_create for future proofing Bail early for iterator fd/FILE creation failure Dereference char ptr to check for NUL in sanitize_string() Move map insertion from create_test_buffers() to test_dmabuf_iter() Add ACK to selftests/bpf: Add test for open coded dmabuf_iter v6: https://lore.kernel.org/all/[email protected] v6 -> v7: Zero uninitialized name bytes following the end of name strings per s390x BPF CI Reorder sanitize_string bounds checks per Song Liu Add Song's Ack to: selftests/bpf: Add test for dmabuf_iter Rebase onto bpf-next/master per BPF CI ==================== Link: https://patch.msgid.link/[email protected] Signed-off-by: Alexei Starovoitov <[email protected]>
2 parents 079e5c5 + 7594dcb commit db22b13

File tree

9 files changed

+632
-22
lines changed

9 files changed

+632
-22
lines changed

‎drivers/dma-buf/dma-buf.c

Lines changed: 78 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,9 @@
1919
#include <linux/anon_inodes.h>
2020
#include <linux/export.h>
2121
#include <linux/debugfs.h>
22+
#include <linux/list.h>
2223
#include <linux/module.h>
24+
#include <linux/mutex.h>
2325
#include <linux/seq_file.h>
2426
#include <linux/sync_file.h>
2527
#include <linux/poll.h>
@@ -35,35 +37,91 @@
3537

3638
static inline int is_dma_buf_file(struct file *);
3739

38-
#if IS_ENABLED(CONFIG_DEBUG_FS)
39-
static DEFINE_MUTEX(debugfs_list_mutex);
40-
static LIST_HEAD(debugfs_list);
40+
static DEFINE_MUTEX(dmabuf_list_mutex);
41+
static LIST_HEAD(dmabuf_list);
4142

42-
static void __dma_buf_debugfs_list_add(struct dma_buf *dmabuf)
43+
static void __dma_buf_list_add(struct dma_buf *dmabuf)
4344
{
44-
mutex_lock(&debugfs_list_mutex);
45-
list_add(&dmabuf->list_node, &debugfs_list);
46-
mutex_unlock(&debugfs_list_mutex);
45+
mutex_lock(&dmabuf_list_mutex);
46+
list_add(&dmabuf->list_node, &dmabuf_list);
47+
mutex_unlock(&dmabuf_list_mutex);
4748
}
4849

49-
static void __dma_buf_debugfs_list_del(struct dma_buf *dmabuf)
50+
static void __dma_buf_list_del(struct dma_buf *dmabuf)
5051
{
5152
if (!dmabuf)
5253
return;
5354

54-
mutex_lock(&debugfs_list_mutex);
55+
mutex_lock(&dmabuf_list_mutex);
5556
list_del(&dmabuf->list_node);
56-
mutex_unlock(&debugfs_list_mutex);
57+
mutex_unlock(&dmabuf_list_mutex);
5758
}
58-
#else
59-
static void __dma_buf_debugfs_list_add(struct dma_buf *dmabuf)
59+
60+
/**
61+
* dma_buf_iter_begin - begin iteration through global list of all DMA buffers
62+
*
63+
* Returns the first buffer in the global list of DMA-bufs that's not in the
64+
* process of being destroyed. Increments that buffer's reference count to
65+
* prevent buffer destruction. Callers must release the reference, either by
66+
* continuing iteration with dma_buf_iter_next(), or with dma_buf_put().
67+
*
68+
* Return:
69+
* * First buffer from global list, with refcount elevated
70+
* * NULL if no active buffers are present
71+
*/
72+
struct dma_buf *dma_buf_iter_begin(void)
6073
{
74+
struct dma_buf *ret = NULL, *dmabuf;
75+
76+
/*
77+
* The list mutex does not protect a dmabuf's refcount, so it can be
78+
* zeroed while we are iterating. We cannot call get_dma_buf() since the
79+
* caller may not already own a reference to the buffer.
80+
*/
81+
mutex_lock(&dmabuf_list_mutex);
82+
list_for_each_entry(dmabuf, &dmabuf_list, list_node) {
83+
if (file_ref_get(&dmabuf->file->f_ref)) {
84+
ret = dmabuf;
85+
break;
86+
}
87+
}
88+
mutex_unlock(&dmabuf_list_mutex);
89+
return ret;
6190
}
6291

63-
static void __dma_buf_debugfs_list_del(struct dma_buf *dmabuf)
92+
/**
93+
* dma_buf_iter_next - continue iteration through global list of all DMA buffers
94+
* @dmabuf: [in] pointer to dma_buf
95+
*
96+
* Decrements the reference count on the provided buffer. Returns the next
97+
* buffer from the remainder of the global list of DMA-bufs with its reference
98+
* count incremented. Callers must release the reference, either by continuing
99+
* iteration with dma_buf_iter_next(), or with dma_buf_put().
100+
*
101+
* Return:
102+
* * Next buffer from global list, with refcount elevated
103+
* * NULL if no additional active buffers are present
104+
*/
105+
struct dma_buf *dma_buf_iter_next(struct dma_buf *dmabuf)
64106
{
107+
struct dma_buf *ret = NULL;
108+
109+
/*
110+
* The list mutex does not protect a dmabuf's refcount, so it can be
111+
* zeroed while we are iterating. We cannot call get_dma_buf() since the
112+
* caller may not already own a reference to the buffer.
113+
*/
114+
mutex_lock(&dmabuf_list_mutex);
115+
dma_buf_put(dmabuf);
116+
list_for_each_entry_continue(dmabuf, &dmabuf_list, list_node) {
117+
if (file_ref_get(&dmabuf->file->f_ref)) {
118+
ret = dmabuf;
119+
break;
120+
}
121+
}
122+
mutex_unlock(&dmabuf_list_mutex);
123+
return ret;
65124
}
66-
#endif
67125

68126
static char *dmabuffs_dname(struct dentry *dentry, char *buffer, int buflen)
69127
{
@@ -115,7 +173,7 @@ static int dma_buf_file_release(struct inode *inode, struct file *file)
115173
if (!is_dma_buf_file(file))
116174
return -EINVAL;
117175

118-
__dma_buf_debugfs_list_del(file->private_data);
176+
__dma_buf_list_del(file->private_data);
119177

120178
return 0;
121179
}
@@ -689,7 +747,7 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
689747
file->f_path.dentry->d_fsdata = dmabuf;
690748
dmabuf->file = file;
691749

692-
__dma_buf_debugfs_list_add(dmabuf);
750+
__dma_buf_list_add(dmabuf);
693751

694752
return dmabuf;
695753

@@ -1630,7 +1688,7 @@ static int dma_buf_debug_show(struct seq_file *s, void *unused)
16301688
size_t size = 0;
16311689
int ret;
16321690

1633-
ret = mutex_lock_interruptible(&debugfs_list_mutex);
1691+
ret = mutex_lock_interruptible(&dmabuf_list_mutex);
16341692

16351693
if (ret)
16361694
return ret;
@@ -1639,7 +1697,7 @@ static int dma_buf_debug_show(struct seq_file *s, void *unused)
16391697
seq_printf(s, "%-8s\t%-8s\t%-8s\t%-8s\texp_name\t%-8s\tname\n",
16401698
"size", "flags", "mode", "count", "ino");
16411699

1642-
list_for_each_entry(buf_obj, &debugfs_list, list_node) {
1700+
list_for_each_entry(buf_obj, &dmabuf_list, list_node) {
16431701

16441702
ret = dma_resv_lock_interruptible(buf_obj->resv, NULL);
16451703
if (ret)
@@ -1676,11 +1734,11 @@ static int dma_buf_debug_show(struct seq_file *s, void *unused)
16761734

16771735
seq_printf(s, "\nTotal %d objects, %zu bytes\n", count, size);
16781736

1679-
mutex_unlock(&debugfs_list_mutex);
1737+
mutex_unlock(&dmabuf_list_mutex);
16801738
return 0;
16811739

16821740
error_unlock:
1683-
mutex_unlock(&debugfs_list_mutex);
1741+
mutex_unlock(&dmabuf_list_mutex);
16841742
return ret;
16851743
}
16861744

‎include/linux/dma-buf.h

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -370,10 +370,8 @@ struct dma_buf {
370370
*/
371371
struct module *owner;
372372

373-
#if IS_ENABLED(CONFIG_DEBUG_FS)
374373
/** @list_node: node for dma_buf accounting and debugging. */
375374
struct list_head list_node;
376-
#endif
377375

378376
/** @priv: exporter specific private data for this buffer object. */
379377
void *priv;
@@ -636,4 +634,6 @@ int dma_buf_vmap(struct dma_buf *dmabuf, struct iosys_map *map);
636634
void dma_buf_vunmap(struct dma_buf *dmabuf, struct iosys_map *map);
637635
int dma_buf_vmap_unlocked(struct dma_buf *dmabuf, struct iosys_map *map);
638636
void dma_buf_vunmap_unlocked(struct dma_buf *dmabuf, struct iosys_map *map);
637+
struct dma_buf *dma_buf_iter_begin(void);
638+
struct dma_buf *dma_buf_iter_next(struct dma_buf *dmbuf);
639639
#endif /* __DMA_BUF_H__ */

‎kernel/bpf/Makefile

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,9 @@ obj-$(CONFIG_BPF_SYSCALL) += relo_core.o
5353
obj-$(CONFIG_BPF_SYSCALL) += btf_iter.o
5454
obj-$(CONFIG_BPF_SYSCALL) += btf_relocate.o
5555
obj-$(CONFIG_BPF_SYSCALL) += kmem_cache_iter.o
56+
ifeq ($(CONFIG_DMA_SHARED_BUFFER),y)
57+
obj-$(CONFIG_BPF_SYSCALL) += dmabuf_iter.o
58+
endif
5659

5760
CFLAGS_REMOVE_percpu_freelist.o = $(CC_FLAGS_FTRACE)
5861
CFLAGS_REMOVE_bpf_lru_list.o = $(CC_FLAGS_FTRACE)

‎kernel/bpf/dmabuf_iter.c

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
// SPDX-License-Identifier: GPL-2.0-only
2+
/* Copyright (c) 2025 Google LLC */
3+
#include <linux/bpf.h>
4+
#include <linux/btf_ids.h>
5+
#include <linux/dma-buf.h>
6+
#include <linux/kernel.h>
7+
#include <linux/seq_file.h>
8+
9+
static void *dmabuf_iter_seq_start(struct seq_file *seq, loff_t *pos)
10+
{
11+
if (*pos)
12+
return NULL;
13+
14+
return dma_buf_iter_begin();
15+
}
16+
17+
static void *dmabuf_iter_seq_next(struct seq_file *seq, void *v, loff_t *pos)
18+
{
19+
struct dma_buf *dmabuf = v;
20+
21+
++*pos;
22+
23+
return dma_buf_iter_next(dmabuf);
24+
}
25+
26+
struct bpf_iter__dmabuf {
27+
__bpf_md_ptr(struct bpf_iter_meta *, meta);
28+
__bpf_md_ptr(struct dma_buf *, dmabuf);
29+
};
30+
31+
static int __dmabuf_seq_show(struct seq_file *seq, void *v, bool in_stop)
32+
{
33+
struct bpf_iter_meta meta = {
34+
.seq = seq,
35+
};
36+
struct bpf_iter__dmabuf ctx = {
37+
.meta = &meta,
38+
.dmabuf = v,
39+
};
40+
struct bpf_prog *prog = bpf_iter_get_info(&meta, in_stop);
41+
42+
if (prog)
43+
return bpf_iter_run_prog(prog, &ctx);
44+
45+
return 0;
46+
}
47+
48+
static int dmabuf_iter_seq_show(struct seq_file *seq, void *v)
49+
{
50+
return __dmabuf_seq_show(seq, v, false);
51+
}
52+
53+
static void dmabuf_iter_seq_stop(struct seq_file *seq, void *v)
54+
{
55+
struct dma_buf *dmabuf = v;
56+
57+
if (dmabuf)
58+
dma_buf_put(dmabuf);
59+
}
60+
61+
static const struct seq_operations dmabuf_iter_seq_ops = {
62+
.start = dmabuf_iter_seq_start,
63+
.next = dmabuf_iter_seq_next,
64+
.stop = dmabuf_iter_seq_stop,
65+
.show = dmabuf_iter_seq_show,
66+
};
67+
68+
static void bpf_iter_dmabuf_show_fdinfo(const struct bpf_iter_aux_info *aux,
69+
struct seq_file *seq)
70+
{
71+
seq_puts(seq, "dmabuf iter\n");
72+
}
73+
74+
static const struct bpf_iter_seq_info dmabuf_iter_seq_info = {
75+
.seq_ops = &dmabuf_iter_seq_ops,
76+
.init_seq_private = NULL,
77+
.fini_seq_private = NULL,
78+
.seq_priv_size = 0,
79+
};
80+
81+
static struct bpf_iter_reg bpf_dmabuf_reg_info = {
82+
.target = "dmabuf",
83+
.feature = BPF_ITER_RESCHED,
84+
.show_fdinfo = bpf_iter_dmabuf_show_fdinfo,
85+
.ctx_arg_info_size = 1,
86+
.ctx_arg_info = {
87+
{ offsetof(struct bpf_iter__dmabuf, dmabuf),
88+
PTR_TO_BTF_ID_OR_NULL },
89+
},
90+
.seq_info = &dmabuf_iter_seq_info,
91+
};
92+
93+
DEFINE_BPF_ITER_FUNC(dmabuf, struct bpf_iter_meta *meta, struct dma_buf *dmabuf)
94+
BTF_ID_LIST_SINGLE(bpf_dmabuf_btf_id, struct, dma_buf)
95+
96+
static int __init dmabuf_iter_init(void)
97+
{
98+
bpf_dmabuf_reg_info.ctx_arg_info[0].btf_id = bpf_dmabuf_btf_id[0];
99+
return bpf_iter_reg_target(&bpf_dmabuf_reg_info);
100+
}
101+
102+
late_initcall(dmabuf_iter_init);
103+
104+
struct bpf_iter_dmabuf {
105+
/*
106+
* opaque iterator state; having __u64 here allows to preserve correct
107+
* alignment requirements in vmlinux.h, generated from BTF
108+
*/
109+
__u64 __opaque[1];
110+
} __aligned(8);
111+
112+
/* Non-opaque version of bpf_iter_dmabuf */
113+
struct bpf_iter_dmabuf_kern {
114+
struct dma_buf *dmabuf;
115+
} __aligned(8);
116+
117+
__bpf_kfunc_start_defs();
118+
119+
__bpf_kfunc int bpf_iter_dmabuf_new(struct bpf_iter_dmabuf *it)
120+
{
121+
struct bpf_iter_dmabuf_kern *kit = (void *)it;
122+
123+
BUILD_BUG_ON(sizeof(*kit) > sizeof(*it));
124+
BUILD_BUG_ON(__alignof__(*kit) != __alignof__(*it));
125+
126+
kit->dmabuf = NULL;
127+
return 0;
128+
}
129+
130+
__bpf_kfunc struct dma_buf *bpf_iter_dmabuf_next(struct bpf_iter_dmabuf *it)
131+
{
132+
struct bpf_iter_dmabuf_kern *kit = (void *)it;
133+
134+
if (kit->dmabuf)
135+
kit->dmabuf = dma_buf_iter_next(kit->dmabuf);
136+
else
137+
kit->dmabuf = dma_buf_iter_begin();
138+
139+
return kit->dmabuf;
140+
}
141+
142+
__bpf_kfunc void bpf_iter_dmabuf_destroy(struct bpf_iter_dmabuf *it)
143+
{
144+
struct bpf_iter_dmabuf_kern *kit = (void *)it;
145+
146+
if (kit->dmabuf)
147+
dma_buf_put(kit->dmabuf);
148+
}
149+
150+
__bpf_kfunc_end_defs();

‎kernel/bpf/helpers.c

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3386,6 +3386,11 @@ BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE)
33863386
BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE)
33873387
BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE | KF_TRUSTED_ARGS)
33883388
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str_dynptr, KF_SLEEPABLE | KF_TRUSTED_ARGS)
3389+
#ifdef CONFIG_DMA_SHARED_BUFFER
3390+
BTF_ID_FLAGS(func, bpf_iter_dmabuf_new, KF_ITER_NEW | KF_SLEEPABLE)
3391+
BTF_ID_FLAGS(func, bpf_iter_dmabuf_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
3392+
BTF_ID_FLAGS(func, bpf_iter_dmabuf_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
3393+
#endif
33893394
BTF_KFUNCS_END(common_btf_ids)
33903395

33913396
static const struct btf_kfunc_id_set common_kfunc_set = {

‎tools/testing/selftests/bpf/bpf_experimental.h

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -591,4 +591,9 @@ extern int bpf_iter_kmem_cache_new(struct bpf_iter_kmem_cache *it) __weak __ksym
591591
extern struct kmem_cache *bpf_iter_kmem_cache_next(struct bpf_iter_kmem_cache *it) __weak __ksym;
592592
extern void bpf_iter_kmem_cache_destroy(struct bpf_iter_kmem_cache *it) __weak __ksym;
593593

594+
struct bpf_iter_dmabuf;
595+
extern int bpf_iter_dmabuf_new(struct bpf_iter_dmabuf *it) __weak __ksym;
596+
extern struct dma_buf *bpf_iter_dmabuf_next(struct bpf_iter_dmabuf *it) __weak __ksym;
597+
extern void bpf_iter_dmabuf_destroy(struct bpf_iter_dmabuf *it) __weak __ksym;
598+
594599
#endif

‎tools/testing/selftests/bpf/config

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@ CONFIG_CRYPTO_AES=y
2222
CONFIG_DEBUG_INFO=y
2323
CONFIG_DEBUG_INFO_BTF=y
2424
CONFIG_DEBUG_INFO_DWARF4=y
25+
CONFIG_DMABUF_HEAPS=y
26+
CONFIG_DMABUF_HEAPS_SYSTEM=y
2527
CONFIG_DUMMY=y
2628
CONFIG_DYNAMIC_FTRACE=y
2729
CONFIG_FPROBE=y
@@ -106,6 +108,7 @@ CONFIG_SECURITY=y
106108
CONFIG_SECURITYFS=y
107109
CONFIG_SYN_COOKIES=y
108110
CONFIG_TEST_BPF=m
111+
CONFIG_UDMABUF=y
109112
CONFIG_USERFAULTFD=y
110113
CONFIG_VSOCKETS=y
111114
CONFIG_VXLAN=y

0 commit comments

Comments
 (0)