Kernel instrumentation: trampolines and kprobes

Which tracepoints and kernel functions Filetrace’s BPF programs attach to, how the trampoline and kprobe probe sets differ, and how to choose one.

Filetrace’s agent attaches BPF programs to kernel tracepoints and kernel functions. Kernel functions are traced through BPF trampolines (fentry and fexit) where the kernel can attach them, and through kprobes where it cannot, as on arm64 before Linux 6.4. This page lists what each probe records and explains why the kprobe set reads kernel counters instead of return values.

The programs are compiled once and adapted to the running kernel through CO-RE, using the kernel’s BTF type information. A probe counts as attached only if the kernel accepted the program and every kernel field it reads was found.

​

Choose a probe set

The probes setting selects the set:

Value	Behavior
auto (default)	Trampolines where the kernel supports them, kprobes otherwise.
trampoline	Trampolines only.
kprobe	The kprobe set, on any kernel.

Set it in the configuration file or on the agent’s command line:

[agent]
probes = "auto"

sudo filetrace-agent --probes kprobe
sudo filetrace doctor --probes kprobe   # check one set without starting the agent

filetrace doctor loads and attaches every probe of a set, reports which set is in use and what the kernel accepted, and removes the probes again. filetrace without arguments shows the set the running agent uses, for example Linux eBPF (trampolines), standard mode.

​

Trampoline set

Probe	Records
sched_process_exec, sched_process_fork, sched_process_exit (tracepoints)	Process identity and lifetime
inet_sock_set_state (tracepoint)	TCP connect, listen, establish and close
inet_sendmsg, inet_recvmsg, inet6_sendmsg, inet6_recvmsg (fexit)	Bytes per socket, process and remote endpoint
do_splice_direct (fexit)	sendfile: file and socket in the same kernel operation
vfs_read (fexit)	Which regular file each thread read last, and how much
__destroy_inode (fentry)	The end of a deleted file, after which its inode number can name another

The probes see the kernel’s own file and socket objects, so nothing is keyed on file descriptor numbers, and descriptor reuse cannot misattribute a transfer.

​

Kprobe set

The trampoline set needs the return value of each traced function. Where the kernel cannot attach trampolines, the usual substitute is a return probe (kretprobe), and it does not hold up for this workload: a return probe keeps one of a small pool of instances for as long as the call blocks, as many as there are CPUs on a kernel without full preemption. A few threads waiting in recv are enough for every other return to go unseen.

The kprobe set therefore probes function entries only and reads the results from counters the kernel keeps anyway:

Probe	Records
tcp_sendmsg, tcp_cleanup_rbuf	TCP bytes, as the advance of the socket’s send and receive sequence numbers
inet_sock_set_state	Connection start, and the last bytes before the FIN, which takes a sequence number itself
ip_send_skb, ip6_send_skb, skb_consume_udp	UDP datagrams, each with its length and its peer
do_splice_direct	sendfile: the file, capped by what is left of it, claims the bytes the socket takes next
vfs_read	Which file each thread reads; the size comes from the task’s I/O accounting
inet_sock_destruct, __destroy_inode	The end of a socket and of a deleted file

Sequence numbers cannot miss a call: whatever was sent since the socket was last looked at is the difference.

​

Differences of the kprobe set

• The bytes of the last send on a connection are counted when the socket is next used or closed.
• Kernel TLS sockets count encrypted bytes.
• Raw and ICMP sockets are not counted.
• File reads need a kernel built with CONFIG_TASK_XACCT.
• What a thread reads with readv between a file read and a send is counted as read from that file.
• The kernel holds about 8 MB more in BPF maps for the kprobe set’s socket table.

​

Kernels and architectures

Collection needs Linux 6.1 or later with BTF. Filetrace is tested on Linux 6.1, 6.12 and 7.0 on arm64; Debian’s 6.1 arm64 kernel has no BPF trampolines and exercises the kprobe set as shipped. Continuous integration builds and tests x86-64 and arm64. Other kernels are expected to work through CO-RE but are untested.