filetrace doctor: check whether a machine can be monitored

Reference for filetrace doctor: kernel and BPF checks, every probe attached for the check, the features they add up to, the pipeline and the status.

filetrace doctor checks whether this machine can be monitored and how well. It checks the kernel, BTF, BPF and the capabilities needed to load programs, then loads every probe into the kernel, attaches it for the duration of the check and removes it again. It reports what the attached probes add up to, what a running agent has received and dropped, and ends with a status. It does not need a running agent.

​

Usage

filetrace doctor [--probes auto|trampoline|kprobe] [--json]

Run it as root: loading BPF programs needs root, or CAP_BPF with CAP_PERFMON.

sudo filetrace doctor

FILETRACE DIAGNOSTICS

Platform:           Linux
Kernel:             7.0.14-linuxkit
Architecture:       aarch64
BTF:                AVAILABLE
CAP_BPF:            AVAILABLE
CAP_PERFMON:        AVAILABLE
BPF:                AVAILABLE (libbpf v1.5)
BPF trampolines:    AVAILABLE
BPF programs:       AVAILABLE (loaded; every kernel field they read was found)
Backend:            Linux eBPF (trampolines)

PROBES
  [OK] tracepoint sched_process_exec
  [OK] tracepoint sched_process_fork
  [OK] tracepoint sched_process_exit
  [OK] tracepoint inet_sock_set_state
  [OK] fexit inet_sendmsg
  [OK] fexit inet_recvmsg
  [OK] fexit inet6_sendmsg
  [OK] fexit inet6_recvmsg
  [OK] fexit vfs_read
  [OK] fexit do_splice_direct
  [OK] fentry __destroy_inode
  Each probe above was loaded into the kernel and attached for this check.

FEATURES
  [OK] Process tracking
  [NONE] File open tracking: open and close are not traced; a file is identified when it is read or sent
  [WARN] File read tracking: read(2) and pread(2) on regular files; readv, io_uring and memory-mapped reads are not observed
  [NONE] File write tracking: writes are not traced
  [OK] TCP tracking
  [OK] UDP tracking
  [OK] IPv6
  [OK] Sendfile attribution
  [WARN] Read+send correlation: a thread that reads a file and then sends is reported as INFERRED; reads and sends on different threads are not associated
  [NONE] Splice attribution: transfers through pipes with splice(2) are not followed
  [WARN] io_uring attribution: bytes sent through io_uring are counted per socket; file reads through io_uring are not observed
  [NONE] Mmap provenance: data read through memory mappings is not observed
  [NONE] Packet-level egress: bytes are counted when the socket accepts them, not per packet
  [WARN] Container namespaces: cgroup and namespace identifiers are recorded per process; container names are not resolved

EVENT PIPELINE
  Agent:            not running; start it with 'sudo filetrace agent run'

STATUS: READY

Every probe this build uses is attached. Run 'filetrace capabilities'
for the list of features and their limits.

​

Reading the report

​

System checks

The first block names the platform, the kernel release and the architecture, then gives each check as AVAILABLE, LIMITED or MISSING, with a detail in parentheses where there is one:

Check	What it verifies
Kernel version	Shown only when the version cannot be determined or is older than 6.1, the oldest kernel Filetrace is built for
BTF	/sys/kernel/btf/vmlinux exists; the kernel must be built with CONFIG_DEBUG_INFO_BTF
CAP_BPF, CAP_PERFMON	The capabilities to load and attach BPF programs; the detail says when they are held through CAP_SYS_ADMIN
Kernel lockdown	Shown only when lockdown in confidentiality mode forbids reading kernel memory from BPF
BPF	The kernel can create BPF ring buffers and task storage maps; the detail gives the libbpf version
BPF trampolines	Kernel functions can be traced through BPF trampolines. When they cannot, the check is LIMITED, gives the reason and says that kprobes are used instead. Not shown with --probes kprobe.
BPF programs	The programs load, and every kernel field they read was found
Backend	The backend and the probe set in use: Linux eBPF (trampolines) or Linux eBPF (kprobes)

​

PROBES

Every instrumentation point, loaded into the kernel and attached for the duration of the check, then removed. A probe is reported as attached only if the kernel accepted the program and every kernel field it reads was found.

Tag	Meaning
[OK]	Attached
[WARN]	Not attached; the probe is optional and the features that depend on it are reduced. The reason follows.
[FAIL]	Not attached, and the probe is required. The reason follows.

The probes that count network bytes are required: inet_sendmsg and inet_recvmsg in the trampoline set; tcp_sendmsg, tcp_cleanup_rbuf, inet_sock_destruct, ip_send_skb and skb_consume_udp in the kprobe set, which also requires the inet_sock_set_state tracepoint. The IPv6 and file probes are optional.

Kernel functions are traced through BPF trampolines (fentry and fexit) where the kernel has them, and through kprobes otherwise. --probes checks one of the two sets. The kprobe set probes different functions and reads TCP byte counts from the sockets’ own counters:

sudo filetrace doctor --probes kprobe

BPF programs:       AVAILABLE (loaded; every kernel field they read was found)
Backend:            Linux eBPF (kprobes)

PROBES
  [OK] tracepoint sched_process_exec
  [OK] tracepoint sched_process_fork
  [OK] tracepoint sched_process_exit
  [OK] tracepoint inet_sock_set_state
  [OK] kprobe tcp_sendmsg
  [OK] kprobe tcp_cleanup_rbuf
  [OK] kprobe inet_sock_destruct
  [OK] kprobe ip_send_skb
  [OK] kprobe skb_consume_udp
  [OK] kprobe ip6_send_skb
  [OK] kprobe vfs_read
  [OK] kprobe do_splice_direct
  [OK] kprobe __destroy_inode
  Each probe above was loaded into the kernel and attached for this check.

That report ends with:

STATUS: READY

Kernel functions are traced with kprobes instead of BPF trampolines:
each traced call costs more, and TCP bytes are read from the
sockets' own counters. Run 'filetrace capabilities' for the
differences.

Kernel instrumentation describes what each probe records.

​

FEATURES

What the attached probes add up to, one line per feature, with the reason when a feature is not available in full:

Tag	Meaning
[OK]	Available in full
[WARN]	Available in part
[OFF]	Turned off
[NONE]	Not available

When every feature has the same status for the same reason, as when privileges are missing, the reason is stated once on a Reason: line. capabilities lists the same features for the running agent.

​

EVENT PIPELINE

What the running agent received and dropped. With an agent running, the section gives its PID and mode, the events received and dropped, and the sizes of the ring buffer and the event log. Otherwise it says that the agent is not running, or that it is running but its socket is not accessible to your user.

​

STATUS

Status	Meaning	Exit code
READY	Every probe this build uses is attached	0
NEEDS PRIVILEGES	The check could not load BPF programs; run it with sudo	3
UNAVAILABLE	Monitoring is not possible: BTF or BPF support is missing, the programs do not load, or no network probe attaches	5
DEGRADED	Some probes are missing, or the running agent has lost events within the last 60 seconds	9

A few lines of advice follow the status: which command to run next, or what the kernel needs. Without privileges, the BPF check is LIMITED with the detail not checked, every feature is reported as not verified, and the advice is to run sudo filetrace doctor.

​

Options

--probes · string · default: "auto"

Probe set to check: auto, trampoline or kprobe. auto tries BPF trampolines and falls back to kprobes.

--format · string · default: "table"

table or json.

--json · boolean

Short for --format json.

​

JSON output

sudo filetrace doctor --json

{"status":"ready","platform":"Linux","kernel":"7.0.14-linuxkit","architecture":"aarch64","backend":"Linux eBPF (trampolines)","checks":[{"name":"BTF","status":"ok"},{"name":"CAP_BPF","status":"ok"},{"name":"CAP_PERFMON","status":"ok"},{"name":"BPF","status":"ok","detail":"libbpf v1.5"},{"name":"BPF trampolines","status":"ok"},{"name":"BPF programs","status":"ok","detail":"loaded; every kernel field they read was found"}],"probes":[{"name":"tracepoint sched_process_exec","status":"ok"},{"name":"tracepoint sched_process_fork","status":"ok"},{"name":"tracepoint sched_process_exit","status":"ok"},{"name":"tracepoint inet_sock_set_state","status":"ok"},{"name":"fexit inet_sendmsg","status":"ok"},{"name":"fexit inet_recvmsg","status":"ok"},{"name":"fexit inet6_sendmsg","status":"ok"},{"name":"fexit inet6_recvmsg","status":"ok"},{"name":"fexit vfs_read","status":"ok"},{"name":"fexit do_splice_direct","status":"ok"},{"name":"fentry __destroy_inode","status":"ok"}],"features":[{"feature":"process_tracking","status":"FULL"},{"feature":"file_open_tracking","status":"UNAVAILABLE","detail":"open and close are not traced; a file is identified when it is read or sent"},{"feature":"file_read_tracking","status":"PARTIAL","detail":"read(2) and pread(2) on regular files; readv, io_uring and memory-mapped reads are not observed"},{"feature":"file_write_tracking","status":"UNAVAILABLE","detail":"writes are not traced"},{"feature":"tcp_tracking","status":"FULL"},{"feature":"udp_tracking","status":"FULL","detail":"received datagrams show their sender only when the application asks for it"},{"feature":"ipv6","status":"FULL"},{"feature":"sendfile_attribution","status":"FULL"},{"feature":"read_send_correlation","status":"PARTIAL","detail":"a thread that reads a file and then sends is reported as INFERRED; reads and sends on different threads are not associated"},{"feature":"splice_attribution","status":"UNAVAILABLE","detail":"transfers through pipes with splice(2) are not followed"},{"feature":"io_uring_attribution","status":"PARTIAL","detail":"bytes sent through io_uring are counted per socket; file reads through io_uring are not observed"},{"feature":"mmap_provenance","status":"UNAVAILABLE","detail":"data read through memory mappings is not observed"},{"feature":"packet_level_egress","status":"UNAVAILABLE","detail":"bytes are counted when the socket accepts them, not per packet"},{"feature":"container_namespaces","status":"PARTIAL","detail":"cgroup and namespace identifiers are recorded per process; container names are not resolved"}],"agent":{"state":"stopped"}}

status is ready, needs_privileges, unavailable or degraded. Each check and probe has a status of ok, warn or fail, and each feature one of FULL, PARTIAL, DISABLED or UNAVAILABLE. agent.state is running, stopped or unreachable; a running agent adds pid, mode, events_received, events_dropped, ring_buffer_bytes and event_log_bytes.

​

Examples

sudo filetrace doctor
sudo filetrace doctor --probes kprobe
sudo filetrace doctor --json

Troubleshooting explains what to do about each status other than READY.

​

Exit codes

Code	Meaning
0	READY
2	Invalid arguments
3	NEEDS PRIVILEGES
5	UNAVAILABLE
6	The configuration file cannot be read or is invalid
9	DEGRADED: some probes are missing, or the agent is losing events