Filetrace overhead, memory use and sizing

Measured system call overhead and memory use of the Filetrace agent, and how to size its tables, event queue and event log for a busy machine.

The Filetrace agent adds a small cost to the system calls it traces and holds its state in fixed-size memory. This page shows measured overhead per system call, the agent’s memory and CPU use, and the settings that trade memory for completeness on a busy machine.

​

Measured overhead

benchmarks/run.sh in the repository times the traced system calls with and without the agent. One run of each probe set on Linux 7.0.14, arm64, 12 CPUs, inside the Docker Desktop VM, in nanoseconds per call:

Trampolines	No agent	minimal	standard
send, 1 KB over loopback TCP	1448	1571	1580
pread, 4 KB from the page cache	242	241	304
pread + send, 16 KB	2547	2670	2786
sendfile, 64 KB	2592	2446	2333

Kprobes	No agent	minimal	standard
send, 1 KB over loopback TCP	1491	1594	1549
pread, 4 KB from the page cache	240	241	384
pread + send, 16 KB	2552	2744	2981
sendfile, 64 KB	2900	2367	2263

Agent	minimal	standard
Resident memory	6 MB	12 MB
CPU with no workload	0.10%	0.10%
Events lost	0	0

These are measurements of one machine with the exporters off, with run-to-run noise of several percent (the sendfile rows differ by less than that). They are not guarantees. The CPU figure is the mean of twenty one-second samples that ranged from 0.05% to 0.19%, in a VM whose own services keep sending. The kprobe figures come from forcing that set on a kernel that does not need it; kprobes on the kernels that do may cost more.

Besides the agent’s own memory, the kernel holds the BPF maps: roughly 10 MB at the default sizes, and about 8 MB more for the kprobe set’s socket table.

​

Where the cost goes

High-frequency data never becomes one event per system call. Byte counts accumulate in a kernel hash map keyed by socket, process and file; every 250 ms the agent swaps that map for an empty one and reads the idle one. Only process and socket lifecycle events travel through the ring buffer, so a traced send or read updates a counter instead of producing an event for the agent to process.

​

Measure on your machine

sudo filetrace stats
sudo filetrace stats resources --watch

stats shows the events received and dropped, the agent’s CPU and memory, the size and peak utilization of the event queue, and how full each table is:

RESOURCES
CPU:              0.19%
Memory:           11.8 MB
Open FDs:         41

COLLECTOR
Event queue:      4.00 MB
Peak utilization: 0%
Event log:        1.00 MB, holding 163 events
Processes:        20 of 4,096
Sockets:          19 of 16,384
Flows:            13 of 16,384
Files:            17 of 16,384
Transfers:        4 of 16,384

With the Prometheus page on, process_cpu_time_seconds_total, process_memory_usage_bytes and filetrace_events_dropped_total track the same over time.

​

Size the agent

Symptom	Setting	Default
Lost events of kind event_queue, or high peak utilization of the event queue	[agent] ring_buffer_size	4MB (1MB in minimal mode)
Lost events of kind counter_table	[limits] kernel_counters, or a shorter [agent] flush_interval	16384, 250ms
A table at its capacity in stats, so older entries are forgotten early	[limits] processes, sockets, flows, files, transfers, kernel_files	4096 for processes, 16384 for the rest
events --since does not reach back far enough	[agent] event_log_size	1MB (256KB in minimal mode)
Too many label sets in the exported metrics	[limits] metric_series	2048

Larger tables cost memory in the agent and, for the kernel tables, in BPF maps. Restart the agent after changing the configuration.

​

Reduce the cost

• Minimal mode. mode = "minimal" records processes, sockets and network byte counts only, without file identity or attribution. It uses about half the memory of standard mode in the measurement above.
• Turn off what you do not need. [network] tcp, udp, ipv4 and ipv6, and [file] track_read and track_sendfile, each switch a part of collection off.
• Exclude noise. [filters] exclude_processes and exclude_paths leave processes and paths out, for example exclude_processes = ["filetrace-agent"] to leave out the agent’s own traffic to an exporter.

See the configuration reference for every setting.