Observability recipes

End-to-end recipes that combine Filetrace with Prometheus, Grafana and an OpenTelemetry Collector: alert rules, dashboard queries, collector routing, and joining file transfers with application telemetry.

This page turns the Prometheus and OTLP exporters into working observability: alert rules you can paste into Prometheus, panels for a Grafana dashboard, a collector that routes logs and metrics to different backends, and queries that join Filetrace with the rest of your telemetry.

The examples assume the agent serves metrics on 127.0.0.1:9464 and exports OTLP to a collector on 127.0.0.1:4317.

​

What Filetrace adds to your telemetry

Application telemetry tells you that a request was served. Filetrace tells you which file left the machine, by which process, to which address, and how sure that association is. The two join on the host and the process:

Signal	Carries	Use it for
Metrics	Process name, transport, direction, transfer method, confidence	Rates, spikes, capacity, alerting
OTLP log records	file.path, process.pid, network.peer.address, filetrace.process.script	Forensics, audit, per-file questions
gRPC API	The same records on demand	Interactive tools and dashboards

Metrics deliberately carry no file path or address, so their series stay bounded. When you need the path or the destination, use a log record or the API.

​

Prometheus alert rules

Save this as filetrace.rules.yml and point rule_files at it. Every expression uses only the metrics and labels on the Prometheus page.

groups:
  - name: filetrace
    rules:
      - alert: FiletraceAgentDown
        expr: up{job="filetrace"} == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Filetrace agent on {{ $labels.instance }} is not answering"
          description: "No scrape has succeeded for 5 minutes. File transfers on this host are not being recorded."

      - alert: FiletraceEventLoss
        expr: sum by (instance) (increase(filetrace_events_dropped_total[5m])) > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Filetrace is losing events on {{ $labels.instance }}"
          description: "Attribution is degraded. Check [limits] kernel_counters and [agent] ring_buffer_size; see the performance guide."

      - alert: FiletraceEgressSpike
        expr: |
          sum by (instance) (
            rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[5m])
          ) > 50 * 1024 * 1024
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Outbound traffic on {{ $labels.instance }} is above 50 MB/s"
          description: "Sustained egress above the expected baseline. Check which process is responsible with the dashboard or `filetrace top`."

      - alert: FiletraceLargeFileTransfer
        expr: |
          sum by (instance, process_executable_name) (
            increase(filetrace_file_sent_bytes_total[5m])
          ) > 1024 * 1024 * 1024
        for: 0m
        labels:
          severity: info
        annotations:
          summary: "{{ $labels.process_executable_name }} sent over 1 GB of files in 5 minutes on {{ $labels.instance }}"
          description: "A single process moved more than 1 GB of file data. Confirm that this is expected."

      - alert: FiletraceUnattributedEgress
        expr: |
          (
            sum by (instance) (rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[15m]))
            -
            sum by (instance) (rate(filetrace_file_sent_bytes_total[15m]))
          )
          /
          sum by (instance) (rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[15m]))
          > 0.8
        for: 30m
        labels:
          severity: info
        annotations:
          summary: "Most outbound bytes on {{ $labels.instance }} are not attributed to a file"
          description: "More than 80% of egress has no file association. This is normal for protocols that stream generated data, and worth checking for uploads."

      - alert: FiletraceTransferStuck
        expr: sum by (instance) (filetrace_transfers_active) > 0
        for: 1h
        labels:
          severity: info
        annotations:
          summary: "A file transfer on {{ $labels.instance }} has been in progress for an hour"
          description: "A long-lived transfer may be a stalled upload or a large download. Inspect it with `filetrace transfers --active`."

      - alert: FiletraceMetricOverflow
        expr: count({otel_metric_overflow="true"}) > 0
        for: 15m
        labels:
          severity: info
        annotations:
          summary: "Filetrace metric series overflowed on {{ $labels.instance }}"
          description: "More distinct processes than [limits] metric_series. Raise the limit or accept the overflow series."

FiletraceUnattributedEgress is informational, not a fault. A server that generates responses in memory, a proxy that streams from another socket, or a database that sends query results all produce egress with no file behind it. Use it to find the processes worth investigating, not to page someone.

​

Grafana dashboard

A compact dashboard with five panels. Each query is self-contained; paste it into a Grafana panel with the Prometheus data source that scrapes Filetrace.

Outbound bytes per second, by process

sum by (process_executable_name) (
  rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[5m])
)

File bytes versus unattributed egress

sum(rate(filetrace_file_sent_bytes_total[5m])) as "file bytes"
sum(rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[5m])) as "all egress"

Transfers in progress, by confidence

sum by (filetrace_transfer_confidence) (filetrace_transfers_active)

Transfers started per minute, by method

sum by (filetrace_transfer_method) (rate(filetrace_transfers_started_total[1m])) * 60

Lost events, by kind

sum by (filetrace_loss_kind) (rate(filetrace_events_dropped_total[5m]))

Agent resource use

rate(process_cpu_time_seconds_total{job="filetrace"}[5m])
process_memory_usage_bytes{job="filetrace"}

process_cpu_time_seconds_total and process_memory_usage_bytes are common names. Select the agent’s series with job="filetrace" so they do not mix with other exporters.

​

OpenTelemetry Collector

The agent sends OTLP/gRPC without TLS to a collector on the same host. The collector adds credentials, batching and routing, and forwards each signal to the backend that fits it: metrics to Prometheus remote write, log records to Loki, and both to a SIEM.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert
  # Keep the file path out of backends that must not store it.
  transform:
    log_statements:
      - context: log
        statements:
          - delete_key(attributes, "file.path") where attributes["filetrace.event.type"] == "file_seen"

exporters:
  prometheusremotewrite:
    endpoint: https://prometheus.example.com/api/v1/write
    headers:
      Authorization: "Bearer ${env:PROM_TOKEN}"
  loki:
    endpoint: https://loki.example.com/loki/api/v1/push
    headers:
      Authorization: "Bearer ${env:LOKI_TOKEN}"
  otlphttp/siem:
    endpoint: https://siem.example.com/otlp
    headers:
      Authorization: "Bearer ${env:SIEM_TOKEN}"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [loki, otlphttp/siem]

Point the agent at the collector:

[otlp]
enabled = true
endpoint = "127.0.0.1:4317"
logs = true
metrics = true
events = ["file_transfer", "event_loss"]

file_paths = true (the default) sends full paths to the collector. Set file_paths = false when the backend must not store them; records then carry the device and inode instead. command_line = false (the default) keeps process arguments, which can hold secrets, on the host.

​

Joining Filetrace with application telemetry

Filetrace does not emit traces. It emits log records and metrics that share identifiers with the rest of your telemetry, so you can pivot from an application signal to the file that left the machine.

Join key	Filetrace record	Application telemetry
Host	host.name resource attribute	host.name on every signal
Process	process.pid, process.executable.name	process.pid in logs and traces
Script	filetrace.process.script	The route or handler name in a span
Peer	network.peer.address, network.peer.port	net.peer.name or server.address
Time	Event timestamp	Span start and end

From a slow request to the file it sent. In your tracing UI, find the span for the request and note its host and PID. In Loki, query the transfer records for that process in the same window:

{service_name="filetrace"}
  | json
  | filetrace_event_type = "file_transfer"
  | process_pid = `1842`
  | line_format "{{.file_path}} -> {{.network_peer_address}} ({{.filetrace_transfer_bytes}} bytes, {{.filetrace_transfer_confidence}})"

From a destination to every file sent to it. When an address looks wrong, list what went there:

{service_name="filetrace"}
  | json
  | filetrace_event_type = "file_transfer"
  | network_peer_address = `203.0.113.25`
  | line_format "{{.file_path}} by {{.process_executable_name}} ({{.filetrace_process_script}})"

From a file to every process that read it. When a sensitive file is involved, find every sender:

{service_name="filetrace"}
  | json
  | filetrace_event_type = "file_transfer"
  | file_path = `/data/customers.csv`

The same questions are one command away on the host:

sudo filetrace inspect file /data/customers.csv
sudo filetrace files --remote 203.0.113.25 --since 24h
sudo filetrace files --script '/srv/shop/*'

​

Correlating with traces through exemplars

Filetrace metrics are counters and gauges without exemplars, so a Grafana panel cannot jump straight to a trace. Two patterns work well:

1. Log-based pivot. Keep file_transfer log records in Loki and add a Grafana derived field on process_pid that links to your tracing UI filtered by that PID and the record’s time range.
2. Dashboard variable. Add a process_executable_name variable to the dashboard and filter both the Filetrace panels and the application panels by it. The process name is the common key.

​

Sizing and retention

• Metrics are cheap: a fixed set of series plus one per process, method and confidence, bounded by [limits] metric_series (default 2048). Scrape every 15–30 seconds.
• Log records are the volume. file_transfer records are emitted per transfer, not per byte, but a busy web server can produce thousands per minute. Batch in the collector and set a retention that matches your audit needs.
• The agent’s own history is in memory and starts when the agent starts. For questions older than a restart, use the metrics and log records you exported, not filetrace files.

See overhead and sizing for the queue and table settings, and metrics and attributes for every name.

​

Related pages

• Prometheus — enable the page, every metric and label, PromQL examples.
• OpenTelemetry — OTLP settings, log record format, delivery and retries.
• Metrics and attributes reference — every exported name.
• Use cases — scenarios that put these recipes to work.