> ## Documentation Index
> Fetch the complete documentation index at: https://filetrace.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability recipes

> End-to-end recipes that combine Filetrace with Prometheus, Grafana and an OpenTelemetry Collector: alert rules, dashboard queries, collector routing, and joining file transfers with application telemetry.

This page turns the [Prometheus](https://filetrace.mintlify.site/integrations/prometheus.md) and [OTLP](https://filetrace.mintlify.site/integrations/opentelemetry.md) exporters into working observability: alert rules you can paste into Prometheus, panels for a Grafana dashboard, a collector that routes logs and metrics to different backends, and queries that join Filetrace with the rest of your telemetry.

The examples assume the agent serves metrics on `127.0.0.1:9464` and exports OTLP to a collector on `127.0.0.1:4317`.

## What Filetrace adds to your telemetry

Application telemetry tells you that a request was served. Filetrace tells you **which file left the machine, by which process, to which address, and how sure that association is**. The two join on the host and the process:

| Signal | Carries | Use it for |
| - | - | - |
| Metrics | Process name, transport, direction, transfer method, confidence | Rates, spikes, capacity, alerting |
| OTLP log records | `file.path`, `process.pid`, `network.peer.address`, `filetrace.process.script` | Forensics, audit, per-file questions |
| gRPC API | The same records on demand | Interactive tools and dashboards |

Metrics deliberately carry no file path or address, so their series stay bounded. When you need the path or the destination, use a log record or the API.

## Prometheus alert rules

Save this as `filetrace.rules.yml` and point `rule_files` at it. Every expression uses only the metrics and labels on the [Prometheus page](https://filetrace.mintlify.site/integrations/prometheus.md).

```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
groups:
  - name: filetrace
    rules:
      - alert: FiletraceAgentDown
        expr: up{job="filetrace"} == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Filetrace agent on {{ $labels.instance }} is not answering"
          description: "No scrape has succeeded for 5 minutes. File transfers on this host are not being recorded."

      - alert: FiletraceEventLoss
        expr: sum by (instance) (increase(filetrace_events_dropped_total[5m])) > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Filetrace is losing events on {{ $labels.instance }}"
          description: "Attribution is degraded. Check [limits] kernel_counters and [agent] ring_buffer_size; see the performance guide."

      - alert: FiletraceEgressSpike
        expr: |
          sum by (instance) (
            rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[5m])
          ) > 50 * 1024 * 1024
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Outbound traffic on {{ $labels.instance }} is above 50 MB/s"
          description: "Sustained egress above the expected baseline. Check which process is responsible with the dashboard or `filetrace top`."

      - alert: FiletraceLargeFileTransfer
        expr: |
          sum by (instance, process_executable_name) (
            increase(filetrace_file_sent_bytes_total[5m])
          ) > 1024 * 1024 * 1024
        for: 0m
        labels:
          severity: info
        annotations:
          summary: "{{ $labels.process_executable_name }} sent over 1 GB of files in 5 minutes on {{ $labels.instance }}"
          description: "A single process moved more than 1 GB of file data. Confirm that this is expected."

      - alert: FiletraceUnattributedEgress
        expr: |
          (
            sum by (instance) (rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[15m]))
            -
            sum by (instance) (rate(filetrace_file_sent_bytes_total[15m]))
          )
          /
          sum by (instance) (rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[15m]))
          > 0.8
        for: 30m
        labels:
          severity: info
        annotations:
          summary: "Most outbound bytes on {{ $labels.instance }} are not attributed to a file"
          description: "More than 80% of egress has no file association. This is normal for protocols that stream generated data, and worth checking for uploads."

      - alert: FiletraceTransferStuck
        expr: sum by (instance) (filetrace_transfers_active) > 0
        for: 1h
        labels:
          severity: info
        annotations:
          summary: "A file transfer on {{ $labels.instance }} has been in progress for an hour"
          description: "A long-lived transfer may be a stalled upload or a large download. Inspect it with `filetrace transfers --active`."

      - alert: FiletraceMetricOverflow
        expr: count({otel_metric_overflow="true"}) > 0
        for: 15m
        labels:
          severity: info
        annotations:
          summary: "Filetrace metric series overflowed on {{ $labels.instance }}"
          description: "More distinct processes than [limits] metric_series. Raise the limit or accept the overflow series."
```

> [!NOTE]
>
> `FiletraceUnattributedEgress` is informational, not a fault. A server that generates responses in memory, a proxy that streams from another socket, or a database that sends query results all produce egress with no file behind it. Use it to find the processes worth investigating, not to page someone.

## Grafana dashboard

A compact dashboard with five panels. Each query is self-contained; paste it into a Grafana panel with the Prometheus data source that scrapes Filetrace.

**Outbound bytes per second, by process**

```promql theme={"theme":{"light":"github-light","dark":"github-dark"}}
sum by (process_executable_name) (
  rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[5m])
)
```

**File bytes versus unattributed egress**

```promql theme={"theme":{"light":"github-light","dark":"github-dark"}}
sum(rate(filetrace_file_sent_bytes_total[5m])) as "file bytes"
sum(rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[5m])) as "all egress"
```

**Transfers in progress, by confidence**

```promql theme={"theme":{"light":"github-light","dark":"github-dark"}}
sum by (filetrace_transfer_confidence) (filetrace_transfers_active)
```

**Transfers started per minute, by method**

```promql theme={"theme":{"light":"github-light","dark":"github-dark"}}
sum by (filetrace_transfer_method) (rate(filetrace_transfers_started_total[1m])) * 60
```

**Lost events, by kind**

```promql theme={"theme":{"light":"github-light","dark":"github-dark"}}
sum by (filetrace_loss_kind) (rate(filetrace_events_dropped_total[5m]))
```

**Agent resource use**

```promql theme={"theme":{"light":"github-light","dark":"github-dark"}}
rate(process_cpu_time_seconds_total{job="filetrace"}[5m])
process_memory_usage_bytes{job="filetrace"}
```

> [!NOTE]
>
> `process_cpu_time_seconds_total` and `process_memory_usage_bytes` are common names. Select the agent's series with `job="filetrace"` so they do not mix with other exporters.

## OpenTelemetry Collector

The agent sends OTLP/gRPC without TLS to a collector on the same host. The collector adds credentials, batching and routing, and forwards each signal to the backend that fits it: metrics to Prometheus remote write, log records to Loki, and both to a SIEM.

```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert
  # Keep the file path out of backends that must not store it.
  transform:
    log_statements:
      - context: log
        statements:
          - delete_key(attributes, "file.path") where attributes["filetrace.event.type"] == "file_seen"

exporters:
  prometheusremotewrite:
    endpoint: https://prometheus.example.com/api/v1/write
    headers:
      Authorization: "Bearer ${env:PROM_TOKEN}"
  loki:
    endpoint: https://loki.example.com/loki/api/v1/push
    headers:
      Authorization: "Bearer ${env:LOKI_TOKEN}"
  otlphttp/siem:
    endpoint: https://siem.example.com/otlp
    headers:
      Authorization: "Bearer ${env:SIEM_TOKEN}"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [loki, otlphttp/siem]
```

Point the agent at the collector:

```toml theme={"theme":{"light":"github-light","dark":"github-dark"}}
[otlp]
enabled = true
endpoint = "127.0.0.1:4317"
logs = true
metrics = true
events = ["file_transfer", "event_loss"]
```

> [!WARNING]
>
> `file_paths = true` (the default) sends full paths to the collector. Set `file_paths = false` when the backend must not store them; records then carry the device and inode instead. `command_line = false` (the default) keeps process arguments, which can hold secrets, on the host.

## Joining Filetrace with application telemetry

Filetrace does not emit traces. It emits log records and metrics that share identifiers with the rest of your telemetry, so you can pivot from an application signal to the file that left the machine.

| Join key | Filetrace record | Application telemetry |
| - | - | - |
| Host | `host.name` resource attribute | `host.name` on every signal |
| Process | `process.pid`, `process.executable.name` | `process.pid` in logs and traces |
| Script | `filetrace.process.script` | The route or handler name in a span |
| Peer | `network.peer.address`, `network.peer.port` | `net.peer.name` or `server.address` |
| Time | Event timestamp | Span start and end |

**From a slow request to the file it sent.** In your tracing UI, find the span for the request and note its host and PID. In Loki, query the transfer records for that process in the same window:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="filetrace"}
  | json
  | filetrace_event_type = "file_transfer"
  | process_pid = `1842`
  | line_format "{{.file_path}} -> {{.network_peer_address}} ({{.filetrace_transfer_bytes}} bytes, {{.filetrace_transfer_confidence}})"
```

**From a destination to every file sent to it.** When an address looks wrong, list what went there:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="filetrace"}
  | json
  | filetrace_event_type = "file_transfer"
  | network_peer_address = `203.0.113.25`
  | line_format "{{.file_path}} by {{.process_executable_name}} ({{.filetrace_process_script}})"
```

**From a file to every process that read it.** When a sensitive file is involved, find every sender:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="filetrace"}
  | json
  | filetrace_event_type = "file_transfer"
  | file_path = `/data/customers.csv`
```

The same questions are one command away on the host:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
sudo filetrace inspect file /data/customers.csv
sudo filetrace files --remote 203.0.113.25 --since 24h
sudo filetrace files --script '/srv/shop/*'
```

## Correlating with traces through exemplars

Filetrace metrics are counters and gauges without exemplars, so a Grafana panel cannot jump straight to a trace. Two patterns work well:

1. **Log-based pivot.** Keep `file_transfer` log records in Loki and add a Grafana derived field on `process_pid` that links to your tracing UI filtered by that PID and the record's time range.
2. **Dashboard variable.** Add a `process_executable_name` variable to the dashboard and filter both the Filetrace panels and the application panels by it. The process name is the common key.

## Sizing and retention

* **Metrics** are cheap: a fixed set of series plus one per process, method and confidence, bounded by `[limits] metric_series` (default 2048). Scrape every 15–30 seconds.
* **Log records** are the volume. `file_transfer` records are emitted per transfer, not per byte, but a busy web server can produce thousands per minute. Batch in the collector and set a retention that matches your audit needs.
* **The agent's own history** is in memory and starts when the agent starts. For questions older than a restart, use the metrics and log records you exported, not `filetrace files`.

See [overhead and sizing](https://filetrace.mintlify.site/guides/performance.md) for the queue and table settings, and [metrics and attributes](https://filetrace.mintlify.site/integrations/telemetry-reference.md) for every name.

## Related pages

* [Prometheus](https://filetrace.mintlify.site/integrations/prometheus.md) — enable the page, every metric and label, PromQL examples.
* [OpenTelemetry](https://filetrace.mintlify.site/integrations/opentelemetry.md) — OTLP settings, log record format, delivery and retries.
* [Metrics and attributes reference](https://filetrace.mintlify.site/integrations/telemetry-reference.md) — every exported name.
* [Use cases](https://filetrace.mintlify.site/use-cases/overview.md) — scenarios that put these recipes to work.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.