Monitoring Pack (Grafana)
This page explains how platform administrators can set up the OpenTremor monitoring stack with Docker Compose, connect GlitchTip for error tracking, and import the Grafana dashboard. OpenTremor Core exposes a /metrics endpoint; see Metrics for the metric catalogue.
Start the monitoring stack with Docker Compose
Section titled “Start the monitoring stack with Docker Compose”The docker-compose.yaml file runs Prometheus, Grafana, Loki, and Alloy on the same Docker host as OpenTremor:
GRAFANA_ADMIN_PASSWORD=change-me \OPENTREMOR_NETWORK=opentremor-prod_default \docker compose up -d| Variable | Default | Description |
|---|---|---|
GRAFANA_ADMIN_PASSWORD | Required | Password for Grafana’s admin user |
OPENTREMOR_NETWORK | opentremor-prod_default | Docker network for the OpenTremor stack. Prometheus joins it to scrape opentremor-core:8000 |
GRAFANA_PORT | 3000 | Grafana port, bound to 127.0.0.1. Use an SSH tunnel or reverse proxy to reach it |
Grafana provisions the Prometheus and Loki data sources and the dashboard. Open Dashboards → OpenTremor.
Collect container logs with Alloy and Loki
Section titled “Collect container logs with Alloy and Loki”Alloy reads container stdout through the Docker socket and sends logs to Loki. Loki stores logs on a local volume for 30 days. Alloy and Loki reporting is disabled.
Each log stream has three labels:
| Label | Value |
|---|---|
deployment | The Compose project, such as opentremor-prod |
service | The JSON line’s service value, or the Compose service name for non-JSON lines |
level | The JSON line’s level value. Non-JSON lines have no level |
The request_id, org_id, and job_id fields remain in each log line rather than becoming labels. Search for a request ID with {service="core"} |= "<request id>", or parse JSON fields with | json.
The Docker socket gives Alloy control of the Docker host, even when mounted read-only. Run this stack only on a host you already trust with that access.
You can use your own log stack instead, such as ELK, Datadog or CloudWatch. The services write JSON to stdout, so any collector works. See Logs and Troubleshooting.
The services write JSON to stdout, so you can use another log collector. See Logs and Troubleshooting.
Use your own Prometheus and Grafana containers
Section titled “Use your own Prometheus and Grafana containers”Mount the configuration into your containers:
prometheus: image: prom/prometheus:v3.13.0 volumes: - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
grafana: image: grafana/grafana:13.1.0 volumes: - ./grafana/provisioning:/etc/grafana/provisioning:ro - ./grafana/dashboards:/var/lib/grafana/dashboards:roGrafana provisioning also registers a Loki data source at http://loki:3100. Without Loki, the Logs panels have no data; the other panels still work.
Error tracking: GlitchTip
Section titled “Error tracking: GlitchTip”GlitchTip groups errors from the API, task scheduler, and dashboard with their stack traces. It accepts Sentry event format. The glitchtip/docker-compose.yaml file runs GlitchTip with its own Postgres database on the same Docker host as OpenTremor.
cd glitchtipcp .env.example .env # set GLITCHTIP_SECRET_KEY and GLITCHTIP_DB_PASSWORDdocker compose up -d| Variable | Default | Description |
|---|---|---|
GLITCHTIP_SECRET_KEY | Required | Session signing key. Generate it with openssl rand -hex 32 |
GLITCHTIP_DB_PASSWORD | Required | Password for GlitchTip’s Postgres database |
GLITCHTIP_DOMAIN | http://localhost:8010 | Address people use to open the UI |
GLITCHTIP_PORT | 8010 | UI port, bound to 127.0.0.1. Use an SSH tunnel or reverse proxy to reach it |
OPENTREMOR_NETWORK | opentremor-prod_default | Docker network for the OpenTremor stack. GlitchTip joins it with the glitchtip alias |
EMAIL_URL | — | Optional alert email URL, such as smtp+tls://user:[email protected]:587. Without it, GlitchTip sends no email |
GlitchTip retains events for 30 days. User registration and organization creation are disabled. It runs without Valkey, using Postgres for its cache and task queue.
Connect OpenTremor to GlitchTip
Section titled “Connect OpenTremor to GlitchTip”- Create the first account:
docker compose exec glitchtip ./manage.py createsuperuser. - Open the GlitchTip UI, create an organization, then create a project for each service, such as
core,task-scheduler, anddashboard. You can use one project for all three. - Copy the project DSN from Settings → Client Keys. Replace the DSN host with
glitchtip:8000, which is reachable on the OpenTremor network. For example:http://<key>@glitchtip:8000/<project id>. - Set
SENTRY_DSNto this value in the OpenTremor stack’s.envfile, then restart the stack. Services tag events withservice, so you can filter a shared project by service.
The dashboard forwards browser errors through its server, so GlitchTip does not need a public address. See Logs and Troubleshooting for details about what each service sends.
The 5xx error rate and Logs panels link to GlitchTip. Set the GlitchTip variable at the top of the dashboard to your GlitchTip address and organization slug, such as http://localhost:8010/opentremor. The Logs panel link uses the Request ID value.
Back up the postgres-data volume with the rest of the host to keep GlitchTip’s error history.
Filter metrics by organization
Section titled “Filter metrics by organization”The dashboard’s Organization variable supports multiple selections and defaults to All. The metric panels use this variable to filter by org_id. Requests without a resolved organization, such as health checks, public report links, the GitHub webhook, and SSO callbacks before login, appear under _none.
Monitoring stack files
Section titled “Monitoring stack files”opentremor-monitoring/├── docker-compose.yaml Prometheus, Grafana, Loki, and Alloy├── glitchtip/│ ├── docker-compose.yaml GlitchTip and its Postgres database│ └── .env.example GlitchTip settings├── prometheus/│ └── prometheus.yml Scrape config (target: opentremor-core:8000)├── loki/│ └── loki.yaml Single-binary Loki, filesystem storage, 30-day retention├── alloy/│ └── config.alloy Reads container stdout, sets labels, and sends logs to Loki└── grafana/ ├── provisioning/ │ ├── datasources/prometheus.yml Registers the Prometheus data source │ ├── datasources/loki.yml Registers the Loki data source │ └── dashboards/dashboard.yml Points Grafana to the dashboards folder └── dashboards/ └── opentremor.json Pre-built dashboardSet up Grafana manually
Section titled “Set up Grafana manually”Add the Prometheus data source
Section titled “Add the Prometheus data source”In Grafana, open Connections → Data sources → Add data source → Prometheus. Set the URL to the address Prometheus can reach, such as http://prometheus:9090.
To use the Logs panels, add a Loki data source the same way, such as http://loki:3100.
Import the dashboard
Section titled “Import the dashboard”In Grafana, open Dashboards → Import → Upload JSON file and upload grafana/dashboards/opentremor.json. Select the Prometheus and Loki data sources you added, then click Import.
Dashboard panels
Section titled “Dashboard panels”| Section | Panel | PromQL highlight |
|---|---|---|
| HTTP Traffic | Request rate | rate(http_request_duration_seconds_count[2m]) |
| HTTP Traffic | Latency p50/p95/p99 | histogram_quantile(0.95, ...) |
| HTTP Traffic | 5xx error rate | Ratio of status=~"5.." to total |
| HTTP Traffic | Rate by endpoint | Grouped by handler |
| HTTP Traffic | Rate by status code | Grouped by status |
| Analysis Pipeline | Ingest rate | rate(mcp_ingest_total[2m]) |
| Analysis Pipeline | Resources ingested/s | rate(mcp_ingested_resources_total[2m]) |
| Analysis Pipeline | Analysis submissions/s | rate(mcp_analysis_total[2m]) |
| Analysis Pipeline | LLM response time (p50/p95/p99) | histogram_quantile(0.95, rate(mcp_llm_analysis_duration_seconds_bucket[5m])) |
| Findings | By severity (stacked) | rate(mcp_findings_total[5m]) per severity |
| Findings | By resource type | rate(mcp_findings_total[5m]) per resource_type |
| Totals | Stat panels | Cumulative counters since last restart |
| Backups | Time since last successful backup | time() - max(backup_last_success_timestamp_seconds), red after 48 hours |
| Backups | Last backup size | max(backup_last_size_bytes) |
| Backups | Failed backups (24 h) | sum(increase(backup_failures_total[24h])) |
| Logs | Errors and warnings per minute | count_over_time({level=~"warning|error|critical"}[1m]) per service and level |
| Logs | Logs | {service=~"$service"} |= "$request_id" |
| Logs | Failed jobs | error lines from the job runner |
The Backups panels are platform-wide. Backups have no org_id label, and the Organization variable does not filter them. See Backups and Restore.
The Logs panels use the Service and Request ID variables, not the Organization variable. Enter an X-Request-ID value in Request ID to find all lines for that request, including logs for jobs it started. The 5xx error rate panel links to logs for the same time range.
Scrape metrics in Kubernetes
Section titled “Scrape metrics in Kubernetes”If you use the Prometheus Operator, add a ServiceMonitor. Otherwise, annotate the pod for scraping:
podAnnotations: prometheus.io/scrape: "true" prometheus.io/port: "8000" prometheus.io/path: /metricsSee Helm (Kubernetes) for the values reference.