Skip to content
DashboardWebsite

Monitoring Pack (Grafana)

This page explains how platform administrators can set up the OpenTremor monitoring stack with Docker Compose, connect GlitchTip for error tracking, and import the Grafana dashboard. OpenTremor Core exposes a /metrics endpoint; see Metrics for the metric catalogue.


Start the monitoring stack with Docker Compose

Section titled “Start the monitoring stack with Docker Compose”

The docker-compose.yaml file runs Prometheus, Grafana, Loki, and Alloy on the same Docker host as OpenTremor:

Terminal window
GRAFANA_ADMIN_PASSWORD=change-me \
OPENTREMOR_NETWORK=opentremor-prod_default \
docker compose up -d
VariableDefaultDescription
GRAFANA_ADMIN_PASSWORDRequiredPassword for Grafana’s admin user
OPENTREMOR_NETWORKopentremor-prod_defaultDocker network for the OpenTremor stack. Prometheus joins it to scrape opentremor-core:8000
GRAFANA_PORT3000Grafana port, bound to 127.0.0.1. Use an SSH tunnel or reverse proxy to reach it

Grafana provisions the Prometheus and Loki data sources and the dashboard. Open Dashboards → OpenTremor.

Collect container logs with Alloy and Loki

Section titled “Collect container logs with Alloy and Loki”

Alloy reads container stdout through the Docker socket and sends logs to Loki. Loki stores logs on a local volume for 30 days. Alloy and Loki reporting is disabled.

Each log stream has three labels:

LabelValue
deploymentThe Compose project, such as opentremor-prod
serviceThe JSON line’s service value, or the Compose service name for non-JSON lines
levelThe JSON line’s level value. Non-JSON lines have no level

The request_id, org_id, and job_id fields remain in each log line rather than becoming labels. Search for a request ID with {service="core"} |= "<request id>", or parse JSON fields with | json.

The Docker socket gives Alloy control of the Docker host, even when mounted read-only. Run this stack only on a host you already trust with that access.

You can use your own log stack instead, such as ELK, Datadog or CloudWatch. The services write JSON to stdout, so any collector works. See Logs and Troubleshooting.

The services write JSON to stdout, so you can use another log collector. See Logs and Troubleshooting.

Use your own Prometheus and Grafana containers

Section titled “Use your own Prometheus and Grafana containers”

Mount the configuration into your containers:

prometheus:
image: prom/prometheus:v3.13.0
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
grafana:
image: grafana/grafana:13.1.0
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning:ro
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro

Grafana provisioning also registers a Loki data source at http://loki:3100. Without Loki, the Logs panels have no data; the other panels still work.


GlitchTip groups errors from the API, task scheduler, and dashboard with their stack traces. It accepts Sentry event format. The glitchtip/docker-compose.yaml file runs GlitchTip with its own Postgres database on the same Docker host as OpenTremor.

Terminal window
cd glitchtip
cp .env.example .env # set GLITCHTIP_SECRET_KEY and GLITCHTIP_DB_PASSWORD
docker compose up -d
VariableDefaultDescription
GLITCHTIP_SECRET_KEYRequiredSession signing key. Generate it with openssl rand -hex 32
GLITCHTIP_DB_PASSWORDRequiredPassword for GlitchTip’s Postgres database
GLITCHTIP_DOMAINhttp://localhost:8010Address people use to open the UI
GLITCHTIP_PORT8010UI port, bound to 127.0.0.1. Use an SSH tunnel or reverse proxy to reach it
OPENTREMOR_NETWORKopentremor-prod_defaultDocker network for the OpenTremor stack. GlitchTip joins it with the glitchtip alias
EMAIL_URL—Optional alert email URL, such as smtp+tls://user:[email protected]:587. Without it, GlitchTip sends no email

GlitchTip retains events for 30 days. User registration and organization creation are disabled. It runs without Valkey, using Postgres for its cache and task queue.

  1. Create the first account: docker compose exec glitchtip ./manage.py createsuperuser.
  2. Open the GlitchTip UI, create an organization, then create a project for each service, such as core, task-scheduler, and dashboard. You can use one project for all three.
  3. Copy the project DSN from Settings → Client Keys. Replace the DSN host with glitchtip:8000, which is reachable on the OpenTremor network. For example: http://<key>@glitchtip:8000/<project id>.
  4. Set SENTRY_DSN to this value in the OpenTremor stack’s .env file, then restart the stack. Services tag events with service, so you can filter a shared project by service.

The dashboard forwards browser errors through its server, so GlitchTip does not need a public address. See Logs and Troubleshooting for details about what each service sends.

The 5xx error rate and Logs panels link to GlitchTip. Set the GlitchTip variable at the top of the dashboard to your GlitchTip address and organization slug, such as http://localhost:8010/opentremor. The Logs panel link uses the Request ID value.

Back up the postgres-data volume with the rest of the host to keep GlitchTip’s error history.


The dashboard’s Organization variable supports multiple selections and defaults to All. The metric panels use this variable to filter by org_id. Requests without a resolved organization, such as health checks, public report links, the GitHub webhook, and SSO callbacks before login, appear under _none.


opentremor-monitoring/
├── docker-compose.yaml Prometheus, Grafana, Loki, and Alloy
├── glitchtip/
│ ├── docker-compose.yaml GlitchTip and its Postgres database
│ └── .env.example GlitchTip settings
├── prometheus/
│ └── prometheus.yml Scrape config (target: opentremor-core:8000)
├── loki/
│ └── loki.yaml Single-binary Loki, filesystem storage, 30-day retention
├── alloy/
│ └── config.alloy Reads container stdout, sets labels, and sends logs to Loki
└── grafana/
├── provisioning/
│ ├── datasources/prometheus.yml Registers the Prometheus data source
│ ├── datasources/loki.yml Registers the Loki data source
│ └── dashboards/dashboard.yml Points Grafana to the dashboards folder
└── dashboards/
└── opentremor.json Pre-built dashboard

In Grafana, open Connections → Data sources → Add data source → Prometheus. Set the URL to the address Prometheus can reach, such as http://prometheus:9090.

To use the Logs panels, add a Loki data source the same way, such as http://loki:3100.

In Grafana, open Dashboards → Import → Upload JSON file and upload grafana/dashboards/opentremor.json. Select the Prometheus and Loki data sources you added, then click Import.


SectionPanelPromQL highlight
HTTP TrafficRequest raterate(http_request_duration_seconds_count[2m])
HTTP TrafficLatency p50/p95/p99histogram_quantile(0.95, ...)
HTTP Traffic5xx error rateRatio of status=~"5.." to total
HTTP TrafficRate by endpointGrouped by handler
HTTP TrafficRate by status codeGrouped by status
Analysis PipelineIngest raterate(mcp_ingest_total[2m])
Analysis PipelineResources ingested/srate(mcp_ingested_resources_total[2m])
Analysis PipelineAnalysis submissions/srate(mcp_analysis_total[2m])
Analysis PipelineLLM response time (p50/p95/p99)histogram_quantile(0.95, rate(mcp_llm_analysis_duration_seconds_bucket[5m]))
FindingsBy severity (stacked)rate(mcp_findings_total[5m]) per severity
FindingsBy resource typerate(mcp_findings_total[5m]) per resource_type
TotalsStat panelsCumulative counters since last restart
BackupsTime since last successful backuptime() - max(backup_last_success_timestamp_seconds), red after 48 hours
BackupsLast backup sizemax(backup_last_size_bytes)
BackupsFailed backups (24 h)sum(increase(backup_failures_total[24h]))
LogsErrors and warnings per minutecount_over_time({level=~"warning|error|critical"}[1m]) per service and level
LogsLogs{service=~"$service"} |= "$request_id"
LogsFailed jobserror lines from the job runner

The Backups panels are platform-wide. Backups have no org_id label, and the Organization variable does not filter them. See Backups and Restore.

The Logs panels use the Service and Request ID variables, not the Organization variable. Enter an X-Request-ID value in Request ID to find all lines for that request, including logs for jobs it started. The 5xx error rate panel links to logs for the same time range.


If you use the Prometheus Operator, add a ServiceMonitor. Otherwise, annotate the pod for scraping:

values.yaml
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8000"
prometheus.io/path: /metrics

See Helm (Kubernetes) for the values reference.