Logs and Troubleshooting
This page helps engineers and platform administrators read OpenTremor logs, configure error reporting, and collect diagnostics when an install has a problem. The API and task scheduler write one JSON object per line to stdout. They do not send logs anywhere themselves. Docker or Kubernetes collects stdout, and you choose where it goes: Loki, ELK, Datadog, CloudWatch, or docker logs.
What a log line contains
Section titled “What a log line contains”{"ts": "2026-10-25T09:12:03.412+00:00", "level": "error", "logger": "opentremor_platform.controllers.services.jobs.runner", "msg": "Job 6705… failed", "service": "core", "version": "1.14.0", "request_id": "b1c9…", "org_id": "org_…", "job_id": "6705…", "exc_info": "Traceback …"}| Field | Meaning |
|---|---|
ts | When the line was written, in UTC |
level | debug, info, warning, error or critical |
logger | The Python module that wrote it |
service | core for the API, task-scheduler for the scheduler |
version | The release of that service |
request_id | The request’s X-Request-ID. - outside a request |
org_id | The organization the request or job belongs to, when known |
job_id | The background job the line comes from, when there is one |
exc_info | The stack trace, on a line that reports an exception |
Follow a request
Section titled “Follow a request”Every API response carries an X-Request-ID header. Search the logs for that value to find every line about the request.
This includes lines from a background job the request started. The job logs under the ID of the request that created it, and the job document stores that ID as request_id. Each task scheduler tick sends its own X-Request-ID to the API, so the scheduler’s line about a tick and the API’s lines about the call share one ID.
A 500 response also carries the ID in its body: {"detail": "Internal server error", "request_id": "b1c9…"}. The dashboard shows it under the error as Reference, with a copy button. Ask the person who hit the error for that reference. Use it to find the request’s log lines.
Find request access lines
Section titled “Find request access lines”The API writes one info line per request, for example GET /v1/orgs/{org_id}/teams 200 12ms, with the fields http_method, route, status_code and duration_ms. It records the route as declared, not the URL that was called. /health and /metrics are logged at debug.
Search the logs in Grafana
Section titled “Search the logs in Grafana”OpenTremor Monitoring ships Loki and Alloy next to Prometheus and Grafana. Alloy reads each container’s stdout and sends it to Loki, which keeps logs for 30 days. The Logs row of the OpenTremor dashboard shows:
- errors and warnings per minute, per service;
- log lines filtered by the Service and Request ID variables;
- failed jobs: the job runner’s
errorlines.
Paste an X-Request-ID into Request ID to follow a request. The 5xx error rate panel links to logs over the same time range.
Loki indexes only three labels: deployment, service and level. request_id, org_id and job_id stay in the log line, so search for them with a line filter:
{service="core"} |= "b1c9…"{service="core", level="error"} | json | org_id = "org_…"Error reporting
Section titled “Error reporting”The API, task scheduler and dashboard can report errors to a Sentry-compatible server: GlitchTip, a self-hosted Sentry, or sentry.io. Reporting is off by default. No image carries a DSN, so an install reports nowhere until you set one.
Set SENTRY_DSN in the stack’s .env or errorReporting.dsn in the Helm values, then restart. All three services read SENTRY_DSN and SENTRY_ENVIRONMENT. Only the API reads SENTRY_TRACES_SAMPLE_RATE.
| Service | What it reports |
|---|---|
| API | Unhandled exceptions, every error log line (failed jobs included), and a sample of requests as traces: request, MongoDB, LLM and other outbound calls |
| Task scheduler | Failed ticks and every other error log line |
| Dashboard | Server rendering errors and browser errors |
Each event carries the tags service, release (such as [email protected]) and request_id. API events also carry org_id, deployment.mode and, for a job, job_id. Search for an X-Request-ID to find the error a person hit.
The browser sends its errors to the dashboard’s own /monitoring route, and the dashboard server forwards them to the DSN. The reporting server needs no public address, and ad blockers don’t drop the events. The route forwards only envelopes addressed to the configured DSN.
What an error report never contains
Section titled “What an error report never contains”- No request body, cookie, header or query string. The URL keeps its path only.
- No local variables from the stack trace. They could hold a password argument.
- No database query, prompt or model output. Customer code and prompts never leave the instance this way.
- No breadcrumbs from the dashboard, since console lines and fetched URLs can carry a secret. On the API, outbound call URLs keep their origin only, because a Slack webhook URL is itself a secret.
- Every event goes through the same redactor as the logs. See What is never logged.
When something goes wrong
Section titled “When something goes wrong”Send the startup lines
Section titled “Send the startup lines”At boot, the API logs one Ready line with the deployment mode, the MongoDB version and whether the job worker runs, then one Packs loaded line. Every line also carries version. Send these first.
The support bundle
Section titled “The support bundle”A platform admin downloads it from Platform Admin → System → General → Download diagnostics, or with GET /v1/admin/diagnostics. It is a zip of five files:
| File | Contents |
|---|---|
versions.json | The platform, core, each pack, Python and MongoDB |
checks.json | Each check, with ok, warning or failed and one line of detail: MongoDB answers, trusted proxies are set, at least one pack is loaded |
config.json | The effective configuration, with every secret replaced by [redacted] |
recent-problems.jsonl | The last 500 log lines at warning and above, from the replica that answered |
failed-jobs.json | The last 20 failed jobs across every organization: ID, organization, error, request ID |
Nothing is sent anywhere. Read the zip, then attach it to your support request. Each download is recorded in the audit log as diagnostics.download.
Run diagnostics when the API won’t start
Section titled “Run diagnostics when the API won’t start”The same checks run from the command line, without a running server:
docker compose run --rm -v "$PWD":/out opentremor-core python -m opentremor_core.doctor /outIt reads the same configuration as the server, prints a table of checks, and writes the same zip to the directory you name: here /out, mounted from the current directory. A process that never served has no log lines to include. The command exits with status 1 when a check fails. If the configuration itself is invalid, it stops with the validation error instead, and that error is what to send.
Configure logging and error reporting
Section titled “Configure logging and error reporting”| Variable | Default | Description |
|---|---|---|
LOG_FORMAT | json | json for a log collector, text for a person reading a terminal. The development stack uses text |
LOG_LEVEL | info | The log level at startup |
SENTRY_DSN | — | Where error reports go. Empty, nothing is reported |
SENTRY_ENVIRONMENT | production | The environment name on each event |
SENTRY_TRACES_SAMPLE_RATE | 0.05 | The share of API requests traced. The other services send no traces |
After startup, the API’s logging.level platform setting sets the log level. Change it live in Platform Admin → System → General, or with PATCH /v1/admin/settings (logging_level). LOG_FORMAT and the SENTRY_* variables are read from the environment before the API reads its configuration file, so configuration errors are logged in the same format and reported too.
The single-VM stack (docker-compose.prod.yaml) rotates each container’s log at 50 MB and keeps five files.
What is never logged
Section titled “What is never logged”Logs must be safe to send to us when you ask for help. No password, secret, token or key may appear in a log line, a request span or an error report.
- Request paths. Access lines, error lines and request spans use the route template, such as
/invite/{token}, never the path that was called. They never include the query string, which can carry an OAuthcode. - Ingress. HAProxy logs the path without its query string and masks the token after
/invite/. - Outbound calls. The HTTP client logs only warnings and errors. Its
infolines would carry URLs, and a Slack webhook URL is itself a secret. - Every line is redacted on its way out. A value under a key that looks like a credential (
password,secret,token,api_key,authorization,cookie…) becomes[redacted]. So do secrets with a known shape, wherever they appear:BearerandBasiccredentials, a password in a URL, PEM private keys, GitHub, GitLab and Slack tokens, and OpenTremor access tokens.
A secret in a field with an unusual name or in an unknown format can still get past the redactor. If you find one in a log, report it as a security issue.