Skip to content
DashboardWebsite

Logs and Troubleshooting

This page helps engineers and platform administrators read OpenTremor logs, configure error reporting, and collect diagnostics when an install has a problem. The API and task scheduler write one JSON object per line to stdout. They do not send logs anywhere themselves. Docker or Kubernetes collects stdout, and you choose where it goes: Loki, ELK, Datadog, CloudWatch, or docker logs.


{"ts": "2026-10-25T09:12:03.412+00:00", "level": "error", "logger": "opentremor_platform.controllers.services.jobs.runner",
"msg": "Job 6705… failed", "service": "core", "version": "1.14.0",
"request_id": "b1c9…", "org_id": "org_…", "job_id": "6705…", "exc_info": "Traceback …"}
FieldMeaning
tsWhen the line was written, in UTC
leveldebug, info, warning, error or critical
loggerThe Python module that wrote it
servicecore for the API, task-scheduler for the scheduler
versionThe release of that service
request_idThe request’s X-Request-ID. - outside a request
org_idThe organization the request or job belongs to, when known
job_idThe background job the line comes from, when there is one
exc_infoThe stack trace, on a line that reports an exception

Every API response carries an X-Request-ID header. Search the logs for that value to find every line about the request.

This includes lines from a background job the request started. The job logs under the ID of the request that created it, and the job document stores that ID as request_id. Each task scheduler tick sends its own X-Request-ID to the API, so the scheduler’s line about a tick and the API’s lines about the call share one ID.

A 500 response also carries the ID in its body: {"detail": "Internal server error", "request_id": "b1c9…"}. The dashboard shows it under the error as Reference, with a copy button. Ask the person who hit the error for that reference. Use it to find the request’s log lines.

The API writes one info line per request, for example GET /v1/orgs/{org_id}/teams 200 12ms, with the fields http_method, route, status_code and duration_ms. It records the route as declared, not the URL that was called. /health and /metrics are logged at debug.


OpenTremor Monitoring ships Loki and Alloy next to Prometheus and Grafana. Alloy reads each container’s stdout and sends it to Loki, which keeps logs for 30 days. The Logs row of the OpenTremor dashboard shows:

  • errors and warnings per minute, per service;
  • log lines filtered by the Service and Request ID variables;
  • failed jobs: the job runner’s error lines.

Paste an X-Request-ID into Request ID to follow a request. The 5xx error rate panel links to logs over the same time range.

Loki indexes only three labels: deployment, service and level. request_id, org_id and job_id stay in the log line, so search for them with a line filter:

{service="core"} |= "b1c9…"
{service="core", level="error"} | json | org_id = "org_…"

The API, task scheduler and dashboard can report errors to a Sentry-compatible server: GlitchTip, a self-hosted Sentry, or sentry.io. Reporting is off by default. No image carries a DSN, so an install reports nowhere until you set one.

Set SENTRY_DSN in the stack’s .env or errorReporting.dsn in the Helm values, then restart. All three services read SENTRY_DSN and SENTRY_ENVIRONMENT. Only the API reads SENTRY_TRACES_SAMPLE_RATE.

ServiceWhat it reports
APIUnhandled exceptions, every error log line (failed jobs included), and a sample of requests as traces: request, MongoDB, LLM and other outbound calls
Task schedulerFailed ticks and every other error log line
DashboardServer rendering errors and browser errors

Each event carries the tags service, release (such as [email protected]) and request_id. API events also carry org_id, deployment.mode and, for a job, job_id. Search for an X-Request-ID to find the error a person hit.

The browser sends its errors to the dashboard’s own /monitoring route, and the dashboard server forwards them to the DSN. The reporting server needs no public address, and ad blockers don’t drop the events. The route forwards only envelopes addressed to the configured DSN.

  • No request body, cookie, header or query string. The URL keeps its path only.
  • No local variables from the stack trace. They could hold a password argument.
  • No database query, prompt or model output. Customer code and prompts never leave the instance this way.
  • No breadcrumbs from the dashboard, since console lines and fetched URLs can carry a secret. On the API, outbound call URLs keep their origin only, because a Slack webhook URL is itself a secret.
  • Every event goes through the same redactor as the logs. See What is never logged.

At boot, the API logs one Ready line with the deployment mode, the MongoDB version and whether the job worker runs, then one Packs loaded line. Every line also carries version. Send these first.

A platform admin downloads it from Platform Admin → System → General → Download diagnostics, or with GET /v1/admin/diagnostics. It is a zip of five files:

FileContents
versions.jsonThe platform, core, each pack, Python and MongoDB
checks.jsonEach check, with ok, warning or failed and one line of detail: MongoDB answers, trusted proxies are set, at least one pack is loaded
config.jsonThe effective configuration, with every secret replaced by [redacted]
recent-problems.jsonlThe last 500 log lines at warning and above, from the replica that answered
failed-jobs.jsonThe last 20 failed jobs across every organization: ID, organization, error, request ID

Nothing is sent anywhere. Read the zip, then attach it to your support request. Each download is recorded in the audit log as diagnostics.download.

Run diagnostics when the API won’t start

Section titled “Run diagnostics when the API won’t start”

The same checks run from the command line, without a running server:

Terminal window
docker compose run --rm -v "$PWD":/out opentremor-core python -m opentremor_core.doctor /out

It reads the same configuration as the server, prints a table of checks, and writes the same zip to the directory you name: here /out, mounted from the current directory. A process that never served has no log lines to include. The command exits with status 1 when a check fails. If the configuration itself is invalid, it stops with the validation error instead, and that error is what to send.


VariableDefaultDescription
LOG_FORMATjsonjson for a log collector, text for a person reading a terminal. The development stack uses text
LOG_LEVELinfoThe log level at startup
SENTRY_DSN—Where error reports go. Empty, nothing is reported
SENTRY_ENVIRONMENTproductionThe environment name on each event
SENTRY_TRACES_SAMPLE_RATE0.05The share of API requests traced. The other services send no traces

After startup, the API’s logging.level platform setting sets the log level. Change it live in Platform Admin → System → General, or with PATCH /v1/admin/settings (logging_level). LOG_FORMAT and the SENTRY_* variables are read from the environment before the API reads its configuration file, so configuration errors are logged in the same format and reported too.

The single-VM stack (docker-compose.prod.yaml) rotates each container’s log at 50 MB and keeps five files.


Logs must be safe to send to us when you ask for help. No password, secret, token or key may appear in a log line, a request span or an error report.

  • Request paths. Access lines, error lines and request spans use the route template, such as /invite/{token}, never the path that was called. They never include the query string, which can carry an OAuth code.
  • Ingress. HAProxy logs the path without its query string and masks the token after /invite/.
  • Outbound calls. The HTTP client logs only warnings and errors. Its info lines would carry URLs, and a Slack webhook URL is itself a secret.
  • Every line is redacted on its way out. A value under a key that looks like a credential (password, secret, token, api_key, authorization, cookie…) becomes [redacted]. So do secrets with a known shape, wherever they appear: Bearer and Basic credentials, a password in a URL, PEM private keys, GitHub, GitLab and Slack tokens, and OpenTremor access tokens.

A secret in a field with an unusual name or in an unknown format can still get past the redactor. If you find one in a log, report it as a security issue.