Skip to content
DashboardWebsite

Backups and Restore

Just want to do it in the dashboard? See the guide: Set up backups.

This page explains how platform administrators configure, check, and restore encrypted backups of the MongoDB database. Each backup is encrypted with age to public keys you set. The server never holds the matching private keys.

MongoDB is the only stateful service. TLS certificates can be issued again, so they are not backed up.

In the dashboard: Platform Admin → System → Backups → Encryption key. Choose Add a key:

  1. Generate and download creates a key pair in your browser and downloads the identity file (the private key). Or enter the age public key (age1…) for an identity you already have.
  2. Choose the identity file you saved. The browser uses it to decrypt a short message from the server. This proves the identity can decrypt a challenge before a backup depends on it. The file is not uploaded.
  3. Label the key with where you keep the identity file. Confirm with your two-factor code, or your password if your account has no two-factor authentication.

Store the identity file offline (see Keep the keys), then delete the downloaded copy.

The first key takes effect at once. Later changes take effect after 24 hours:

  • A new key is used starting 24 hours after it is added.
  • A removed key stops being used 24 hours after removal is requested.
  • Every platform administrator is emailed. Any of them can cancel a pending change from the same tab.

A backup is encrypted to every key in force when that backup starts. Keep the identity for each key until no retained backup needs it.

Pinned by the operator. Set public keys in the environment, separated by commas:

Terminal window
BACKUP_AGE_RECIPIENTS=age1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3d2hd

The backup.recipients configuration setting does the same. While either source is set, the dashboard shows the keys read-only, and a session cannot change them. To create an identity on your own machine:

Terminal window
docker run --rm -v "$PWD":/out alpine:3.22 sh -c 'apk add -q age && age-keygen -o /out/backup-identity.txt'

The command prints the public key. backup-identity.txt holds the identity.

In the dashboard: Platform Admin → System → Backups → Schedule & storage. You can also update these settings with PATCH /v1/admin/settings. The S3 fields appear only when the target is s3.

SettingDefaultDescription
backup_enabledfalseWhether the scheduler runs backups. Back up now works either way.
backup_interval_hours24Hours between successful backups. Allowed range: 1–168.
backup_retention_days30Backups older than this are deleted, except the newest successful one. Allowed range: 1–3650.
backup_targetlocallocal or s3
backup_s3_endpoint—An S3-compatible endpoint, such as https://s3.fr-par.scw.cloud
backup_s3_region—The bucket’s region, such as fr-par
backup_s3_bucket—The bucket
backup_s3_prefixopentremor-backups/Prepended to each object key
backup_s3_access_key_id—
backup_s3_secret_access_key—Write-only: never returned by the API
backup_egressPublic addresses onlyWhere backup_s3_endpoint may point. Allow private networks for a MinIO instance on your network.

Any S3-compatible store works, including Scaleway Object Storage, OVHcloud, AWS S3, and MinIO. Keep the bucket in the same region as the server if your contracts promise in-region backups.

Targets:

  • s3 uploads the backup after writing it to BACKUP_LOCAL_DIR, then removes the local file.
  • local keeps backups in BACKUP_LOCAL_DIR (default /var/lib/opentremor/backups). A backup on the server’s disk does not survive the loss of that disk. With several API replicas, only the replica that ran the backup holds its file.

Either way, BACKUP_LOCAL_DIR needs room for one encrypted backup.

For an S3 bucket, you can also add a lifecycle rule to expire objects after backup_retention_days + 2 days. This can delete old objects if the platform stops pruning them.

In the dashboard: Platform Admin → System → Backups → Overview. Choose Back up now to start one immediately. The page lists configuration problems, if any, and runs with their size and SHA-256 checksum. A banner appears when the last successful backup is more than twice the configured interval old.

The task scheduler calls POST /v1/admin/backups/tick every 15 minutes. A tick starts a backup when the last success is older than backup_interval_hours. After a failed attempt, the scheduler waits at least an hour before another attempt. A run whose server stops while it is running is marked failed by a later tick.

Prometheus metrics are available on /metrics:

MetricDescription
backup_last_success_timestamp_secondsStart time of the newest successful backup. Take the max across replicas.
backup_last_size_bytesSize of that backup
backup_failures_totalFailed runs, counted per process

Alert when time() - max(backup_last_success_timestamp_seconds) exceeds twice the interval.

A restore needs the backup file, the age identity for a key that encrypted that file, and the three runtime keys. Without the runtime keys, the data comes back but some parts no longer work:

KeyIf it is lost
age identityBackups encrypted to that identity are unreadable
SECRETS_ENCRYPTION_KEYEvery stored credential (GitHub App, SSO, notification channels, S3 key) must be entered again
TWO_FACTOR_SECRET_KEYEvery two-factor enrolment stops working
JWT_SECRETEveryone is signed out. Nothing else.

Rules:

  • Keep each age identity and a copy of the three runtime keys offline, in two places: for example, in a password manager and in a sealed copy kept elsewhere.
  • Never store an identity on the server, and never store runtime keys in the backup bucket.
  • The backup file does not contain the runtime keys. If an identity leaks, the data in backups encrypted to it is exposed, but stored credentials remain encrypted.
  • After removing a key, keep its identity until every backup encrypted to it has been deleted. The newest successful backup is exempt from automatic retention deletion, so it may need to be deleted manually.
  • If an identity leaks, add a new key and remove the leaked one. Both dashboard changes wait 24 hours. To switch immediately, set the new key in BACKUP_AGE_RECIPIENTS and restart the API.

Restore with the stack stopped. These steps use the single-VM layout (docker-compose.prod.yaml).

  1. Get the file. Download it from the bucket, or from Platform Admin → System → Backups for the local target. Check it against the SHA-256 shown for the run. The run also lists the public keys the backup is encrypted to, which tells you which identity can decrypt it:

    Terminal window
    sha256sum opentremor-backup-20261025T020000Z-bk_3f2a….zip.age
  2. Decrypt and unpack it on a machine that holds the identity:

    Terminal window
    docker run --rm -v "$PWD":/w -w /w alpine:3.22 sh -c \
    'apk add -q age unzip && age -d -i backup-identity.txt backup.zip.age > backup.zip && unzip -q backup.zip -d restore'

    restore/manifest.json names the database, date, and platform version.

  3. Put the runtime keys back in .env: SECRETS_ENCRYPTION_KEY, TWO_FACTOR_SECRET_KEY, and JWT_SECRET, using the values from when the backup was taken.

  4. Load the backup into MongoDB with only MongoDB running:

    Terminal window
    docker compose -f docker-compose.prod.yaml stop
    docker compose -f docker-compose.prod.yaml start mongodb
    docker compose -f docker-compose.prod.yaml cp restore mongodb:/restore
    docker compose -f docker-compose.prod.yaml exec mongodb mongorestore --drop --dir /restore

    --drop replaces each collection in the backup. Collections that are not in the backup are left alone.

  5. Start the stack. The API recreates its indexes at boot.

  6. Delete backup.zip, restore/, and the identity copy from that machine.

A backup can be restored onto the release that took it or a later release. The later release applies its migrations at startup. An older release refuses to start on it.

  • Not a snapshot. A standalone MongoDB has no snapshot across collections, so a backup taken during an analysis can hold a job without all of its findings.
  • Not per organization. One file holds every organization. To give an organization its data, use the organization export (GET /v1/orgs/{org_id}/export).
  • Not encrypted to a key the server holds. If every copy of an identity that can decrypt a retained backup is lost, that backup cannot be read.