Backups and Restore
Just want to do it in the dashboard? See the guide: Set up backups.
This page explains how platform administrators configure, check, and restore encrypted backups of the MongoDB database. Each backup is encrypted with age to public keys you set. The server never holds the matching private keys.
MongoDB is the only stateful service. TLS certificates can be issued again, so they are not backed up.
Set up backups
Section titled “Set up backups”1. Set an encryption key
Section titled “1. Set an encryption key”In the dashboard: Platform Admin → System → Backups → Encryption key. Choose Add a key:
- Generate and download creates a key pair in your browser and downloads the identity file
(the private key). Or enter the age public key (
age1…) for an identity you already have. - Choose the identity file you saved. The browser uses it to decrypt a short message from the server. This proves the identity can decrypt a challenge before a backup depends on it. The file is not uploaded.
- Label the key with where you keep the identity file. Confirm with your two-factor code, or your password if your account has no two-factor authentication.
Store the identity file offline (see Keep the keys), then delete the downloaded copy.
The first key takes effect at once. Later changes take effect after 24 hours:
- A new key is used starting 24 hours after it is added.
- A removed key stops being used 24 hours after removal is requested.
- Every platform administrator is emailed. Any of them can cancel a pending change from the same tab.
A backup is encrypted to every key in force when that backup starts. Keep the identity for each key until no retained backup needs it.
Pinned by the operator. Set public keys in the environment, separated by commas:
BACKUP_AGE_RECIPIENTS=age1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3d2hdThe backup.recipients configuration setting does the same. While either source is set, the
dashboard shows the keys read-only, and a session cannot change them. To create an identity on your
own machine:
docker run --rm -v "$PWD":/out alpine:3.22 sh -c 'apk add -q age && age-keygen -o /out/backup-identity.txt'The command prints the public key. backup-identity.txt holds the identity.
2. Choose a target and schedule
Section titled “2. Choose a target and schedule”In the dashboard: Platform Admin → System → Backups → Schedule & storage. You can also update
these settings with PATCH /v1/admin/settings. The S3 fields appear only when the target is s3.
| Setting | Default | Description |
|---|---|---|
backup_enabled | false | Whether the scheduler runs backups. Back up now works either way. |
backup_interval_hours | 24 | Hours between successful backups. Allowed range: 1–168. |
backup_retention_days | 30 | Backups older than this are deleted, except the newest successful one. Allowed range: 1–3650. |
backup_target | local | local or s3 |
backup_s3_endpoint | — | An S3-compatible endpoint, such as https://s3.fr-par.scw.cloud |
backup_s3_region | — | The bucket’s region, such as fr-par |
backup_s3_bucket | — | The bucket |
backup_s3_prefix | opentremor-backups/ | Prepended to each object key |
backup_s3_access_key_id | — | |
backup_s3_secret_access_key | — | Write-only: never returned by the API |
backup_egress | Public addresses only | Where backup_s3_endpoint may point. Allow private networks for a MinIO instance on your network. |
Any S3-compatible store works, including Scaleway Object Storage, OVHcloud, AWS S3, and MinIO. Keep the bucket in the same region as the server if your contracts promise in-region backups.
Targets:
s3uploads the backup after writing it toBACKUP_LOCAL_DIR, then removes the local file.localkeeps backups inBACKUP_LOCAL_DIR(default/var/lib/opentremor/backups). A backup on the server’s disk does not survive the loss of that disk. With several API replicas, only the replica that ran the backup holds its file.
Either way, BACKUP_LOCAL_DIR needs room for one encrypted backup.
For an S3 bucket, you can also add a lifecycle rule to expire objects after
backup_retention_days + 2 days. This can delete old objects if the platform stops pruning them.
3. Check backups
Section titled “3. Check backups”In the dashboard: Platform Admin → System → Backups → Overview. Choose Back up now to start one immediately. The page lists configuration problems, if any, and runs with their size and SHA-256 checksum. A banner appears when the last successful backup is more than twice the configured interval old.
The task scheduler calls POST /v1/admin/backups/tick every 15 minutes. A tick starts a backup when
the last success is older than backup_interval_hours. After a failed attempt, the scheduler waits
at least an hour before another attempt. A run whose server stops while it is running is marked
failed by a later tick.
Prometheus metrics are available on /metrics:
| Metric | Description |
|---|---|
backup_last_success_timestamp_seconds | Start time of the newest successful backup. Take the max across replicas. |
backup_last_size_bytes | Size of that backup |
backup_failures_total | Failed runs, counted per process |
Alert when time() - max(backup_last_success_timestamp_seconds) exceeds twice the interval.
Keep the keys
Section titled “Keep the keys”A restore needs the backup file, the age identity for a key that encrypted that file, and the three runtime keys. Without the runtime keys, the data comes back but some parts no longer work:
| Key | If it is lost |
|---|---|
| age identity | Backups encrypted to that identity are unreadable |
SECRETS_ENCRYPTION_KEY | Every stored credential (GitHub App, SSO, notification channels, S3 key) must be entered again |
TWO_FACTOR_SECRET_KEY | Every two-factor enrolment stops working |
JWT_SECRET | Everyone is signed out. Nothing else. |
Rules:
- Keep each age identity and a copy of the three runtime keys offline, in two places: for example, in a password manager and in a sealed copy kept elsewhere.
- Never store an identity on the server, and never store runtime keys in the backup bucket.
- The backup file does not contain the runtime keys. If an identity leaks, the data in backups encrypted to it is exposed, but stored credentials remain encrypted.
- After removing a key, keep its identity until every backup encrypted to it has been deleted. The newest successful backup is exempt from automatic retention deletion, so it may need to be deleted manually.
- If an identity leaks, add a new key and remove the leaked one. Both dashboard changes wait 24
hours. To switch immediately, set the new key in
BACKUP_AGE_RECIPIENTSand restart the API.
Restore
Section titled “Restore”Restore with the stack stopped. These steps use the single-VM layout (docker-compose.prod.yaml).
-
Get the file. Download it from the bucket, or from Platform Admin → System → Backups for the
localtarget. Check it against the SHA-256 shown for the run. The run also lists the public keys the backup is encrypted to, which tells you which identity can decrypt it:Terminal window sha256sum opentremor-backup-20261025T020000Z-bk_3f2a….zip.age -
Decrypt and unpack it on a machine that holds the identity:
Terminal window docker run --rm -v "$PWD":/w -w /w alpine:3.22 sh -c \'apk add -q age unzip && age -d -i backup-identity.txt backup.zip.age > backup.zip && unzip -q backup.zip -d restore'restore/manifest.jsonnames the database, date, and platform version. -
Put the runtime keys back in
.env:SECRETS_ENCRYPTION_KEY,TWO_FACTOR_SECRET_KEY, andJWT_SECRET, using the values from when the backup was taken. -
Load the backup into MongoDB with only MongoDB running:
Terminal window docker compose -f docker-compose.prod.yaml stopdocker compose -f docker-compose.prod.yaml start mongodbdocker compose -f docker-compose.prod.yaml cp restore mongodb:/restoredocker compose -f docker-compose.prod.yaml exec mongodb mongorestore --drop --dir /restore--dropreplaces each collection in the backup. Collections that are not in the backup are left alone. -
Start the stack. The API recreates its indexes at boot.
-
Delete
backup.zip,restore/, and the identity copy from that machine.
A backup can be restored onto the release that took it or a later release. The later release applies its migrations at startup. An older release refuses to start on it.
What a backup is not
Section titled “What a backup is not”- Not a snapshot. A standalone MongoDB has no snapshot across collections, so a backup taken during an analysis can hold a job without all of its findings.
- Not per organization. One file holds every organization. To give an organization its data,
use the organization export (
GET /v1/orgs/{org_id}/export). - Not encrypted to a key the server holds. If every copy of an identity that can decrypt a retained backup is lost, that backup cannot be read.