Data Retention

OpenTremor’s most sensitive data isn’t the audit trail or the request telemetry — it’s the raw infrastructure-as-code content it analyzes (resources.value: Terraform source, diffs, whatever was ingested) and the security findings derived from it. Both used to have no retention policy at all: kept forever, with no delete path, while the lower-stakes telemetry/usage data was capped at a fixed 30 days. This page documents the fix — a retention config section, admin-editable via PATCH /admin/settings, with a lifecycle that matches each data category’s actual sensitivity and audit value instead of the reverse.

The policy, and why each field has the value it does

Data	Field	Default	Why
Telemetry spans + usage events	telemetry_days	30 days	Pure observability — short-lived by design, unchanged from before this feature.
Audit log (privileged actions)	audit_days	730 days	Decoupled from telemetry on purpose — compliance/audit evidence (SOC2, ISO) typically needs to outlive perf data by a lot. Previously silently shared telemetry.retention_days, capped at 365 days max; that cap is gone.
Resolved low/medium-severity findings	finding_low_medium_days	180 days	Low ongoing audit value once resolved — kept just long enough to be useful, then purged to shrink the collection and reduce exposure.
Resolved high/critical-severity findings	finding_high_critical_days	730 days	Kept much longer — “we found X and fixed it” is exactly the evidence an audit typically wants, so this trades storage for audit value on purpose.
Raw analyzed resource content	resource_content_days	90 days	The highest-exposure data (may contain secrets, account IDs, internal architecture) gets the shortest life of anything here — the raw value payload is redacted, along with every embedded Finding.evidence snippet on resource.analysis (verbatim code copied straight out of the same source), since evidence is the same exposure by another name. The finding’s other fields (severity, title, rule_id, …) and the analysis structure itself survive.

Open and needs-review findings are never purged, regardless of age or severity. This isn’t a config option — controllers/services/product_retention.py’s purge step only ever calls it with terminal statuses (acknowledged/suppressed/false_positive), so an active or un-triaged finding can’t silently disappear on a timer no matter how the other fields are set.

Two more fields control whether/how often this runs automatically rather than for how long anything is kept:

Field	Default	Bounds
sweep_enabled	true	—
sweep_interval_hours	24	1–168

Two enforcement mechanisms, two different “when it takes effect” answers

telemetry_days/audit_days are TTL-index-backed (MongoDB only — ignored on the in-memory backend, same as every other Mongo-specific setting). Changing either via PATCH /admin/settings pushes the new value live onto the running TTL index with a collMod call, immediately — not just re-read on the next request, and not requiring a restart or re-prepare. This is a real fix, not a documentation nuance: before this feature, retention_days only affected a newly created index, so changing it after boot silently did nothing to an already-running deployment.

Everything else is sweep-based, because a Mongo TTL index can only expire “N seconds after one fixed date field” — it can’t express “delete this if status is X and severity is Y and it’s old enough.” Findings and resources need an actual application-level sweep (POST /admin/retention/run) that filters on multiple fields. This is why resource_content_days and the two finding_*_days fields only take effect the next time a sweep actually runs, not instantly on PATCH.

Running a sweep

Core itself has no scheduler — see Task Scheduler for the standalone process that calls POST /admin/retention/run on a heartbeat. That endpoint is self-gating: called again before sweep_interval_hours has elapsed, it’s a no-op that returns the previous result (skipped: true) instead of redoing work, which is what makes it safe for the scheduler to call frequently without knowing the configured interval itself. The dashboard’s Platform Admin → Settings → Retention sweep → Run sweep now button (or POST /admin/retention/run?force=true) always bypasses the gate for an immediate run. See Platform Admin — Running a retention sweep manually for the full endpoint reference, including GET /admin/retention/status.

Changing the policy

curl -X PATCH http://localhost:8000/admin/settings \
  -H "X-API-Key: <admin-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "retention_audit_days": 1825,
    "retention_finding_high_critical_days": 1825,
    "retention_resource_content_days": 30
  }'

Set a field to null to clear the override and fall back to the hardcoded default — same convention as every other /admin/settings field, see Configuration Reference — Platform settings.

Self-hosted vs. cloud

Nothing about this policy is different between deployment modes — a self-hosted instance is just as capable of tightening or loosening any of these as the operator’s own cloud deployment. There is no cloud-only retention feature.

What this doesn’t cover (yet)

One gap is known and deliberately out of scope for this feature, not silently dropped:

• No purge for findings whose resource has fallen out of the current namespace state. A resource that’s been re-analyzed and no longer exists in any namespace doesn’t have its findings automatically staled out — only the severity/status/age-based purge above applies. Would need a “current namespace membership” concept this codebase doesn’t have yet.

Org-offboarding was the other gap listed here — closed: see Platform Admin — Deleting an organization for the cascade delete this data-retention work originally flagged as missing.