Skip to content
DashboardWebsite

MongoDB Backend

Every collection is scoped to an org_id, with one exception (allowed_models, platform-wide). See Architecture — Data model for the full picture. The two collections most frequently accessed at request time are:

One document per unique (org, content hash) pair.

{
"_id": "acme-corp:a3f1c2d4e5b6…",
"org_id": "acme-corp",
"hash": "a3f1c2d4e5b6…",
"value": {
"type": "async_function",
"name": "export_invoices",
"action": "add",
"body": "+async def export_invoices(…",
"metadata": {"pack": "python-code-change", "source_path": "billing/api/invoices.py"}
},
"analysis": null,
"last_ingested_at": "2026-07-15T08:59:40.890869+00:00"
}

_id is "{org_id}:{hash}". MongoDB’s native unique constraint enforces per-organization deduplication, not global deduplication. The same content submitted by two organizations is stored twice because value.body can carry tenant-identifying data. analysis starts as null and is updated once an analysis (client- or server-led) is submitted. last_ingested_at is stamped on creation and refreshed on every add_to_changeset call, including a re-ingest of the same hash. analysed_with records what produced analysis: server (one of the org’s LLM connections, with its connection_id and model), client (the caller’s own agent) or scan (the injection pattern scan, no LLM). The Data Retention sweep uses this timestamp to redact value once it’s stale. It retains analysis and hash.

While a unit is claimed, the document also carries claimed_until (the lease end) and claimed_by (the key or user that claimed it; null for the server-side runner). A claim is taken with one atomic find_one_and_update per unit, so two callers never win the same unit.

One document per (org, changeset, revision, resource).

{
"_id": "acme-corp:production:9f2c1e7a…:a3f1c2d4e5b6…",
"org_id": "acme-corp",
"changeset": "production",
"revision_id": "9f2c1e7a…",
"hash": "a3f1c2d4e5b6…",
"created_at": "2026-07-15T08:59:40.890869+00:00"
}

_id is "{org_id}:{changeset}:{revision_id}:{hash}". Writes use $setOnInsert, so re-ingesting the same input into the same revision is idempotent.

One document per (org, changeset), _id "{org_id}:{changeset}": created_at, the current_revision_id, repository (provider/external_id, set once by the first review or ingest that names one), pull_request, and revisions[]. Each revision holds revision_id, created_at, source, the rule_types an ingest named (pack → variant), and outcome once its analysis completes (findings opened, closed and still open; written once). GET /v1/changesets reads this collection; a claim reads a revision’s scope to build its rules. review_team_ids records the teams a review was routed to. A changeset stores no product: its product is its repository’s product, read when asked, so moving a repository moves its history with it.

Collection_id shapeNotes
organizationsorg_idlocked; agent_access (allowed/read_only/forbidden); core’s injection_settings and code_suggestions; quota.monthly_budget_usd — null = unlimited; pending_deletion_at — nullable, set by DELETE /v1/admin/organizations/{org_id}, hard-deleted for real once past org_offboarding.grace_period_days, see Deleting an organization
usersuser_idemail unique-indexed
memberships"org:user"role: owner/admin/member/viewer/auditor, or custom:<slug>
api_keyskey_id (UUID)key_hash unique-indexed; raw key never stored; optional name and ip_allowlist; the pipeline role is for keys only
access_tokenstoken_idA person’s token for an agent: token_hash unique, scope, optional org_id, session_version (a session bump revokes it); expires_at TTL-indexed
oauth_clientsclient_idDynamically registered OAuth clients (client_name, redirect_uris). Platform-wide, no org_id
oauth_codescode hashSingle-use authorization codes; expires_at TTL-indexed
oauth_grantsgrant_idOne connected app per consent: refresh_hash unique, user_id, client_id, scope, optional org_id; expires_at TTL-indexed
installed_packs"org:pack"packs the organization enabled; a pack with no row is off
custom_rules"org:rule_id"one row per rule; packs[] — a rule can target more than one pack; enabled toggles inclusion without deleting; requires_review routes a match straight to needs_review regardless of LLM confidence
teams"org:team_id"Pure organizational grouping — no access-control effect. team_id is a slug of name, immutable after creation
team_members"org:team_id:user_id"Join table — a user can be on any number of teams. Deleting a team cascades: every row for that team_id is removed first (delete_team_members_for_team)
llm_credentials"org:connection_id"One LLM connection per row (the name predates connections). encrypted_key and any CA bundle are Fernet-encrypted and never returned by the API
jobsjob_id (UUID)Platform-owned — org-scoped async work, generic over what the work is. updated_at stamped on every write (creation and every status/progress update) — what POST /v1/admin/jobs/sweep compares against jobs.stuck_timeout_minutes. A queued job also carries kind, payload (references only), attempts, and while running claimed_by/lease_until — the worker’s claim, renewed while it runs. The analysis engine adds its own fields to the same document (changeset, pack, llm_backend/model/rule_type/retry_of backing POST /v1/jobs/{job_id}/retry, current_resource) via JobStatus, which widens the platform’s JobRecord
usage_eventsautoTTL-indexed; type is the meter name. Drives get_monthly_cost and get_monthly_quantity
findings"org:hash:rule_id"persists across re-analysis; status (incl. needs_review) never auto-resets, except to closed when a completed revision no longer reports it (closed_in holds the revision and the status to restore); repositories[], reason_code, and ticket for its Jira issue
integrations"org:provider"github, gitlab or jira. Jira: site URL, flavor, encrypted token, targets per team and product, rules, transitions and sync state. GitHub: github_app holds the org’s own encrypted App credentials; installation_id is set by the installation webhook. GitLab: the encrypted token, webhook secret and optional CA
github_manifest_statesstate_id (UUID)Single-use CSRF token for the GitHub App manifest flow’s redirect round-trip, carrying the LLM settings the integration is created with; state_hash unique-indexed, raw token never stored, same pattern as report_links.token_hash; expires_at TTL-indexed, ~10 minutes
report_linkslink_id (UUID)token_hash unique-indexed; raw token never stored; expires_at TTL-indexed
org_invitesinvite_id (UUID)Same shape as report_links but reusable (not consumed on use) and longer-lived — invite_link_ttl_hours default 168h/7 days vs. report links’ 24h; revoked flag alongside the TTL
sso_connectionsorg_idOne SSO connection per org, OIDC or SAML; the OIDC client secret is encrypted
saml_responses"org:assertion_id"Accepted SAML assertions, so a replay is refused, and the one-time ticket that finishes the sign-in; ticket_hash unique, expires_at TTL-indexed
scim_configsorg_idOne SCIM config per org; only the bearer token’s hash is stored
org_roles"org:role_id"Custom roles and the permissions each grants
user_tokenstoken_hashSingle-use password-reset and email-verification tokens; token_hash unique, expires_at TTL-indexed
pricing_configfixed singleton idThe instance’s price list, see Pricing
notification_channels"org:channel_id"Destination encrypted and never returned; events[], last_delivery
notification_apps"org:provider"An org’s own Slack app (client_id, encrypted client_secret)
notification_oauth_statesstate_idSingle-use state for the guided Slack install; TTL-indexed (notifications.oauth_state_ttl_minutes)
pr_gates"org:changeset"Which commit on which repository a changeset’s check lives on, so triage can re-post the status and the report comment
repositories"org:provider:external_id"The repositories the integrations reach; team_ids, product_id, status (active/removed); review_drafts and code_suggestions override the defaults; scan holds the last default-branch scan
products"org:product_id"Named groups of repositories, with an optional description; their teams review every repository in the group
repository_ownershiporg_idThe ownership rules and the default review team
review_feedbackfeedback_idAppend-only reviewer verdicts, see Review Feedback Loop
rule_proposals"org:proposal_id"Rule changes from feedback or from ADRs, pending/deferred/applied/dismissed
knowledge_documents"org:provider:external_id:path"One per ADR or agent-instructions file a repository’s sync read: its blob_sha, status, outcome, and injection_scored
builtin_rule_syncsfingerprint of the pack rule filesOne row per version of the pack files; the replica that inserts it syncs built-in rules into every org
spansspan_id (UUID)TTL-indexed telemetry; key_id/user_id are independent, never merged — exactly one is non-null per span
audit_sinkssink_idWhere an org’s (or, with no org_id, the platform’s) audit log is pushed: HTTPS or syslog over TLS, a cursor on stored_at, a lease, and failure backoff. See Audit Log Streaming
backupsrun_idPlatform-owned. One row per backup run: status, trigger, target, location, documents per collection. See Backups and Restore
backup_keyskey_idPlatform-owned. age recipients set from the dashboard; a change takes effect at active_from, after a delay
address_listslist_idPlatform-owned. Named IP or host lists, static or refreshed from a URL or file. See Network Access
network_rulesrule_idPlatform-owned. One rule per (service, path_prefix): lists, CIDRs, report or enforce
network_eventsper refusal and minuteRefused requests, grouped by minute with a count; TTL-indexed
network_countersper rule and hourMatches per rule per hour, behind the Firewall view; TTL-indexed
audit_eventsevent_id (UUID)TTL-indexed on its own retention.audit_days — decoupled from spans’ retention.telemetry_days, see Data Retention; org_id nullable for platform-wide actions; actor_user_id/actor_key_id independent, never merged; before/after only the fields that changed, never a secret; actor_token_id when a person acted through an agent; request_id
allowed_models"backend:model"No org_id — platform-wide, superadmin-managed via /admin/models. input_price_per_1m_usd/output_price_per_1m_usd are manual entry (no provider exposes a pricing API); an empty collection means unrestricted (see API — Allowed models)
retention_sweep_statefixed singleton ("retention_sweep_state")Last retention sweep’s result — started_at/duration_ms/counts (a map keyed by sweep step, e.g. findings_purged)/error, same upsert-by-fixed-_id pattern as platform_settings/mongo_prepare_state. Backs GET /v1/admin/retention/status
rate_limits"policy:identity:window"Platform-owned. Brute-force attempt counters — one document per identity per fixed window, count incremented atomically ($inc upsert, so concurrent replicas can’t lose a count and share one budget). bucket groups an identity’s windows so a successful login can clear them all; expires_at TTL-indexed. Nothing here is read after the fact — the collection is a counter, not a record. See rate_limit
job_sweep_statefixed singleton ("job_sweep_state")Platform-owned. Last stuck-job sweep’s result — started_at/duration_ms/jobs_marked_failed/error, same upsert-by-fixed-_id pattern as retention_sweep_state. Backs GET /v1/admin/jobs/sweep-status

Created by MongoBackend.setup(), called at application startup.

CollectionIndexFieldsOptionsCovers
resourcesanalysis_idxanalysis ascsparseclaim_unanalysed
changeset_entriesorg_hash_idxorg_id, hash asc—add_to_changeset existence check
changeset_entriesorg_changeset_revision_idxorg_id asc, changeset asc, revision_id asc—find_by_changeset and a claim, both scoped to one revision
allowed_modelsbackend_idxllm_backend asc—(advisory — lookups are by full _id, this covers a future by-backend query)
api_keyskey_hash_idxkey_hash ascuniqueauth lookup
api_keysorg_idxorg_id asc—list_api_keys
membershipsuser_idx / org_idxuser_id / org_id asc—membership lookups
llm_credentialsorg_idxorg_id asc—list_llm_credentials
jobsorg_idxorg_id asc—job lookups
jobsstatus_updated_idxstatus, updated_at asc—sweep_stuck_jobs’ cross-org scan — deliberately not scoped to a single org_id, unlike every other jobs query
jobsstatus_created_idxstatus asc, created_at asc—the worker’s claim of the oldest queued job
jobsstatus_lease_idxstatus asc, lease_until asc—finding running jobs whose lease lapsed
usage_eventsorg_idx / ttl_idxorg_id / timestamp ascTTL on timestampget_monthly_cost, get_monthly_quantity, auto-expiry
findingsorg_idxorg_id asc—list_findings
findingsorg_teams_idxorg_id asc, assigned_team_ids asc—findings routed to a team
findingsorg_repositories_idxorg_id asc, repositories asc—findings by repository, and so by product
findingsorg_ticket_idxorg_id asc, ticket.key ascsparsethe Jira sync’s linked findings and the ?jira= filter
findingsorg_status_idxorg_id asc, status asc—the Jira sync’s candidates by status, and status filters
teamsorg_idxorg_id asc—list_teams
team_membersorg_team_idx / org_user_idx(org_id, team_id) / (org_id, user_id) asc—list_team_members / list_all_team_memberships
integrationsorg_idx / installation_idx / installation_unique_idx / github_app_id_idxorg_id / installation_id / (provider, installation_id) / github_app.app_id asc— / — / unique, partial on a string installation_id / —org endpoints / webhook lookup by installation / one integration per installation / lookup by App id for a new App’s first installation event
integrationswebhook_id_idxwebhook_id ascsparseGitLab webhook lookup
integrationsprovider_idxprovider asc—cross-org sweeps over one provider, such as the Jira sync
github_manifest_statesstate_hash_idx / ttl_idxstate_hash asc / expires_at ascunique / TTL on expires_at (expireAfterSeconds=0)callback lookup; storage-cleanup backstop only — the router checks expires_at explicitly and deletes the record on first use regardless
report_linkstoken_hash_idx / ttl_idxtoken_hash asc / expires_at ascunique / TTL on expires_at (expireAfterSeconds=0)token lookup; storage-cleanup backstop only — the router checks expires_at explicitly on every read, since the TTL sweep runs roughly once a minute
org_invitestoken_hash_idx / org_idx / ttl_idxtoken_hash asc / org_id asc / expires_at ascunique / — / TTL on expires_at (expireAfterSeconds=0)token lookup; org-scoped listing; same TTL-backstop reasoning as report_links
org_invitesemail_idxemail ascsparseGET /v1/auth/me/invitations for addressed invites
spansts_idx / ttl_idx / org_idxtimestamp desc / timestamp asc / org_idTTL on timestamp (retention.telemetry_days)span queries (newest first), auto-expiry
audit_eventsorg_idx / ts_idx / actor_user_idx / actor_key_idx / ttl_idxorg_id / timestamp desc / actor_user_id / actor_key_id / timestamp ascTTL on timestamp (retention.audit_days — its own independent window, no longer shared with spans)audit queries (newest first, org or actor filtered), auto-expiry
audit_eventsstored_idxstored_at asc, _id asc—the platform audit sinks’ cursor
audit_eventsorg_stored_idxorg_id asc, stored_at asc, _id asc—an org audit sink’s cursor
rate_limitsttl_idx / bucket_idxexpires_at asc / bucket ascTTL on expires_at (expireAfterSeconds=0)auto-expiry of spent windows; clearing every window of one identity after a successful login. Lookups themselves are by full _id, so neither index serves a read path
changesetsorg_created_idxorg_id asc, created_at desc—GET /v1/changesets, newest first
changesetsorg_repository_idxorg_id asc, repository asc—a repository’s changesets in the product view and scope reports
pr_gatesorg_idxorg_id asc—org purge and export
repositoriesorg_name_idxorg_id asc, full_name asc—lookup and listing by full_name
repositoriesorg_teams_idxorg_id asc, team_ids asc—a team’s repositories
repositoriesorg_product_idxorg_id asc, product_id asc—a product’s repositories
productsorg_name_idxorg_id asc, name asc—listing by name
custom_rulesorg_idxorg_id asc—list_custom_rules
custom_rulesorg_packs_idxorg_id asc, packs asc—the rules that apply to one pack
report_templatesorg_idxorg_id asc—listing
report_templatesorg_format_idxorg_id asc, format asc—the default template for a format
rule_categoriesorg_idxorg_id asc—listing
review_feedbackorg_rule_idxorg_id asc, rule_id asc—rule health, per rule
review_feedbackorg_created_idxorg_id asc, created_at desc—recent feedback, newest first
review_feedbackorg_comment_idxorg_id asc, provenance.comment_id ascsparsededuplicating a verdict from the same PR comment
rule_proposalsorg_status_idxorg_id asc, status asc—the To review tab
rule_proposalsorg_rule_idxorg_id asc, rule_id asc—open proposals for one rule
knowledge_documentsorg_repository_idxorg_id asc, provider asc, external_id asc—a repository’s documents in the knowledge sync
usersemail_ci_idxemail ascunique, case-insensitive collationget_user_by_email, case-insensitive
usersidentity_idxidentities.provider asc, identities.subject ascunique, partialsign-in by linked identity, one account per (provider, subject)
sso_connectionsverified_domains_idxverified_domains asc—routing a sign-in to the org that verified its domain
scim_configstoken_hash_idxtoken_hash ascuniqueSCIM bearer-token lookup
notification_channelsorg_idxorg_id asc—listing
audit_sinksorg_idxorg_id asc—an org’s sinks
backupsactive_idxactive ascunique, partialat most one running backup across every replica
backupsstarted_idxstarted_at desc—run history, newest first, and retention
notification_appsorg_idxorg_id asc—org purge and export
notification_oauth_statesstate_hash_idxstate_hash ascuniqueSlack install callback lookup
notification_oauth_statesttl_idxexpires_at ascTTL (expireAfterSeconds=0)auto-expiry
org_rolesorg_idxorg_id asc—listing custom roles
address_listsname_idxname ascuniquerules name lists by name
network_rulesservice_prefix_idxservice asc, path_prefix ascuniqueone rule per service and path prefix
network_eventsts_idxtimestamp desc—the Traffic view, newest first
network_eventsorg_idxorg_id ascsparseorg purge and export
network_eventsref_ts_idxlayer asc, ref_id asc, timestamp desc—the refusals of one rule or key
network_eventsttl_idxtimestamp ascTTLauto-expiry
network_countershour_idxhour asc—hourly traffic totals
network_countersttl_idxexpires_at ascTTL (expireAfterSeconds=0)auto-expiry
user_tokenstoken_hash_idxtoken_hash ascuniquelink lookup
user_tokensuser_purpose_idxuser_id asc, purpose asc—replacing a person’s previous link of the same purpose
user_tokensttl_idxexpires_at ascTTL (expireAfterSeconds=0)auto-expiry
access_tokenstoken_hash_idxtoken_hash ascuniquebearer-token lookup
access_tokensuser_idxuser_id asc—a person’s tokens
access_tokensttl_idxexpires_at ascTTL (expireAfterSeconds=0)auto-expiry
oauth_codesttl_idxexpires_at ascTTL (expireAfterSeconds=0)cleanup backstop; consume_oauth_code checks expires_at itself
saml_responsesticket_hash_idxticket_hash ascunique, partialfinishing a SAML sign-in with its one-time ticket
saml_responsesttl_idxexpires_at ascTTL (expireAfterSeconds=0)auto-expiry at the assertion’s NotOnOrAfter
oauth_grantsrefresh_hash_idxrefresh_hash ascuniquerefresh-token lookup
oauth_grantsuser_idxuser_id asc—a person’s connected apps
oauth_grantsttl_idxexpires_at ascTTL (expireAfterSeconds=0)auto-expiry

The server creates every index at startup (setup(), idempotent). To prepare a database before the first start or create a least-privilege app user, run opentremor-admin init-db, which applies the same declarations.

TTL indexes here are live-updatable. spans/usage_events and audit_events’s ttl_idx expireAfterSeconds isn’t fixed at index-creation time the way a TTL index normally is — PATCH /v1/admin/settings with a changed retention_telemetry_days/retention_audit_days pushes the new value onto the already-running index via a collMod command immediately. See Data Retention.

Upgrading a database created under an older retention default

Section titled “Upgrading a database created under an older retention default”

Index creation reconciles rather than fails. Mongo’s createIndex is only idempotent when the existing index matches exactly: an index of the same name with different options raises IndexOptionsConflict instead of updating it. Because startup declares every index unconditionally, a database whose ttl_idx was created under a different expireAfterSeconds would otherwise make the process crash-loop on upgrade. This can happen because audit_events gained its own retention window after previously sharing spans’.

Startup now collMods a conflicting TTL window into place and carries on, logging the change. opentremor-admin init-db runs the same declarations, so the command and the app can’t drift apart.

Conflicts that aren’t a TTL window — a changed unique flag, or a changed key spec — are reported and startup stops, naming the collection, the index and the dropIndex to run. Resolving those conflicts means dropping and recreating the index. This is destructive and potentially slow on a large collection, so the server leaves it to an operator to decide rather than changing the database on its own at boot.


controllers.services.tenancy.org_offboarding.purge_pending_organizations cascades a plain delete_many({"org_id": org_id}) (or the equivalent single-field filter) across every org-scoped collection above — resources, changeset_entries, memberships, api_keys, installed_packs, custom_rules, report_templates, rule_categories, teams, team_members, llm_credentials, jobs, findings, org_invites — then the organizations document itself, last. No new indexes were needed for this: every one of those collections already carries an org_id field, and this only ever runs as a rare, admin-triggered batch operation (a scheduled org purge), not a hot request path. usage_events/audit_events are deliberately excluded from this cascade — see Platform Admin — Deleting an organization for why.


entries = await entries_col.find({"org_id": org_id, "changeset": changeset}, {"hash": 1}).to_list(None)
rks = [f"{org_id}:{e['hash']}" for e in entries]
return await resources_col.find({"_id": {"$in": rks}}, {"_id": 0}).to_list(None)

claim_unanalysed(org_id, changeset, limit, revision_id, owner=...)

Section titled “claim_unanalysed(org_id, changeset, limit, revision_id, owner=...)”
rks = [f"{org_id}:{e['hash']}" for e in revision_entries] # one revision's members
unclaimed = {"$or": [{"claimed_until": None}, {"claimed_until": {"$lt": now}}]}
for _ in range(limit):
doc = await resources_col.find_one_and_update(
{"_id": {"$in": rks}, "analysis": None, **unclaimed},
{"$set": {"claimed_until": now + lease, "claimed_by": owner}},
)

add_to_changeset(org_id, hash, changeset) — idempotent upsert

Section titled “add_to_changeset(org_id, hash, changeset) — idempotent upsert”

The entry key includes the revision ID. In the implementation, ingestion writes each entry with $setOnInsert, so re-ingesting the same unit into the same revision is idempotent.

await entries_col.update_one(
{"_id": f"{org_id}:{changeset}:{revision_id}:{hash}"},
{"$setOnInsert": {
"org_id": org_id,
"changeset": changeset,
"revision_id": revision_id,
"hash": hash,
"created_at": now,
}},
upsert=True,
)

get_monthly_cost(org_id) — aggregation pipeline

Section titled “get_monthly_cost(org_id) — aggregation pipeline”

Spend is the cost of every usage event this month, whatever its type:

pipeline = [
{"$match": {"org_id": org_id, "timestamp": {"$gte": month_start}}},
{"$group": {"_id": None, "total": {"$sum": "$cost_usd"}}},
]

get_monthly_quantity(org_id, meter) sums quantity (default 1) over the same month’s events of type == meter.


storage:
mongodb:
uri: "mongodb://localhost:27017"
database: "opentremor"

For authenticated deployments, include credentials in the URI or inject the URI through the MONGODB_URI environment variable. Use the environment variable in production. See Configuration.

uri: "mongodb://mcp_app:changeme@mongodb:27017/opentremor?authSource=opentremor"

Terminal window
# First-time setup: every index, plus an optional app user
opentremor-admin init-db \
--admin-uri mongodb://admin:secret@localhost:27017 \
--app-user opentremor_app \
--app-password changeme

See Tools for the full CLI reference.


The backend uses PyMongo’s own async driver, AsyncMongoClient. (Motor, its predecessor, was deprecated by MongoDB in May 2026.)

Which calls are awaited:

cursor = col.find({"org_id": "acme-corp"}) # sync — returns a cursor immediately
docs = await cursor.to_list(length=None) # async — fetches documents
cursor = await col.aggregate(pipeline) # async — aggregate() itself is a coroutine
rows = await cursor.to_list(length=None)
await client.close() # async

find() is not awaited but to_list() is — the same pattern used throughout test_storage.py’s Mongo mocks. aggregate() takes two awaits, which aggregate_to_list() in opentremor_platform.controllers.storage.platform_mongo wraps for both storage halves.