Analysis Engine

Analyzer plugin system

class BaseAnalyzer(ABC):
    name: str                    # URL slug  e.g. "terraform-plan"
    description: str
    is_ready: bool = True        # False -> never registered
    version: str = "1.0.0"
    file_globs: list[str] = []    # auto-routing: glob-matched against a diff's changed file paths
    def sniff(self, raw_input: str) -> float: return 0.0   # auto-routing: content confidence, non-diff input

    def ingest(self, raw_input: str) -> list[dict]: ...
    def get_rules(self, rule_type=None) -> str: ...

    @property
    def manifest(self) -> dict: ...   # {name, description, version, file_globs}

registry = AnalyzerRegistry()
for entry_point in importlib.metadata.entry_points(group="opentremor_core.analyzers"):
    registry.register(entry_point.load()())

Nothing in this repo hardcodes a list of analyzers — create_app() walks the opentremor_core.analyzers entry-point group at startup instead, so installing a package that declares itself under that group is the entire registration step. Each analyzer is a self-contained package in its own repository: its own parser, its own rules, its own manifest. Each unit an analyzer’s ingest() returns is an AnalysisUnit.model_dump()-shaped dict — a model that is deliberately analyzer-agnostic (a Terraform resource block, a diff-changed block, an Ansible task, a Packer builder block, … are all “just” units). The router looks up the analyzer by name from the URL path and delegates parsing and rule retrieval to it, then:

• filters by the caller org’s installed_analyzers overrides (default: every registered analyzer is enabled for every org)
• appends the org’s enabled custom rules targeting that analyzer (if any) after the built-in rules returned by get_rules(), grouped by category, via rules_service.get_effective_rules()

An analyzer that’s diff-shaped (parses a unified git diff, e.g. a Terraform code-change analyzer) is what the GitHub App integration (below) can drive automatically on a PR; a plan-shaped analyzer needs the actual CLI output as input, and nothing in this codebase runs that CLI itself.

Auto-routing — picking the analyzer instead of naming one

POST /{namespace}/ingest/auto and POST /{namespace}/analyze/auto (controllers/services/content_router.py) let a caller skip naming an analyzer at all — the mechanism file_globs always existed for. Two detection strategies, tried in order:

flowchart TD
    A["raw_input"] --> B{"unified diff?\n(has 'diff --git a/... b/...')"}
    B -- yes --> C["split per file"]
    C --> D["glob each path against every\norg-enabled analyzer's file_globs"]
    D -- "1+ matches" --> E["run each matched analyzer\non its own files, merged"]
    D -- "0 matches" --> F{"blackhole installed\n& enabled for org?"}
    B -- no --> G["ask every analyzer's sniff(raw_input)\nfor a confidence score"]
    G -- "highest >= 0.5" --> H["route to that analyzer"]
    G -- "nothing clears 0.5" --> F
    F -- yes --> I["route to blackhole\nDetection(analyzer='blackhole', fallback=True, reason=...)"]
    F -- no --> J["Detection(analyzer=None, reason=...)\n— reported, not dropped"]

A file/blob nothing claims always comes back as a Detection (path, detected_kind — a generic, analyzer-independent display label from libs/content_sniff.detect_label, never used to route — analyzer, reason when analyzer is null, and fallback) rather than silently vanishing. This matters concretely for the GitHub webhook (below): before auto-routing, a PR diff was always handed wholesale to terraform-code-change, whose own parser silently discarded every non-.tf file with no trace at all. Multiple analyzers may match the same file (e.g. two future analyzers both claiming *.tf); auto-routing runs all of them rather than picking one.

Third tier — the blackhole fallback. When neither strategy above finds a match, content_router.py checks whether the optional opentremor-analyzer-blackhole plugin (reference) is installed and enabled for the org; if so, the content is routed there instead of being left unmatched, and the resulting Detection has fallback: true so a caller can render “no analyzer found — falling back” rather than treating it as a confident, real match. blackhole is excluded from the normal file_globs/sniff() candidate pools by name — it never wins a real match, it’s only ever reached through this explicit fallback path — and an org can disable it like any other analyzer if it should fail loudly instead of getting a best-effort result. If blackhole isn’t installed or is disabled, routing falls back to the original skip-with-reason Detection(analyzer=None, reason=...).

Units from every matched analyzer land in the same namespace, tagged with their own metadata.analyzer (already how GET /{namespace}/resource/next resolves per-unit rules — see below — so a namespace spanning multiple analyzers was already a supported shape before auto-routing existed). The one scope limit: a single rule_type override doesn’t mean the same thing across different analyzers in one run, so /analyze/auto always uses each matched analyzer’s own default_rule_type — no variant override.

---

Server-side analysis

POST /{analyzer}/{namespace}/analyze lets the server call the LLM itself instead of a client driving the ingest → loop → submit cycle:

flowchart LR
    A["POST .../analyze\n{llm_backend, model, api_key?, raw_input}"] --> B{"unit count <=\nsync_analysis_max_units?"}
    B -- yes --> C["create jobs doc (running, mode=sync)\nrun_analysis() inline\n-> 200 + report + job_id"]
    B -- no --> D["create jobs doc (queued, mode=async)\nasyncio.create_task\n-> 202 + job_id"]
    D -.background.-> E["run_analysis()\n-> jobs doc: done|failed"]

Both branches leave a job document. An inline run is recorded as mode: "sync" and returns its job_id alongside the report, so every analysis — however small — is one thing you can look up, resume and audit. It didn’t used to be: a run at or under sync_analysis_max_units created no job at all, so a small analysis that failed left a 502 in the caller’s hands, an empty Jobs page, and no way to resume the resources it had already ingested. The error responses carry the job_id too, which is the case that most needs it.

A sync job differs from an async one in exactly one way that matters to the API: it can’t be cancelled. It lives entirely inside the request that started it, so by the time anyone could ask, there is nothing left to stop — POST /jobs/{job_id}/cancel returns 409 and says so, rather than accepting a write that would only contradict the status that request is about to write itself. Resuming one is unaffected: a resumed job is always a background job, whatever the run it resumes was.

analysis_runner.run_analysis() procedurally replicates the client-led loop’s outcome — no agentic tool-calling loop is needed server-side, since the server already knows the fixed sequence: ingest() → get_rules() (+ org’s custom pack) → for each unanalysed unit, call the LLM → analysis_service.record_analysis() → write a usage_events entry.

The drain loop

analyze_unanalysed_units() is the part that costs money, and every bound on it is configurable under the analysis config section:

• Concurrency. Units are drained a batch at a time — analysis.max_concurrent_units (default 4) analyzed in parallel, then the next batch. Two units’ analyses never depend on each other, so serializing them bought nothing and cost wall clock: a 100-unit namespace took ~8 minutes for ~20 seconds of actual provider latency. The batch size is the concurrency bound, and it’s per-run — N concurrent jobs can have N × max_concurrent_units calls in flight. The real ceiling on spend remains the org’s monthly budget.
• Deadline. Every provider call carries analysis.llm_timeout_seconds (default 120), passed down to the SDK. Without it each SDK’s own default applies — Anthropic’s is 10 minutes, long enough for one hung call to stall a whole job.
• Retry. A transient failure (429, 5xx, timeout, dropped connection) is retried up to analysis.llm_max_attempts times (default 3, a total — 1 disables retrying) with exponential backoff and full jitter, so a batch that’s all rate-limited at once doesn’t retry in lockstep and re-trigger the limit. A failure retrying can’t fix — a bad key, a rejected schema, an unparseable response — fails the run on the first attempt instead.
• Response size. Every call is capped at analysis.llm_max_output_tokens (default 4096). A response that hits the cap comes back truncated, not as an error — so each backend checks its provider’s own “why did I stop” signal (stop_reason/finish_reason) and raises LLMTruncatedError naming the cap, instead of letting a half-written object surface as a schema-validation failure pointing at the wrong thing. Truncation is not retried: the same prompt under the same cap overflows identically every time.
• Cancellation. Before each batch the loop re-reads its own job document and stops if the status is no longer running. That’s what makes POST /jobs/{job_id}/cancel cost-effective rather than cosmetic. The check goes through storage, not an in-process handle, because a job’s task lives in whichever replica accepted the request and the cancel may be served by another.

Untrusted content, and what the loop does about it

A unit’s body is whatever was in the input. On the GitHub App path that input is a pull-request diff, so on any repo accepting outside contributions it is written by a stranger — and the analysis that comes back is posted to that same PR as a comment under the App’s own identity, plus a commit status that gates the merge. Text that talks the model into a verdict is therefore text that publishes itself as our verdict. Three guards bound that, and none of them is optional or configurable:

• A nonce-delimited content boundary. Every system prompt gets a boundary paragraph appended naming a per-run random marker, and the unit body is fenced between BEGIN/END UNTRUSTED CONTENT <nonce>. The prompt states that everything inside is data to analyse and never instructions, and that content addressing the reviewer — claiming prior approval, amending the rules, asking for a different verdict — is itself a finding to report (rule_id: AGENT-INJECTION, severity HIGH). The marker carries a nonce rather than being a fixed string because a fixed one is quotable: content could close the block and continue “outside” it. The nonce does not exist yet when the payload is written.
• Evidence verification (libs/analysis_guard.verify_evidence). A Finding.evidence entry is defined as a verbatim snippet of the unit body, so the loop checks that rather than trusting it, comparing on collapsed whitespace so re-indentation isn’t read as fabrication. Entries that don’t appear in the body are dropped; a finding left with none is demoted to confidence: LOW, which routes it to needs_review, and tagged unverified-evidence. It is demoted rather than deleted — deleting would let a fabricated-evidence response erase a real finding.
• Markdown sanitization at the publication boundary (libs/analysis_guard.sanitize_for_external_comment). Every LLM-authored string, plus the names the parser lifted out of the attacker’s own file, is entity-escaped before it is rendered into a PR comment — images, links, @-mentions, raw HTML, backticks, and the | and newlines that would otherwise shred the markdown_report_light table the comment is built from. Applied only on that path: the report API and dashboard responses return the values unescaped, because those callers do their own escaping (the HTML template already autoescapes) and entity-escaping stored values would corrupt them for every other consumer.

These bound the blast radius of an injection that already worked; they don’t detect one. Detection is a separate pass, covered below.

Budget accounting is checked per batch, so a run can overshoot the org’s cap by at most max_concurrent_units - 1 units. That’s inherent: a unit’s cost is only known once it’s been paid for.

LLM provider abstraction (libs/llm/): one LLMClient implementation per provider (Anthropic, OpenAI, Mistral), each forcing structured JSON output via its own native mechanism — a forced tool call (Anthropic), response_format=json_schema (OpenAI), or a JSON-mode response with the schema embedded in the prompt (Mistral, whose JSON mode doesn’t accept a schema directly). Since Mistral’s schema adherence is prompt-following rather than provider-enforced, MistralClient gets one repair retry on a validation failure — the invalid output plus the validation error is fed back to the model before it gives up — which Anthropic/OpenAI don’t need. get_llm_client(backend, api_key) is the factory.

“Required” is not equally required across providers. Only OpenAI’s strict json_schema mode enforces the schema by construction; Anthropic’s forced tool call and Mistral’s JSON mode treat required as a strong hint. In practice that means a field the prompt never mentions is the one a model drops — which is exactly what happened to UnitAnalysis.summary, returned absent on a clean unit whose findings were all present and correct. Two changes cover it, in the two places the cause actually lives:

• The system prompt names every field of the response, summary included, rather than describing only the findings list.
• UnitAnalysis.summary is optional at the schema level, and all three backends validate through one shared parse_unit_analysis(), which fills a missing or blank summary from the findings the model did send ("3 findings — 1 HIGH, 2 LOW."). It states what is already in the payload; it never invents a verdict. Nothing else is repaired there — a malformed finding still fails the unit, because a finding is the analysis and guessing at one would be inventing a security result.

The proportion is the point: one missing line of prose used to raise, fail its unit, and take the whole run down with it — the findings included.

Each backend also translates its own SDK’s retryable failures into one shared TransientLLMError, which is what lets libs/llm/retry.py implement the retry policy once instead of knowing three exception hierarchies. Adding a fourth provider means classifying its errors in its own client; the retry logic doesn’t change.

Credential resolution: the request body’s api_key if present, else a per-org credential stored via POST /orgs/{org_id}/llm-credentials (encrypted at rest with libs/crypto.py, Fernet, key derived from config.llm.credential_encryption_key), else 400.

Agent-injection detection

A separate question from “is this Terraform safe”: is this change trying to manipulate whatever automation reads it next — a code-review bot, an agent running in CI, or this product’s own analysis loop. The guards above bound what an injection that already worked can do to our output; this pass tries to notice one.

Channel segmentation

libs/injection_scan/segment.py splits a change into typed spans before anything matches against it, and that ordering is the whole design rather than an implementation detail.

Injection hides where a reviewer’s eye slides over and a machine reads in full — a comment, a docstring, a YAML description:, an HTML comment in Markdown, a commit message. A pattern corpus matched against whole files treats those identically to executable code and to prose, which makes it unusable: measured against 60 real files from this workspace, matching a 721-rule corpus against whole file contents fired on 13 of 60 (on .env inside a document, on the word re-authenticate, on the phrase take effect on next) in 29.4s; matching the same rules against machine-only channels alone fired on 0 of 60 in 1.1s. Sixty files from one workspace is indicative rather than a benchmark, but the effect size is not subtle.

Two rules carry that:

• Added lines only. A diff’s context and removed lines are the code as it already was, or as it no longer is — neither is what the pull request is asking anyone to accept, and scanning them re-reports a file’s whole history on every push, including flagging the commit that removes an injection.
• Line-level channel classification — comment, docstring, html_comment, config_text, prose, fenced_code, code, plus pr_title / pr_body / commit_message for text belonging to no file. MACHINE_ONLY_CHANNELS names the subset a reviewer skims and a machine ingests whole; a hit there weighs far more than the same hit in code or prose. An instruction-shaped sentence is unremarkable in a design document and alarming in a Terraform comment.

Classification is line-level and best-effort by design, not a parser: the input is a diff, so a multi-line construct is usually only half-present and the file is not guaranteed to parse at this commit. It carries just enough state — open block comment, open Markdown fence, open Python docstring — not to misread the line after an opener. Each Span carries (path, channel, line, text), with line read off the hunk header so a finding points at something you can open.

Tier 0 — structural checks

libs/injection_scan/structural.py. No patterns, no corpus, no model — a path glob or a codepoint class. This is the cheapest tier and, inconveniently for the intuition that the problem needs semantics, the one covering the attack that works best in practice.

GitInject ran live prompt-injection attacks against real GitHub workflows for four commercial coding agents. The most successful vector was adding CLAUDE.md / AGENTS.md / GEMINI.md to the pull request’s branch: actions/checkout puts it in the workspace, and the agent loads it as operator-level instruction before reading any code. Prose injected into a PR body was the vector the models resisted best.

Agent instruction files. Globs CLAUDE.md, AGENTS.md, GEMINI.md, .cursorrules, .cursor/rules/**, .github/copilot-instructions.md, .claude/**, .mcp.json, SKILL.md and similar. Conditioned on author trust, because without that it fires on every legitimate maintainer pull request: a maintainer editing CLAUDE.md is doing their job, a first-time contributor from a fork editing it is rewriting the instructions of every bot that will review their own change. Trusted author → one INFO record that blocks nothing. Untrusted → HIGH, which holds the merge. On the GitHub path “trusted” is derived from author_association plus whether the head repo is a fork.

Invisible and reordering codepoints. Unicode Tags block (U+E0000–U+E007F) and bidirectional controls are CRITICAL — the latter is the Trojan Source attack (CVE-2021-42574), where the code you review is not the code that runs — and zero-width characters are HIGH. Evidence is re-rendered with the offenders spelled out (# app<U+200B>rove), without which a finding would quote a line that looks entirely ordinary.

Base64/hex blobs in comments are deliberately not here: they fire on checksums, lockfiles, certificates and minified assets, and a tier claiming near-zero false positives cannot carry that. They belong in tier 1, where a hit is weighted by channel.

Tier 1 — the vendored pattern corpus

libs/injection_scan/patterns.py, matching a pinned subset of Agent Threat Rules — 244 rules, 1051 patterns, MIT-licensed. The rules are not ours: ATR is maintained by people who do nothing else, and writing our own phrase list would have been slower, worse, and permanently ours to maintain.

The subset is vendored and pinned, not depended on. A security control should not silently change what it flags because a dependency resolved differently. src/tools/refresh_atr_rules.py re-clones upstream at a commit, re-applies the recorded filter, and prints an added/removed/changed summary — so a refresh is a summary to read and a test run, not an 800-file diff. rules/ATR-PIN.json records the commit and the filter; rules/LICENSE.ATR is the licence those rules travel under and must stay there. Never hand-edit the subset: change the filter and re-run the tool, or the next refresh reverts you.

What the filter keeps is the part of ATR a pull request can actually supply — the categories about content that manipulates a reader, with conditions on text fields. What it drops (tool_response, tool-poisoning, excessive-autonomy and the rest) describes agent runtime behaviour: tool-call sequences, MCP exchanges, per-minute rates, none of which a diff has any representation of.

Measured with the shipped code: 98.6% recall against ATR’s own 1,309 documented payloads, 2.3% false positives against its 1,240 negatives, and 0 of 60 real files from this workspace in 0.6s. Read the recall figure as a tripwire for a bad refresh, not as a detection rate — it measures that the corpus still matches what it was written to match, and paraphrase defeats every rule in it.

Three things make it usable:

• Only instruction-bearing channels are matched — machine-only channels plus the PR title and body. Excluding prose and code is the 0/60 result. The cost is real: an “ignore previous instructions” sentence in a README is not flagged, because in a security repository’s documentation that is usually a sentence about the attack rather than the attack.
• One finding per location, with the stable rule_id AGENT-PATTERN-MATCH. The corpus overlaps heavily — one line commonly matches five rules — and findings are keyed on (resource_hash, rule_id), so reporting the matched id would give five findings per line and lose a reviewer’s triage decision whenever upstream adds a sixth overlapping rule. The matched ids are kept in the finding’s tags.
• Bounds, because the input is attacker-controlled and the regexes are someone else’s: a 2000-character cap per span, 20 matches per pattern, and a 5-second wall-clock budget. Exhausting the budget raises an AGENT-SCAN-TRUNCATED finding rather than returning quietly — a scanner that gives up silently is indistinguishable from one that found nothing.

How a detection reaches the pull request

A detection is recorded as an ordinary Finding on a synthetic resource, one per file (or per pull-request / commit-messages for text belonging to no file). That is the whole integration: triage status, the needs_review webhook, retention, the report renderer, the PR comment and the commit-status gate already work on findings, so none of them needed changing.

The synthetic units are written pre-analysed — find_unanalysed selects on an empty analysis field, so a unit written without one would be drained by the LLM loop and charged for — and unmetered, via record_analysis(metered=False). Agent-injection detection is free on every plan: the deterministic tiers make no LLM call, burn no demo credit (the credit is gated on LLM-analysed units), and book no resource_analyzed usage event. Their hash covers the detections themselves, so re-scanning an unchanged pull request re-finds the same resource and whatever a human already decided about it still applies.

Every channel a pull request carries text in is scanned, not just the diff: the PR title and body arrive on the webhook payload, and commit messages come from one extra /pulls/{n}/commits call (best-effort — if it fails, the remaining channels are still scanned). A payload in a commit message appears in no file, so a scanner reading only the diff would never see it.

Configuration and the gate

Three fields on the analysis config section, live-editable via PATCH /admin/settings:

Field	Default	What it does
injection_scan_enabled	true	Runs the deterministic tiers on every change
injection_scan_semantic	false	Tier 2, the LLM scorer — off because it reads the payload and spends the org’s own budget
injection_scan_gate	"review"	What an injection finding does to the commit status

The gate has three settings. off records and reports the finding but never blocks. review forces a HIGH/CRITICAL injection finding to pending with “Possible agent manipulation — needs human review” — distinguishable from the generic needs-review hold, so the reason a PR is blocked is legible from the check itself. fail makes it count as a normal finding of its severity. review is the default because “nobody has looked at this yet” is a more honest signal than “broken”, and it blocks a required check just as effectively.

Under review, the hold lasts exactly as long as the finding is untriaged — open or needs_review. Any triage decision releases it, acknowledged included: “a human read this, it is real, and the PR can merge anyway” is precisely the decision the review gate is waiting for, and requiring suppressed instead would force orgs to hide findings they want kept on the record. Under fail the finding runs through the ordinary severity gate, which keys off open, so the same acknowledgement clears it there too.

On the GitHub path, author_association decides whether the author could already change this repository’s automation by other means, which is what keeps the agent-instruction-file rule from firing on every maintainer pull request.

Clearing a held check

A commit status is a push, not a poll: GitHub holds whatever state was last written to a SHA. So triage is a writer of the check, not just of the finding. PATCH /findings/{hash}/{rule_id} re-posts the pull request’s commit status through controllers/services/pr_gate.py, and a PR held at pending goes green the moment the last finding holding it is triaged — no second push, no re-run, no waiting for another webhook event.

The mechanics:

• The webhook job records where it posted (pr_gates, keyed by org_id + namespace: owner, repo, PR number, head SHA, installation id, and the resource count the description quotes) immediately before posting, so a status that exists on GitHub always has a row that can re-post it.
• pr_gate.compute_gate_state() is the single definition of how findings map to a check state. Both the analysis run and the re-post call it, so the two can never disagree about what “blocked” means.
• One resource can sit in several namespaces — the same CLAUDE.md touched by two open PRs lands on one content-addressed unit — so every gate the resource feeds is refreshed, not just one.
• The re-post is best-effort, the same posture as the review webhook: the triage is already persisted when it runs, so a GitHub outage is logged and never turns a successful PATCH into an error. An org that disconnected the App since the analysis ran is skipped silently.
• Findings that belong to no pull request — the API and dashboard paths — never reach GitHub at all.

The stored resource_count is why a re-post quotes the same “N unit(s) analysed” the original did: triaging changes the state of the check, not the size of the run behind it.

---

Findings lifecycle

flowchart LR
    A["analysis_service.record_analysis()\n(client-led PUT or server-led run_analysis)"] --> R["review_service.compute_initial_status()\nmatched rule.requires_review, or LOW/UNKNOWN confidence?"]
    R --> B["storage.upsert_finding()\nkeyed by org_id + resource_hash + rule_id"]
    B --> C{"first time seen?"}
    C -- yes --> D["status = open or needs_review\nfirst_seen = last_seen = now"]
    D -. needs_review .-> W["review_webhook_service.notify_needs_review()\n(trigger=auto)"]
    C -- no --> E["last_seen, occurrence_count refreshed\nstatus UNCHANGED (initial_status ignored)"]
    F["PATCH /findings/{hash}/{rule_id}"] --> G["status = open|needs_review|acknowledged|suppressed|false_positive"]
    G -- needs_review --> W2["review_webhook_service.notify_needs_review()\n(trigger=manual)"]
    G --> P["pr_gate.refresh_pr_gates_for_resource()\nre-post the commit status for every PR\nthis resource's namespaces gate"]
    G -.persists across re-scans.-> E

report_service.build_report_data() (shared by the report endpoints and the GitHub webhook job) joins a namespace’s resources against findings and, by default, drops suppressed/false_positive findings from the rendered report entirely (?include_suppressed=true to see them); acknowledged findings stay visible with a status badge.

---