Analysis Engine

Analyzer plugin system

class BaseAnalyzer(ABC):
    name: str                    # URL slug  e.g. "terraform-plan"
    description: str
    is_ready: bool = True        # False -> never registered
    version: str = "1.0.0"
    file_globs: list[str] = []    # auto-routing: glob-matched against a diff's changed file paths
    def sniff(self, raw_input: str) -> float: return 0.0   # auto-routing: content confidence, non-diff input

    def ingest(self, raw_input: str) -> list[dict]: ...
    def get_rules(self, rule_type=None) -> str: ...

    @property
    def manifest(self) -> dict: ...   # {name, description, version, file_globs}

registry = AnalyzerRegistry()
for entry_point in importlib.metadata.entry_points(group="opentremor_core.analyzers"):
    registry.register(entry_point.load()())

Nothing in this repo hardcodes a list of analyzers — create_app() walks the opentremor_core.analyzers entry-point group at startup instead, so installing a package that declares itself under that group is the entire registration step. Each analyzer is a self-contained package in its own repository: its own parser, its own rules, its own manifest. Each unit an analyzer’s ingest() returns is an AnalysisUnit.model_dump()-shaped dict — a model that is deliberately analyzer-agnostic (a Terraform resource block, a diff-changed block, an Ansible task, a Packer builder block, … are all “just” units). The router looks up the analyzer by name from the URL path and delegates parsing and rule retrieval to it, then:

filters by the caller org’s installed_analyzers overrides (default: every registered analyzer is enabled for every org)
appends the org’s enabled custom rules targeting that analyzer (if any) after the built-in rules returned by get_rules(), grouped by category, via rules_service.get_effective_rules()

An analyzer that’s diff-shaped (parses a unified git diff, e.g. a Terraform code-change analyzer) is what the GitHub App integration (below) can drive automatically on a PR; a plan-shaped analyzer needs the actual CLI output as input, and nothing in this codebase runs that CLI itself.

Auto-routing — picking the analyzer instead of naming one

POST /{namespace}/ingest/auto and POST /{namespace}/analyze/auto (controllers/services/content_router.py) let a caller skip naming an analyzer at all — the mechanism file_globs always existed for. Two detection strategies, tried in order:

flowchart TD
    A["raw_input"] --> B{"unified diff?\n(has 'diff --git a/... b/...')"}
    B -- yes --> C["split per file"]
    C --> D["glob each path against every\norg-enabled analyzer's file_globs"]
    D -- "1+ matches" --> E["run each matched analyzer\non its own files, merged"]
    D -- "0 matches" --> F{"blackhole installed\n& enabled for org?"}
    B -- no --> G["ask every analyzer's sniff(raw_input)\nfor a confidence score"]
    G -- "highest >= 0.5" --> H["route to that analyzer"]
    G -- "nothing clears 0.5" --> F
    F -- yes --> I["route to blackhole\nDetection(analyzer='blackhole', fallback=True, reason=...)"]
    F -- no --> J["Detection(analyzer=None, reason=...)\n— reported, not dropped"]

A file/blob nothing claims always comes back as a Detection (path, detected_kind — a generic, analyzer-independent display label from libs/content_sniff.detect_label, never used to route — analyzer, reason when analyzer is null, and fallback) rather than silently vanishing. This matters concretely for the GitHub webhook (below): before auto-routing, a PR diff was always handed wholesale to terraform-code-change, whose own parser silently discarded every non-.tf file with no trace at all. Multiple analyzers may match the same file (e.g. two future analyzers both claiming *.tf); auto-routing runs all of them rather than picking one.

Third tier — the blackhole fallback. When neither strategy above finds a match, content_router.py checks whether the optional opentremor-analyzer-blackhole plugin (reference) is installed and enabled for the org; if so, the content is routed there instead of being left unmatched, and the resulting Detection has fallback: true so a caller can render “no analyzer found — falling back” rather than treating it as a confident, real match. blackhole is excluded from the normal file_globs/sniff() candidate pools by name — it never wins a real match, it’s only ever reached through this explicit fallback path — and an org can disable it like any other analyzer if it should fail loudly instead of getting a best-effort result. If blackhole isn’t installed or is disabled, routing falls back to the original skip-with-reason Detection(analyzer=None, reason=...).

Units from every matched analyzer land in the same namespace, tagged with their own metadata.analyzer (already how GET /{namespace}/resource/next resolves per-unit rules — see below — so a namespace spanning multiple analyzers was already a supported shape before auto-routing existed). The one scope limit: a single rule_type override doesn’t mean the same thing across different analyzers in one run, so /analyze/auto always uses each matched analyzer’s own default_rule_type — no variant override.

Server-side analysis

POST /{analyzer}/{namespace}/analyze lets the server call the LLM itself instead of a client driving the ingest → loop → submit cycle:

flowchart LR
    A["POST .../analyze\n{llm_backend, model, api_key?, raw_input}"] --> B{"unit count <=\nsync_analysis_max_units?"}
    B -- yes --> C["run_analysis() inline\n-> 200 + report"]
    B -- no --> D["create jobs doc (queued)\nasyncio.create_task\n-> 202 + job_id"]
    D -.background.-> E["run_analysis()\n-> jobs doc: done|failed"]

analysis_runner.run_analysis() procedurally replicates the client-led loop’s outcome — no agentic tool-calling loop is needed server-side, since the server already knows the fixed sequence: ingest() → get_rules() (+ org’s custom pack) → for each unanalysed unit, call the LLM → analysis_service.record_analysis() → write a usage_events entry.

The drain loop

analyze_unanalysed_units() is the part that costs money, and every bound on it is configurable under the analysis config section:

Concurrency. Units are drained a batch at a time — analysis.max_concurrent_units (default 4) analyzed in parallel, then the next batch. Two units’ analyses never depend on each other, so serializing them bought nothing and cost wall clock: a 100-unit namespace took ~8 minutes for ~20 seconds of actual provider latency. The batch size is the concurrency bound, and it’s per-run — N concurrent jobs can have N × max_concurrent_units calls in flight. The real ceiling on spend remains the org’s monthly budget.
Deadline. Every provider call carries analysis.llm_timeout_seconds (default 120), passed down to the SDK. Without it each SDK’s own default applies — Anthropic’s is 10 minutes, long enough for one hung call to stall a whole job.
Retry. A transient failure (429, 5xx, timeout, dropped connection) is retried up to analysis.llm_max_attempts times (default 3, a total — 1 disables retrying) with exponential backoff and full jitter, so a batch that’s all rate-limited at once doesn’t retry in lockstep and re-trigger the limit. A failure retrying can’t fix — a bad key, a rejected schema, an unparseable response — fails the run on the first attempt instead.
Cancellation. Before each batch the loop re-reads its own job document and stops if the status is no longer running. That’s what makes POST /jobs/{job_id}/cancel cost-effective rather than cosmetic. The check goes through storage, not an in-process handle, because a job’s task lives in whichever replica accepted the request and the cancel may be served by another.

Budget accounting is checked per batch, so a run can overshoot the org’s cap by at most max_concurrent_units - 1 units. That’s inherent: a unit’s cost is only known once it’s been paid for.

LLM provider abstraction (libs/llm/): one LLMClient implementation per provider (Anthropic, OpenAI, Mistral), each forcing structured JSON output via its own native mechanism — a forced tool call (Anthropic), response_format=json_schema (OpenAI), or a JSON-mode response with the schema embedded in the prompt (Mistral, whose JSON mode doesn’t accept a schema directly). Since Mistral’s schema adherence is prompt-following rather than provider-enforced, MistralClient gets one repair retry on a validation failure — the invalid output plus the validation error is fed back to the model before it gives up — which Anthropic/OpenAI don’t need. get_llm_client(backend, api_key) is the factory.

Each backend also translates its own SDK’s retryable failures into one shared TransientLLMError, which is what lets libs/llm/retry.py implement the retry policy once instead of knowing three exception hierarchies. Adding a fourth provider means classifying its errors in its own client; the retry logic doesn’t change.

Credential resolution: the request body’s api_key if present, else a per-org credential stored via POST /orgs/{org_id}/llm-credentials (encrypted at rest with libs/crypto.py, Fernet, key derived from config.llm.credential_encryption_key), else 400.

Findings lifecycle

flowchart LR
    A["analysis_service.record_analysis()\n(client-led PUT or server-led run_analysis)"] --> R["review_service.compute_initial_status()\nmatched rule.requires_review, or LOW/UNKNOWN confidence?"]
    R --> B["storage.upsert_finding()\nkeyed by org_id + resource_hash + rule_id"]
    B --> C{"first time seen?"}
    C -- yes --> D["status = open or needs_review\nfirst_seen = last_seen = now"]
    D -. needs_review .-> W["review_webhook_service.notify_needs_review()\n(trigger=auto)"]
    C -- no --> E["last_seen, occurrence_count refreshed\nstatus UNCHANGED (initial_status ignored)"]
    F["PATCH /findings/{hash}/{rule_id}"] --> G["status = open|needs_review|acknowledged|suppressed|false_positive"]
    G -- needs_review --> W2["review_webhook_service.notify_needs_review()\n(trigger=manual)"]
    G -.persists across re-scans.-> E

report_service.build_report_data() (shared by the report endpoints and the GitHub webhook job) joins a namespace’s resources against findings and, by default, drops suppressed/false_positive findings from the rendered report entirely (?include_suppressed=true to see them); acknowledged findings stay visible with a status badge.