Review Feedback

The feedback section controls the review feedback loop. Every field is editable live through PATCH /admin/settings (Platform Admin → Settings → Feedback) and read fresh on each run, so a change takes effect on the next analysis with no restart.

Notice what has no switch here: capture. Recording what a reviewer decided is inert and always worth doing. Every knob below governs use — whether those verdicts reach the analysis prompt, how much of them, and what they are never allowed to touch. That asymmetry is deliberate, because letting feedback steer an analysis is the part that could quietly weaken a security control.

Calibration

Field	Default	Bounds	What it does
calibration_enabled	true	—	Whether reviewer verdicts are composed into the analysis prompt at all. Off turns the loop into a pure record: feedback is still captured, aggregated into rule health, and still generates proposals — nothing reaches the LLM.
min_samples	3	1–100	Decisive verdicts (confirmed + rejected) before a rule steers anything. unclear verdicts don’t count.
max_entries_per_rule	5	1–50	Reviewer comments quoted per rule, newest first.
max_chars	4000	200–40000	Hard ceiling on the whole composed block. This rides every unit of every run, so an unbounded block is an unbounded increase in your own LLM spend. Over budget, whole rule entries are dropped from the tail — never half a quote.
severity_ceiling	HIGH	CRITICAL/HIGH/MEDIUM/LOW	The highest severity calibration may touch. A finding above this is always reported at full volume no matter how often the rule has been dismissed. Setting it to CRITICAL means nothing is exempt.
trusted_only	true	—	Whether only trusted feedback may reach a prompt. See the warning below.

trusted_only: false is a footgun

With it off, any commenter on a public pull request can write into the system prompt of every subsequent analysis for that org. It is configurable only because a deployment whose repositories are all private may legitimately not care. If you are not certain that describes you, leave it on.

Why min_samples matters more than it looks. A confirm/reject ratio over one or two samples is noise, and the cost of acting on noise here is a missed finding. The visible consequence is that a new org sees no calibration for weeks — that is correct behaviour, not a broken feature. The Rule Health page says per rule exactly why it isn’t calibrating yet.

Comment ingestion

Field	Default	Bounds	What it does
github_comments_enabled	true	—	Whether pull-request comments are ingested as feedback at all. Independent of calibration: you can collect comment-sourced feedback while composing none of it, or calibrate purely on dashboard triage while ignoring comments.
command_prefix	@opentremor	2–64 chars	What an explicit verdict in a PR comment must start with.

Rule proposals

Thresholds for the rule changes the evidence suggests. All are rates over decisive verdicts, and all produce a proposal a human accepts or dismisses — never a change.

Field	Default	Bounds	Fires when
proposal_rejection_threshold	0.6	0.0–1.0	Rejection rate at or above this proposes a severity downgrade.
proposal_disable_threshold	0.9	0.0–1.0	Rejection rate at or above this proposes disabling the rule outright — a much stronger claim, so a much higher bar.
proposal_confirm_threshold	0.9	0.0–1.0	Confirmation rate at or above this on a requires_review rule proposes clearing the hold: reviewers always agree with it, so blocking every pull request for a human decision buys nothing.

Example

{
  "feedback_calibration_enabled": true,
  "feedback_min_samples": 5,
  "feedback_severity_ceiling": "MEDIUM",
  "feedback_max_chars": 2000
}

A conservative posture: five verdicts before a rule steers anything, and nothing at HIGH or above is ever toned down by history.

Retention

How long feedback is kept is retention.feedback_days, in the retention section rather than here — it belongs with the other data-lifecycle fields, one sweep, one endpoint.