Skip to content
DashboardWebsite

Self-Hosted LLM Endpoint

Just want to do it in the dashboard? See the guide: Let orgs reach your LLM endpoints.

Data residency makes this setup useful: organizations that cannot send infrastructure code to a third-party API can still run analysis against a model they operate. This page explains how a platform admin and an org admin connect OpenTremor to a self-hosted model server. For the connection fields and egress settings, see LLM Providers and Connections.

In the dashboard: Platform Admin → Connections → LLM Endpoints.

  1. Turn on Allow custom endpoints.
  2. Turn on Allow private network addresses if the model server uses a private address.
  3. Optionally, add hostnames under Allowed hosts to restrict which hosts org admins can use. With no host restrictions, org admins can use any host allowed by the address rules.

Or use the API:

Terminal window
curl -X PATCH "$API/v1/admin/settings" \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"llm_egress": {"enabled": true, "allow_private_networks": true,
"allowed_hosts": ["*.internal.corp"]}}'

Org admins cannot change this setting. A config file does not override it. If custom endpoints are disabled, creating a connection with a base_url returns a 400 error. Link-local addresses remain blocked. See the reference page.

For example, vLLM can serve an OpenAI-compatible endpoint:

Terminal window
vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
--host 0.0.0.0 --port 8000 \
--guided-decoding-backend xgrammar

Ollama works for evaluation and needs only ollama serve. Its endpoint is http://host:11434/v1, and use json_object for structured output.

In the dashboard: Organization → Connections → LLM.

Or use the API:

Terminal window
curl -X POST "$API/v1/orgs/$ORG/llm-connections" \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{
"connection_id": "prod-vllm",
"display_name": "Production vLLM (eu-west)",
"provider_kind": "openai_compatible",
"base_url": "http://llm.internal.corp:8000/v1",
"auth_style": "none",
"structured_output_mode": "json_schema"
}'

Set auth_style to "none" only if the server does not require authentication. For a server that uses a private CA, include the CA bundle as tls_ca_pem. OpenTremor stores it encrypted and does not return it.

In the dashboard: Organization → Connections → LLM. Enter the model name for the connection and select Test.

Or use the API:

Terminal window
curl -X POST "$API/v1/orgs/$ORG/llm-connections/prod-vllm/test?model=Qwen/Qwen3-Coder-30B-A3B-Instruct" \
-H "Authorization: Bearer $TOKEN"

The test makes one real analysis call against a small fixture. It checks the address, authentication, model name, and structured output. A successful test means the full call worked. Run it before starting a large job.

OpenTremor checks the allowed-models catalog before analysis calls. The catalog does not include openai_compatible models by default, so a platform admin must add the model.

In the dashboard: Platform Admin → Catalog → Models → Add a model.

Or use the API:

Terminal window
curl -X POST "$API/v1/admin/models" \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"llm_backend": "openai_compatible", "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
"input_price_per_1m_usd": 0, "output_price_per_1m_usd": 0}'

Set prices to match your organization’s accounting. A zero price does not count toward the organization’s monthly USD budget. You can enter a notional rate for internal chargeback.

In the dashboard: Changesets → select a changeset → Analyze.

Or use the API:

Terminal window
curl -X POST "$API/v1/python-code-change/$CHANGESET/analyze" \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"connection_id": "prod-vllm",
"model": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
"raw_input": "'"$(cat change.diff)"'"}'

The GitHub integration can also use the same connection_id so pull request checks use your endpoint.

Failures caused by an endpoint that does not complete a request are treated as transient and retried up to analysis.llm_max_attempts. If a job reports a truncation error, increase analysis.llm_max_output_tokens. Smaller models may need more output tokens to complete an analysis.