Self-Hosted LLM Endpoint
Just want to do it in the dashboard? See the guide: Let orgs reach your LLM endpoints.
Data residency makes this setup useful: organizations that cannot send infrastructure code to a third-party API can still run analysis against a model they operate. This page explains how a platform admin and an org admin connect OpenTremor to a self-hosted model server. For the connection fields and egress settings, see LLM Providers and Connections.
1. Allow custom endpoints
Section titled “1. Allow custom endpoints”In the dashboard: Platform Admin → Connections → LLM Endpoints.
- Turn on Allow custom endpoints.
- Turn on Allow private network addresses if the model server uses a private address.
- Optionally, add hostnames under Allowed hosts to restrict which hosts org admins can use. With no host restrictions, org admins can use any host allowed by the address rules.
Or use the API:
curl -X PATCH "$API/v1/admin/settings" \ -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -d '{"llm_egress": {"enabled": true, "allow_private_networks": true, "allowed_hosts": ["*.internal.corp"]}}'Org admins cannot change this setting. A config file does not override it. If custom endpoints are disabled, creating a connection with a base_url returns a 400 error. Link-local addresses remain blocked. See the reference page.
2. Run a model server
Section titled “2. Run a model server”For example, vLLM can serve an OpenAI-compatible endpoint:
vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \ --host 0.0.0.0 --port 8000 \ --guided-decoding-backend xgrammarOllama works for evaluation and needs only ollama serve. Its endpoint is http://host:11434/v1, and use json_object for structured output.
3. Create the connection
Section titled “3. Create the connection”In the dashboard: Organization → Connections → LLM.
Or use the API:
curl -X POST "$API/v1/orgs/$ORG/llm-connections" \ -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -d '{ "connection_id": "prod-vllm", "display_name": "Production vLLM (eu-west)", "provider_kind": "openai_compatible", "base_url": "http://llm.internal.corp:8000/v1", "auth_style": "none", "structured_output_mode": "json_schema" }'Set auth_style to "none" only if the server does not require authentication. For a server that uses a private CA, include the CA bundle as tls_ca_pem. OpenTremor stores it encrypted and does not return it.
4. Test the connection
Section titled “4. Test the connection”In the dashboard: Organization → Connections → LLM. Enter the model name for the connection and select Test.
Or use the API:
curl -X POST "$API/v1/orgs/$ORG/llm-connections/prod-vllm/test?model=Qwen/Qwen3-Coder-30B-A3B-Instruct" \ -H "Authorization: Bearer $TOKEN"The test makes one real analysis call against a small fixture. It checks the address, authentication, model name, and structured output. A successful test means the full call worked. Run it before starting a large job.
5. Add the model to the catalog
Section titled “5. Add the model to the catalog”OpenTremor checks the allowed-models catalog before analysis calls. The catalog does not include openai_compatible models by default, so a platform admin must add the model.
In the dashboard: Platform Admin → Catalog → Models → Add a model.
Or use the API:
curl -X POST "$API/v1/admin/models" \ -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -d '{"llm_backend": "openai_compatible", "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "input_price_per_1m_usd": 0, "output_price_per_1m_usd": 0}'Set prices to match your organization’s accounting. A zero price does not count toward the organization’s monthly USD budget. You can enter a notional rate for internal chargeback.
6. Use the connection for analysis
Section titled “6. Use the connection for analysis”In the dashboard: Changesets → select a changeset → Analyze.
Or use the API:
curl -X POST "$API/v1/python-code-change/$CHANGESET/analyze" \ -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -d '{"connection_id": "prod-vllm", "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "raw_input": "'"$(cat change.diff)"'"}'The GitHub integration can also use the same connection_id so pull request checks use your endpoint.
What to expect
Section titled “What to expect”Failures caused by an endpoint that does not complete a request are treated as transient and retried up to analysis.llm_max_attempts. If a job reports a truncation error, increase analysis.llm_max_output_tokens. Smaller models may need more output tokens to complete an analysis.