LLM Providers and Connections
Just want to do it in the dashboard? See the guide: Bring your own LLM connection or Cap the LLM spend and choose a model.
An LLM connection is an endpoint an organization uses to run analysis. It stores the address, credentials, authentication style, and structured-output mode. A self-hosted model needs these settings; a stored API key alone cannot express them.
In the dashboard, an org admin manages connections in Organization → Connections → LLM and can test each one there.
A provider kind is a wire protocol that determines which client the server runs. OpenTremor supports four:
| Kind | What it talks to |
|---|---|
anthropic | The hosted Claude API |
openai | The hosted OpenAI API |
mistral | The hosted Mistral API |
openai_compatible | An endpoint that speaks /v1/chat/completions, such as your own model server, an OpenAI-compatible provider, or a gateway |
The openai_compatible kind supports different vendors through the same format. To add a provider, configure its base URL and add catalog rows for its models.
Self-hosted endpoints
Section titled “Self-hosted endpoints”You can point OpenTremor at a model running inside your network. Analysis requests go to the configured endpoint. If that endpoint is inside your network, the model call stays there.
| Runtime | Base URL | Structured output |
|---|---|---|
| vLLM | http://host:8000/v1 | json_schema — grammar-constrained decoding with xgrammar |
| NVIDIA NIM | http://host:8000/v1 | json_schema |
| Text Generation Inference | http://host:8080/v1 | json_schema, sometimes json_object |
| Ollama | http://host:11434/v1 | json_object |
llama.cpp (llama-server) | http://host:8080/v1 | json_object, often prompt_only |
| LM Studio / LocalAI | http://host:1234/v1 | json_object |
Set auth_style: none for a server with no authentication. OpenTremor does not require a placeholder key.
OpenAI-compatible hosted providers
Section titled “OpenAI-compatible hosted providers”These providers use the same kind with a different address. Each needs its own catalog rows (POST /v1/admin/models with llm_backend: "openai_compatible"), because you set the price per token.
| Provider | Base URL |
|---|---|
| Groq | https://api.groq.com/openai/v1 |
| Together AI | https://api.together.xyz/v1 |
| Fireworks | https://api.fireworks.ai/inference/v1 |
| DeepSeek | https://api.deepseek.com/v1 |
| xAI | https://api.x.ai/v1 |
| OpenRouter | https://openrouter.ai/api/v1 |
| Nebius / Hugging Face Inference | vendor /v1 |
Gateways such as LiteLLM, Portkey, and Cloudflare AI Gateway can present the same interface. You can point one connection at a gateway to use its failover, load balancing, or spend caps. OpenTremor does not reimplement those features.
Choose a structured-output mode
Section titled “Choose a structured-output mode”Each analysis requires the model to return a valid UnitAnalysis object. Hosted provider kinds use their configured structured-output mechanism. An openai_compatible endpoint can support different mechanisms, so its mode is set per connection:
| Mode | Enforcement |
|---|---|
json_schema | The server enforces the schema. |
tool_call | The server is asked to return a forced function call. |
json_object | The server enforces JSON syntax; the prompt supplies the schema. |
prompt_only | The server does not enforce JSON or the schema. |
The default for a new custom connection is prompt_only because an unknown endpoint may not enforce a particular format. Every mode except tool_call includes a repair pass: if a response fails schema validation, the model gets one more try with the error before the unit fails. Use the connection probe to check the endpoint:
curl -X POST "$API/v1/orgs/$ORG/llm-connections/prod-vllm/test?model=qwen3-coder-30b" \ -H "Authorization: Bearer $TOKEN"reachable: true with structured_output: false means the server responded, but did not return valid structured output under the configured mode.
Control which addresses connections can reach
Section titled “Control which addresses connections can reach”An org admin creates connections. A platform admin controls where a custom address may point, in Platform Admin → Connections → LLM Endpoints or through PATCH /v1/admin/settings. Org admins cannot change this setting. The setting is llm.egress; the config file does not affect it.
{"llm_egress": {"enabled": true, "allow_private_networks": true, "allowed_hosts": ["llm.corp.example"]}}By default, custom base_url values are not allowed. A self-hosted deployment can enable them and allow private networks to reach an internal endpoint such as http://10.0.0.5:8000. A multi-tenant cloud instance can keep the default to prevent orgs from probing the operator’s network. Add model server names to allowed_hosts to limit which hosts connections can reach.
Two rules apply in both configurations:
- Link-local addresses are always blocked (
169.254.0.0/16,fe80::/10). Cloud instance metadata can use these addresses. Allowing private networks does not allow access to link-local addresses. - OpenTremor checks resolved addresses, not only hostnames. Every address a hostname resolves to must pass the policy check. Checking resolved addresses prevents a hostname that points to
127.0.0.1from bypassing the policy. OpenTremor checks again when a connection is used, not only when it is saved.
The needs_review webhook URL uses the separate notifications.egress policy. It applies the same address-validation rules.
Account for model costs
Section titled “Account for model costs”Prices come from the models catalog (Platform Admin → Catalog → Models) and are keyed by provider kind and model name, not by connection.
- Rows for
openai_compatibleare not seeded. A price fordeepseek-chatwould also apply to a self-hosted endpoint serving a model with that name, even if that endpoint has no per-token cost. Add catalog rows with the prices you choose. - A self-hosted connection catalogued at
0.00does not add per-token cost to an organization’s monthly USD budget. The free trial, counted in analyses (trial_usage.analysis), is separate.
Some servers return no usage block. OpenTremor records usage as unreported, not as zero tokens. The probe reports this as usage_reported: false.
Use hosted providers without a stored connection
Section titled “Use hosted providers without a stored connection”A request can name a hosted provider directly with connection_id: "anthropic" (or openai or mistral) and an inline api_key. The request goes to the vendor API without a stored connection. If you store a connection whose connection_id matches its provider_kind—for example, anthropic—the same request can omit the inline key.