Skip to content
DashboardWebsite

LLM Providers and Connections

Just want to do it in the dashboard? See the guide: Bring your own LLM connection or Cap the LLM spend and choose a model.

An LLM connection is an endpoint an organization uses to run analysis. It stores the address, credentials, authentication style, and structured-output mode. A self-hosted model needs these settings; a stored API key alone cannot express them.

In the dashboard, an org admin manages connections in Organization → Connections → LLM and can test each one there.

A provider kind is a wire protocol that determines which client the server runs. OpenTremor supports four:

KindWhat it talks to
anthropicThe hosted Claude API
openaiThe hosted OpenAI API
mistralThe hosted Mistral API
openai_compatibleAn endpoint that speaks /v1/chat/completions, such as your own model server, an OpenAI-compatible provider, or a gateway

The openai_compatible kind supports different vendors through the same format. To add a provider, configure its base URL and add catalog rows for its models.

You can point OpenTremor at a model running inside your network. Analysis requests go to the configured endpoint. If that endpoint is inside your network, the model call stays there.

RuntimeBase URLStructured output
vLLMhttp://host:8000/v1json_schema — grammar-constrained decoding with xgrammar
NVIDIA NIMhttp://host:8000/v1json_schema
Text Generation Inferencehttp://host:8080/v1json_schema, sometimes json_object
Ollamahttp://host:11434/v1json_object
llama.cpp (llama-server)http://host:8080/v1json_object, often prompt_only
LM Studio / LocalAIhttp://host:1234/v1json_object

Set auth_style: none for a server with no authentication. OpenTremor does not require a placeholder key.

These providers use the same kind with a different address. Each needs its own catalog rows (POST /v1/admin/models with llm_backend: "openai_compatible"), because you set the price per token.

ProviderBase URL
Groqhttps://api.groq.com/openai/v1
Together AIhttps://api.together.xyz/v1
Fireworkshttps://api.fireworks.ai/inference/v1
DeepSeekhttps://api.deepseek.com/v1
xAIhttps://api.x.ai/v1
OpenRouterhttps://openrouter.ai/api/v1
Nebius / Hugging Face Inferencevendor /v1

Gateways such as LiteLLM, Portkey, and Cloudflare AI Gateway can present the same interface. You can point one connection at a gateway to use its failover, load balancing, or spend caps. OpenTremor does not reimplement those features.

Each analysis requires the model to return a valid UnitAnalysis object. Hosted provider kinds use their configured structured-output mechanism. An openai_compatible endpoint can support different mechanisms, so its mode is set per connection:

ModeEnforcement
json_schemaThe server enforces the schema.
tool_callThe server is asked to return a forced function call.
json_objectThe server enforces JSON syntax; the prompt supplies the schema.
prompt_onlyThe server does not enforce JSON or the schema.

The default for a new custom connection is prompt_only because an unknown endpoint may not enforce a particular format. Every mode except tool_call includes a repair pass: if a response fails schema validation, the model gets one more try with the error before the unit fails. Use the connection probe to check the endpoint:

Terminal window
curl -X POST "$API/v1/orgs/$ORG/llm-connections/prod-vllm/test?model=qwen3-coder-30b" \
-H "Authorization: Bearer $TOKEN"

reachable: true with structured_output: false means the server responded, but did not return valid structured output under the configured mode.

Control which addresses connections can reach

Section titled “Control which addresses connections can reach”

An org admin creates connections. A platform admin controls where a custom address may point, in Platform Admin → Connections → LLM Endpoints or through PATCH /v1/admin/settings. Org admins cannot change this setting. The setting is llm.egress; the config file does not affect it.

PATCH /v1/admin/settings
{"llm_egress": {"enabled": true, "allow_private_networks": true, "allowed_hosts": ["llm.corp.example"]}}

By default, custom base_url values are not allowed. A self-hosted deployment can enable them and allow private networks to reach an internal endpoint such as http://10.0.0.5:8000. A multi-tenant cloud instance can keep the default to prevent orgs from probing the operator’s network. Add model server names to allowed_hosts to limit which hosts connections can reach.

Two rules apply in both configurations:

  • Link-local addresses are always blocked (169.254.0.0/16, fe80::/10). Cloud instance metadata can use these addresses. Allowing private networks does not allow access to link-local addresses.
  • OpenTremor checks resolved addresses, not only hostnames. Every address a hostname resolves to must pass the policy check. Checking resolved addresses prevents a hostname that points to 127.0.0.1 from bypassing the policy. OpenTremor checks again when a connection is used, not only when it is saved.

The needs_review webhook URL uses the separate notifications.egress policy. It applies the same address-validation rules.

Prices come from the models catalog (Platform Admin → Catalog → Models) and are keyed by provider kind and model name, not by connection.

  • Rows for openai_compatible are not seeded. A price for deepseek-chat would also apply to a self-hosted endpoint serving a model with that name, even if that endpoint has no per-token cost. Add catalog rows with the prices you choose.
  • A self-hosted connection catalogued at 0.00 does not add per-token cost to an organization’s monthly USD budget. The free trial, counted in analyses (trial_usage.analysis), is separate.

Some servers return no usage block. OpenTremor records usage as unreported, not as zero tokens. The probe reports this as usage_reported: false.

Use hosted providers without a stored connection

Section titled “Use hosted providers without a stored connection”

A request can name a hosted provider directly with connection_id: "anthropic" (or openai or mistral) and an inline api_key. The request goes to the vendor API without a stored connection. If you store a connection whose connection_id matches its provider_kind—for example, anthropic—the same request can omit the inline key.