# Providers

> A model connection is one provider type plus a base URL, an API key and a model name. Projects route chat requests through one connection; RAG stores use one for embeddings.

Source: https://ragmux.com/docs/providers/
Updated: 2026-10-11
Project: https://github.com/ragmux/ragmux (AGPL-3.0-or-later)

---

A **model connection** is one provider type plus a base URL, an API key and a model
name. Projects route chat requests through one connection; RAG stores use one for
embeddings. Credentials are AES-256-GCM encrypted with `SECRET_KEY` and never leave the
gateway. Connections are managed with `POST /admin/api/models` (see the
[API reference](/docs/api/#model-connections)) or in the dashboard's **Models** tab;
`GET /admin/api/provider-types` lists the types with their defaults.

## Provider types

| `provider_type` | Default `base_url` | Chat endpoint used | Embeddings | Credentials |
|---|---|---|---|---|
| `openai` | `https://api.openai.com/v1` | `POST {base}/chat/completions` | yes — `POST {base}/embeddings` | `Authorization: Bearer <api_key>` |
| `anthropic` | `https://api.anthropic.com` | `POST {base}/v1/messages` | no | `x-api-key: <api_key>`, `anthropic-version: 2023-06-01` |
| `gemini` | `https://generativelanguage.googleapis.com/v1beta` | `POST {base}/models/{model}:generateContent` (`:streamGenerateContent?alt=sse` when streaming) | yes — `POST {base}/models/{model}:batchEmbedContents` | `x-goog-api-key: <api_key>` |
| `deepseek` | `https://api.deepseek.com/v1` | `POST {base}/chat/completions` | reported as unsupported (`supports_embeddings: false`) | `Authorization: Bearer <api_key>` |
| `ollama` | `http://localhost:11434` | `POST {base}/api/chat` (native API) | yes — `POST {base}/api/embed` | none; an `api_key` is sent as `Authorization: Bearer` for a proxy in front of Ollama |
| `custom_openai` | *(required; dashboard suggests `http://localhost:8000/v1`)* | `POST {base}/chat/completions` | yes — `POST {base}/embeddings` | `Authorization: Bearer <api_key>` (optional) |
| `azure_openai` | *(required; dashboard suggests `https://your-resource.openai.azure.com`)* | `POST {endpoint}/openai/deployments/{deployment}/chat/completions?api-version=…` | yes — `POST {endpoint}/openai/deployments/{deployment}/embeddings?api-version=…` | `api-key: <api_key>` (no `Authorization` header) |
| `cohere_rerank` | `https://api.cohere.com` | none — reranking only | no | `Authorization: Bearer <api_key>` |
| `voyage_rerank` | `https://api.voyageai.com` | none — reranking only | no | `Authorization: Bearer <api_key>` |

Base URL handling: a trailing `/` is trimmed. For `openai` and `deepseek` a `/v1` suffix
is appended when the URL does not already end in one. For `custom_openai` the URL is used
exactly as given (so include `/v1` yourself, e.g. `http://vllm:8000/v1`). For `ollama` a
trailing `/v1` is removed because the native endpoints live at the root. When the gateway
runs in Docker, local servers on the host are reachable as `http://host.docker.internal:<port>`.

A `base_url` must be a plain `http://` or `https://` URL with a host and without
credentials, query string or fragment (`azure_openai` alone may carry an `api-version`
query parameter, see [Azure OpenAI](#azure-openai)). Hosts on private or local networks (the Docker
host, a Compose service, `localhost`) are refused unless they are listed in
`PRIVATE_UPSTREAM_ALLOWLIST` or `ALLOW_PRIVATE_UPSTREAMS=true` is set — see
[Private upstreams](/docs/configuration/#private-upstreams). `model_name` may contain letters,
digits and `. _ : / @ -` (up to 128 characters, no leading `/`, no `..`); for `gemini` it
is sent as a single escaped path segment.

`POST /admin/api/models/{id}/test` with `{"mode":"chat"}` or `{"mode":"embedding"}`
sends a ping through the connection and reports the reply, or the dimensions for
embeddings, together with latency; the outcome is kept on the connection
(`last_test_at`, `last_test_ok`, `last_test_latency_ms`, `last_test_error`). The same
test runs on a connection that is not saved yet through `POST /admin/api/models/test`,
which takes the create body plus `mode` and, with an empty `api_key`, a `connection_id`
whose stored key is reused. Each connection also reports `private_upstream`: whether its
`base_url` pointed at a loopback, private or link-local address when it was saved. See
[REST API reference](/docs/api/#model-connections).

## Presets

Hosted providers that speak the OpenAI Chat Completions format need no adapter of their
own: they are `custom_openai` connections with a known base URL. `GET
/admin/api/provider-presets` lists the ready-made ones, each with the `provider_type`,
`base_url` and model names to use, so a client can offer them as a one-step choice:

| Preset `id` | Provider | `provider_type` | `base_url` | Suggested `model_name` |
|---|---|---|---|---|
| `mistral` | Mistral AI | `custom_openai` | `https://api.mistral.ai/v1` | `mistral-large-latest`, `mistral-medium-latest`, `mistral-small-latest`, `codestral-latest`; embeddings: `mistral-embed` |
| `xai` | xAI | `custom_openai` | `https://api.x.ai/v1` | `grok-4`, `grok-3`, `grok-3-mini` |
| `groq` | Groq | `custom_openai` | `https://api.groq.com/openai/v1` | `llama-3.3-70b-versatile`, `llama-3.1-8b-instant`, `openai/gpt-oss-120b`, `openai/gpt-oss-20b` |

Every preset needs an API key (`requires_api_key: true`), sent as `Authorization: Bearer`.
A preset is a convenience, not a stored type: send its `provider_type` and `base_url`,
and a `model_name` of your choice, to `POST /admin/api/models`. The result is an ordinary
`custom_openai` connection that is validated and tested like one typed by hand. The model names are suggestions; any name the
provider serves works, and a provider that retires one does not break existing
connections.

In the dashboard the same list is the **Start from a preset** select on the new-connection
form: picking one fills in the provider type, base URL and a model name, and nothing is
saved until you press **Save connection**. The preset itself is never sent.

Presets carry **no price**. `custom_openai` ships no built-in price rows, because the type
names a URL rather than a vendor and a row keyed on it would price every self-hosted
server like the hosted service. Add the models you use to the
[price table](/docs/api/#model-prices) (provider type `custom_openai`) to get cost figures;
until then usage is counted and a request is logged with `cost_source: "none"` and a cost
of zero.

## Capabilities

What each adapter can do with a provider type. These describe the **adapter**, not the
upstream model: a model that cannot see images still shows `vision` for its provider
type, because the translation exists — the model's own refusal is relayed as it comes.

| | `openai` | `anthropic` | `gemini` | `deepseek` | `ollama` | `custom_openai` | `azure_openai` |
|---|---|---|---|---|---|---|---|
| `chat` | yes | yes | yes | yes | yes | yes | yes |
| `streaming` | yes | yes | yes | yes | yes | yes | yes |
| `embeddings` | yes | no | yes | no | yes | yes | yes |
| `tools` | yes | yes | yes | yes | yes | yes | yes |
| `tool_streaming` | yes | yes | no ¹ | yes | no ¹ | yes | yes |
| `vision` | yes | yes | yes | no | yes | yes | yes |
| `remote_images` | yes | yes | no ² | no | no ² | yes | yes |
| `prompt_caching` | no ³ | yes | no | no | no | no | no ³ |
| `cached_token_usage` | yes | yes | yes | yes | no | no | yes |
| `rerank` | no | no | no | no | no | no | no |

`cohere_rerank` and `voyage_rerank` sit outside that table: `rerank` is the only
capability they have, and `chat` and `embeddings` are both `no`. `chat` is what the
dashboard's model picker filters on: a connection whose type has it can back a project
or a chat route target.

¹ Tool calls still work in streams; they arrive as one complete delta per call instead of
argument fragments, because that is how the upstream sends them — see
[Ollama](#ollama-native-api) and [Gemini](#gemini).
² The upstream cannot fetch an image URL, so the gateway downloads it and sends the bytes;
see [Images](#images).
³ OpenAI and Azure OpenAI cache prompts automatically, so there is no marker to send.

The table gates nothing: an adapter answers for itself with a typed error when a request
asks for something it cannot do, and a second copy of those rules in front of it would
only drift. It is a description, for the dashboard and for this page.

`GET /admin/api/provider-types` reports it per type: the flat `supports_embeddings`,
`requires_api_key`, `supports_streaming` and `supports_tools` fields, plus a
`capabilities` object with the full table above.

## Rerank providers (`cohere_rerank`, `voyage_rerank`)

These two types are connections that can do exactly one thing: reorder retrieved
passages by relevance to a query. **They cannot back a project and they cannot back a
RAG store's embeddings.** `provider.New` and `NewEmbedder` reject them, the dashboard's
model and embedding pickers filter them out, and saving a project or a store that names
one is refused with `400` (`provider "cohere_rerank" cannot be used for chat`,
`... cannot be used for embeddings`). The only place a connection of these types belongs
is a RAG store's `rerank_connection_id` or a `kind = rerank` route target (served by
`POST /v1/rerank`) — see [Reranking](/docs/rag/#reranking).

| | `cohere_rerank` | `voyage_rerank` |
|---|---|---|
| Endpoint | `POST {base}/v2/rerank` | `POST {base}/v1/rerank` |
| Request | `{"model","query","documents","top_n"}` | `{"model","query","documents","top_k","truncation":true}` |
| Response | `{"results":[{"index","relevance_score"}]}` | `{"data":[{"index","relevance_score"}]}` |
| `model_name` | e.g. `rerank-v3.5` | e.g. `rerank-2.5` |

They are ordinary model connections in every other respect: the API key is AES-256-GCM
encrypted at rest with the connection id as additional authenticated data, it is covered
by `ragmux rotate-key`, it is masked in the API, the `base_url` goes through the same
SSRF guard when it is saved, and the call itself travels the gateway's one hardened
outbound path — the netguard dialer, the redirect cap, the fixed transport-error
messages, redaction and the response body cap. `POST /admin/api/models/{id}/test` does
not apply to them: they answer neither a chat nor an embedding ping. Test them from the
RAG store's search panel instead, which shows whether reranking ran or was skipped.

Each passage is cut to 4000 characters before it is sent. Indexes the API answers with
are validated against the list that was sent: anything out of range or repeated is
dropped, and a reply with nothing usable left is a `502`. Every failure — a timeout, a
`429`, a malformed body — leaves the fused retrieval order in place; the request is
answered without reranking and the reason is logged.

## OpenAI-compatible (`openai`, `deepseek`, `custom_openai`)

The client request is forwarded as-is apart from `model` (replaced by the connection's
model name), `stream` (set by the gateway) and `stream_options` (set to
`{"include_usage": true}` on streams so token counts are recorded; removed on
non-streaming calls).

Top-level request fields travel as sent (see
[Request field passthrough](#request-field-passthrough-extra)), but inside `messages` only
`role`, `content`, `name`, `tool_calls` and `tool_call_id` are kept; the **response** is
rewritten as well. Replies and SSE chunks are
decoded into the gateway's own OpenAI-shaped structures and written back out, so only the
fields it knows come through:

- top level: `id`, `object`, `created`, `model`, `choices` and `usage` (with
  `prompt_tokens_details` and `completion_tokens_details`);
- per choice: `index`, `finish_reason` and a `message` (a `delta` on streams) holding
  `role`, `content` and `tool_calls`.

Anything else the upstream adds is **not** forwarded: `reasoning_content`, `refusal`,
`annotations`, `logprobs` and `system_fingerprint` are all dropped, even when the request
asked for them (`logprobs: true` reaches the upstream, as the passthrough below says, and
its answer does not come back). A client that needs one of them cannot get it through the
gateway in this release.

What the gateway asks the upstream for is not what it passes on: the usage-only trailer
chunk reaches the client only when the client itself sent
`stream_options: {"include_usage": true}`. The same rule covers `anthropic`, `gemini`
and `ollama`, whose adapters all emit that trailer so token counts are recorded — see
[the streaming contract](/docs/api/#post-v1chatcompletions).

`custom_openai` covers vLLM, LM Studio, LiteLLM, text-generation-inference's OpenAI
route, Ollama's `/v1` shim and any other server speaking the Chat Completions format.

Prompt caching on these providers is automatic and server-side: nothing is sent for it,
and `prompt_tokens_details.cached_tokens` (DeepSeek's `prompt_cache_hit_tokens`) is
surfaced back to the client. See [Prompt caching](#prompt-caching).

### Request field passthrough (`Extra`)

`/v1/chat/completions` parses the known fields (`model`, `messages`, `stream`,
`temperature`, `top_p`, `max_tokens`, `max_completion_tokens`, `stop`, `n`, `tools`,
`tool_choice`, `response_format`, `stream_options`, `user`) and keeps every other
top-level field verbatim. For OpenAI-compatible providers those extra fields are sent to
the upstream unchanged, so provider-specific options (`seed`, a request-side `logprobs`,
`reasoning_effort`, vLLM's `guided_json`, …) work without gateway support. For `ollama` a
defined subset is
mapped (below); `anthropic` and `gemini` ignore unknown fields.

## Azure OpenAI

`azure_openai` talks to an Azure OpenAI resource. It uses the Chat Completions body and
response, so streaming, tool calls, images and embeddings behave as they do for `openai`;
what differs is how the request is addressed and authenticated:

```
POST {endpoint}/openai/deployments/{deployment}/chat/completions?api-version={version}
POST {endpoint}/openai/deployments/{deployment}/embeddings?api-version={version}
api-key: <api_key>
```

The key travels in the `api-key` header; no `Authorization: Bearer` header is sent. A
model connection has no column for the version, so the four settings map onto the
existing fields:

| Setting | Field | Example |
|---|---|---|
| Endpoint | `base_url` | `https://my-resource.openai.azure.com` |
| Deployment | `model_name` | `my-gpt4o` (the deployment name you chose in Azure, not the model family) |
| API version | `api-version` query parameter of `base_url`, optional | `https://my-resource.openai.azure.com?api-version=2025-04-01-preview` |
| Key | `api_key` | one of the resource's two keys |

`$AUTH` is the bearer header from the login example in [the API reference](/docs/api/).

```bash
curl -X POST http://localhost:8765/admin/api/models \
  -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"name":"azure-gpt4o","provider_type":"azure_openai",
       "base_url":"https://my-resource.openai.azure.com",
       "model_name":"my-gpt4o","api_key":"<azure key>"}'
```

Without an `api-version` the gateway uses `2024-10-21`. Pass another one when a feature
you need (a newer model, a preview capability) requires it. The same connection can serve
a RAG store's embeddings when its deployment is an embedding model; chat and embedding
deployments are separate connections.

The dashboard's connection form shows these settings under their Azure names (endpoint,
**Deployment name**, **API version**) and joins them into `base_url` and `model_name` on
save. Leave the API version blank to use the default.

Validation on save: `base_url` is required and takes no credentials or fragment, and its
only allowed query parameter is a single `api-version` (letters, digits, `.`, `_`, `-`);
a pasted trailing `/` or `/openai` is accepted and removed. `model_name` is the deployment
name: letters, digits, `.`, `_` and `-`, starting with a letter or digit, up to 64
characters. The endpoint goes through the same private-address check as every other
`base_url`; use `PRIVATE_UPSTREAM_ALLOWLIST` for a private endpoint such as an Azure
Private Link address.

The key sits in the same encrypted `api_key` field as every other provider key: AES-256-GCM
at rest, masked in the API, and re-encrypted by `ragmux rotate-key` along with the rest.
Upstream errors are relayed and redacted like any other provider's, so the key never
appears in one.

Azure prices are not built in; add them to the [price table](/docs/api/#model-prices) with
provider type `azure_openai`.

## Anthropic

Requests are translated to the Messages API and responses back to the OpenAI schema:

- `system` and `developer` messages are removed from the message list, joined with blank
  lines and sent as the top-level `system` field (this is where the project system prompt
  and RAG context end up). With a `cache_control` on one of them the field becomes an
  array of blocks instead — see [Prompt caching](#prompt-caching).
- `max_tokens` is required by Anthropic; when the client sends neither `max_tokens` nor
  `max_completion_tokens` the gateway uses **4096**.
- `temperature`, `top_p` and `stop` are mapped; `n`, `response_format`, `user` and unknown
  fields are dropped.
- `response_format` of type `json_schema` or `json_object` is not an error: the request is
  still sent, without it, so the model is not held to the schema. The response carries
  `Warning: 299 ragmux "response_format is not supported for anthropic connections;
  ignored"` (set before the first byte, so streams have it too) and the gateway logs the
  event at info level with the format type only. Ask for JSON in the prompt instead.
- Consecutive messages with the same role are merged, as the Messages API requires.
- Tools: OpenAI `tools[].function` definitions become Anthropic `tools` (`input_schema`
  from `parameters`, `{"type":"object","properties":{}}` when absent); `tool_choice`
  `"auto"` → `{"type":"auto"}`, `"required"` → `{"type":"any"}`, a named function →
  `{"type":"tool","name":…}`, `"none"` removes the tools. Assistant `tool_calls` become
  `tool_use` blocks, `tool` messages become `tool_result` blocks, and `tool_use` in the
  reply becomes OpenAI `tool_calls` (also in streams, via `input_json_delta`).
- Image parts: `image_url` with a `data:` URL is sent as a base64 image block, any other
  URL as a URL image block — Anthropic fetches it itself, which is cheaper than relaying
  the bytes through the gateway. A Bedrock-style gateway in front of Anthropic that has no
  `source.type: url` needs the images inlined instead; that is the one constant
  `anthropicInlineImages` in `internal/provider/images.go`.
- Finish reasons: `end_turn`/`stop_sequence` → `stop`, `max_tokens` → `length`,
  `tool_use` → `tool_calls`, `refusal` → `content_filter`,
  `model_context_window_exceeded` → `length`; any other `stop_reason` (`pause_turn`, a
  value Anthropic adds later) → `stop`, logged at warn level with the value. Usage is mapped to `prompt_tokens`/`completion_tokens`,
  including on streams; see [Prompt caching](#prompt-caching) for how the cache
  counters are folded in.

Anthropic has no embeddings API; choose another connection for RAG stores.

## Gemini

Requests are translated to `generateContent` / `streamGenerateContent`:

- `system`/`developer` messages become `systemInstruction`; `assistant` turns are sent
  with role `model`.
- `temperature`, `top_p`, `max_tokens`/`max_completion_tokens` and `stop` map to
  `generationConfig` (`topP`, `maxOutputTokens`, `stopSequences`); a `response_format`
  whose `type` starts with `json` sets `responseMimeType: application/json`.
- Image parts: a `data:` URL becomes `inlineData`. `fileData.fileUri` carries only what
  Gemini can resolve itself — a Files API object
  (`https://generativelanguage.googleapis.com/v1beta/files/…`) or a `gs://` path. Every
  other image URL is downloaded by the gateway and sent as `inlineData`; with
  `IMAGE_FETCH=false` such a request is refused with `400`. See [Images](#images).
- Finish reasons: `STOP` → `stop`, `MAX_TOKENS` → `length`, safety-related reasons
  (`SAFETY`, `RECITATION`, `BLOCKLIST`, `PROHIBITED_CONTENT`, `SPII`) → `content_filter`.
  A turn that called tools reports `tool_calls` whatever Gemini said, because Gemini has
  no distinct reason for it. `usageMetadata` is mapped to OpenAI usage, including
  `cachedContentTokenCount` — see [Prompt caching](#prompt-caching). Explicit caching
  (`cachedContents`) is not supported.
- Embeddings use `batchEmbedContents`; the model name gets a `models/` prefix if missing.

### Tool calling

Gemini has function calling but neither call ids nor JSON Schema, so the translation does
more work than elsewhere:

- `tools[].function` definitions become one `functionDeclarations` list;
  `tool_choice` maps to `toolConfig.functionCallingConfig`: `"auto"` → `AUTO`,
  `"required"` → `ANY`, `"none"` → `NONE`, a named function → `ANY` with
  `allowedFunctionNames`. Unlike the Anthropic path, `"none"` keeps the declarations —
  Gemini has an exact equivalent, so the model still knows the tools exist.
- Assistant `tool_calls` become `functionCall` parts; arguments that are not a JSON object
  are sent as `{}`.
- **Results are correlated by function name**, not by id. A `tool` message's
  `tool_call_id` is looked up in the assistant `tool_calls` that preceded it and the
  recovered name is sent as a `functionResponse` with role `user` (`Content.role` only
  takes `user` or `model`; consecutive ones merge, which is what parallel results need).
  A result whose call is not in the message history is dropped, with a debug log line,
  rather than turned into an upstream `400`. `response` must be a JSON object: the content
  is used as-is when it parses as one, otherwise wrapped as `{"result": "<text>"}`.
- In streams a `functionCall` always arrives complete inside one chunk, so each call is
  emitted as one `tool_calls` delta carrying the index, id, name and arguments together.
  No argument fragments are invented.

- **Thought signatures round-trip through the client.** A thinking model attaches an
  opaque `thoughtSignature` to the function call it makes, and expects it back when the
  tool result is sent. Ragmux relays it on each tool call as an extra field,
  `ragmux_signature`, in the response (the JSON reply and the streamed delta alike), and
  puts it back on the `functionCall` part when an assistant message carrying it comes in
  on the next turn. A client that appends the assistant message to the history unchanged —
  as the OpenAI SDKs do with the message object they received — sends it back without
  doing anything. Under parallel calling only the first call carries a signature. The
  field means nothing outside the connection that issued it: OpenAI-compatible connections
  have it removed from the request before it is sent.

  **Known limit.** A client that rebuilds the tool calls by hand — in a stream, collecting
  `id`, `name` and `arguments` from the deltas and ignoring other fields — drops the
  signature, and Gemini can then answer the next turn with a `400`. Such a `400` is
  relayed as any other provider error (see [Error relay and
  redaction](#error-relay-and-redaction)). Copy `ragmux_signature` along with the other
  fields of each tool call. A signature that arrives in a stream chunk after the first
  call's delta has already been sent is not relayed.

**Known limit — tool-use tokens.** On a tool round Gemini also reports
`usageMetadata.toolUsePromptTokenCount`, which Ragmux neither reads nor maps. Gemini's
`totalTokenCount` is relayed as `total_tokens` as it arrives, so on those requests
`prompt_tokens + completion_tokens` can be **less than** `total_tokens`, and whatever the
difference covers is not priced: the cost estimate is built from the prompt and
completion counts alone. Google's own references disagree on what `totalTokenCount`
sums — the REST reference says prompt + thoughts + candidates, the published
`generativelanguage` protobuf says prompt + candidates — and neither states whether
tool-use tokens are inside it, so Ragmux does not guess. Every other provider keeps
`prompt_tokens + completion_tokens == total_tokens`. Read a Gemini tool-round bill from
Google's console, not from here.

### Tool schema sanitising

Gemini's `parameters` take an OpenAPI 3.0 subset and **reject** keywords they do not know,
rather than ignoring them. Every OpenAI strict-mode tool carries
`"additionalProperties": false` and most non-trivial ones carry `$defs`/`$ref`, so a
schema forwarded unchanged fails with `Invalid JSON payload received. Unknown name
"additionalProperties"` — an error that names something the caller never wrote by hand.
Each schema is therefore rewritten before it is sent.

Kept: `type`, `description`, `enum`, `items`, `properties`, `required`, `nullable`,
`anyOf`, `minimum`, `maximum`, `minItems`, `maxItems`, `minLength`, `maxLength`,
`pattern`, and `format` only for `enum`/`date-time` on strings and
`float`/`double`/`int32`/`int64` on numbers.

Kept means kept as the type Gemini's `Schema` declares, not kept as written: `description`
and `pattern` must be strings, `nullable` a boolean, `minimum`/`maximum` doubles,
`minItems`/`maxItems`/`minLength`/`maxLength` **non-negative whole numbers** (they are
`int64` there, so `{"minItems":1.5}` and `{"maxLength":1e30}` are dropped), and `enum` an
array of strings, because `Schema.enum` is `repeated string`. A value of any other shape
is dropped like an unknown keyword — `{"type":"integer","enum":[1,2,3]}` loses its `enum`,
and so does an `enum` on a node that is not a string, or on an `anyOf` node, rather than
travelling to Gemini as something it rejects.

Rewritten:

| From | To |
|---|---|
| `const: "x"` | `enum: ["x"]` |
| `const: 7`, `const: true`, `const: {…}` | `type: "integer"` / `"boolean"` / `"object"` — the value cannot be carried, but it still pins the type; dropped if the node already has one |
| `oneOf`, `allOf` | `anyOf` — an `allOf` whose branches are all objects is shallow-merged into the node instead |
| `type: ["string","null"]` | `type: "string"` + `nullable: true`; a wider union keeps its first non-`null` member |
| `exclusiveMinimum: N`, `exclusiveMaximum: N` | `minimum: N`, `maximum: N` |
| `$ref` | the definition from `$defs`/`definitions`, inlined |

A `$ref` is matched on its full JSON Pointer, so `#/$defs/Foo`, `#/definitions/Foo` and a
`Foo` nested under another node are three different definitions, and an external
`https://…#/definitions/Foo` resolves to none of them (it is elided, since that document
is never fetched). Properties and keywords are walked in sorted order, so the same schema
always sanitises to the same bytes.

Dropped: `$schema`, `additionalProperties`, `title`, `default`, `examples`, `$comment`,
`patternProperties`, `not`, `if`/`then`/`else`, `unevaluated*`, other `format` values and
any keyword not listed above. The dropped names are logged once per tool at `debug` with
`provider=gemini`.

`$ref` inlining is depth-capped at 8 and cycle-aware; a recursive or unresolvable
reference becomes `{"type":"string","description":"(recursive schema elided)"}` for a
cycle and `"(unresolved schema reference elided)"` for a definition this document does not
contain, which since references are matched on their full pointer is every external one. A
node that is not an object, or one nothing survived in, becomes `{"type":"object"}`, a node
that carried no `type` takes the one its surviving keywords imply (an `anyOf` node stays
untyped, because its branches carry the types) — that inference is a guess, and a lossy
one: a node carrying only `minimum`/`maximum` becomes `number`, so a field the caller
meant as an integer can come back with a fractional value, and one carrying only
`minItems` becomes `string`, keeping an array keyword on a scalar. Gemini rejects an
untyped node outright, so a guess beats the alternative; give a `type` to anything whose
shape matters. A declaration whose sanitised
schema has no properties left is sent **without** `parameters` — several model versions
reject `{"type":"object","properties":{}}`. Sanitising never fails; anything Gemini still
objects to comes back relayed verbatim.

This is lossy on purpose. A tool that relies on `oneOf` to discriminate between argument
shapes, on `additionalProperties: false` to forbid extra keys, or on a recursive `$ref`
behaves differently on Gemini than on OpenAI. If a tool depends on those, keep it on a
provider whose schema support is wider.

## Ollama (native API)

The `ollama` type talks to Ollama's own `/api/chat` (one JSON object, or newline-delimited
JSON objects when streaming) and `/api/embed` endpoints instead of the OpenAI shim, which
gives access to Ollama-only options:

- `temperature`, `top_p`, `max_tokens` (→ `num_predict`) and `stop` are placed in
  `options`.
- Extra request fields are mapped: `keep_alive` is passed through as-is, `num_ctx` is
  added to `options`, and an `options` object is merged into `options` (any Ollama model
  option, e.g. `{"options":{"num_gpu":1,"repeat_penalty":1.1}}`).
- `response_format` `json_object` sets `format: "json"`; `json_schema` sends the schema as
  `format`.
- Tools use OpenAI's `{type:"function", function:{…}}` shape natively, so `tools` are
  forwarded; assistant `tool_calls` and `tool` messages (with `tool_name`) are translated
  both ways. Ollama sends no call ids, so the gateway generates them (`call_…`).
- In streams each tool call arrives **whole** inside one NDJSON line, so it is relayed as
  one `tool_calls` delta carrying the index, id, name and arguments together. This is a
  property of the protocol, not a gap: Ollama genuinely produces the arguments atomically,
  splitting them into fragments would invent framing that carries no information, and
  every OpenAI-shaped client accumulates `arguments` by string concatenation. Calls spread
  over several lines keep distinct indexes.
- Images are sent to Ollama as inline base64. An `image_url` that is not a `data:` URL is
  downloaded by the gateway first (see [Images](#images)); with `IMAGE_FETCH=false` it is
  rejected with `400`.
- Finish reasons: `stop` → `stop`, `length` → `length`; usage is built from
  `prompt_eval_count` / `eval_count`. Ollama has no prompt cache, so no cache fields are
  reported and the shipped price table charges `ollama` models nothing.

Ollama itself is unauthenticated; if you set an `api_key` it is sent as
`Authorization: Bearer …`, which is useful behind an authenticating reverse proxy.

Ollama usually runs on a private address, which the gateway refuses by default. Add its
hostname to the allowlist in `.env` before creating the connection:

```bash
PRIVATE_UPSTREAM_ALLOWLIST=host.docker.internal,ollama   # Docker host, or a Compose service named "ollama"
```

Then use `base_url` `http://host.docker.internal:11434` (Ollama on the host) or
`http://ollama:11434` (Ollama as a Compose service). `ALLOW_PRIVATE_UPSTREAMS=true`
allows every private host instead; see
[Private upstreams](/docs/configuration/#private-upstreams).

To use Ollama's OpenAI-compatible endpoint instead, create a `custom_openai` connection
with `base_url` ending in `/v1` (for example `http://host.docker.internal:11434/v1`).
The native type is preferable for `keep_alive`, `num_ctx` and batch embeddings.

## Images

A user message may carry OpenAI image parts:
`{"type":"image_url","image_url":{"url":"…"}}`, either an inline
`data:image/png;base64,…` URL or a remote one. Inline images are checked and then passed
to every vision-capable provider as they are. Remote ones depend on the provider:

| Provider | Remote `image_url` |
|---|---|
| `openai`, `custom_openai` | forwarded unchanged; the upstream fetches it |
| `anthropic` | forwarded as a `source.type: url` image block; Anthropic fetches it |
| `gemini` | Files API and `gs://` URIs are forwarded as `fileData`; anything else is fetched by the gateway and inlined |
| `ollama` | fetched by the gateway and inlined — Ollama accepts nothing else |

An inline `data:` URL is held to the same whitelist a downloaded image is, and refused
with the gateway's own `400` when it does not meet it — before anything is sent upstream:

- the URL must carry `;base64`, so `data:image/png,%89PNG…` is refused rather than
  forwarded as if it were encoded (the `data:` scheme itself is matched case-insensitively);
- the media type must be one **this** provider accepts (see the table below), so
  `data:text/html;base64,…` no longer travels as an image;
- the payload must decode as standard base64, **padding included**, and must not be
  empty — an empty one used to vanish silently, leaving the client a `200` for a message
  the model never saw an image in.

These used to pass through untouched and came back as a provider `400` that named nothing
the caller could act on. The check belongs to the translating adapters (`gemini`,
`ollama`, `anthropic`); the OpenAI-compatible types relay the request body verbatim by
design, so there the upstream is still the one that reads the part.

The accepted formats are the ones each upstream documents, and they are not the same set:

| Provider | Accepted image media types |
|---|---|
| `gemini` | `image/png`, `image/jpeg`, `image/webp`, `image/heic`, `image/heif` |
| `anthropic` | `image/png`, `image/jpeg`, `image/gif`, `image/webp` |
| `ollama` | `image/png`, `image/jpeg`, `image/gif`, `image/webp` |

One shared list would be wrong in both directions — it would refuse the HEIC Gemini takes
and promise Anthropic a format its API rejects — so each connection is held to its own,
inline and downloaded alike.

The gateway's own fetch (`IMAGE_FETCH`, on by default) goes out over the same hardened
transport as provider calls but under a **stricter** policy: an image URL that resolves to
a loopback, link-local or private address is refused, and a redirect to another host — or
one that drops from `https` to `http` on the same host — is not followed.

The difference matters. `ALLOW_PRIVATE_UPSTREAMS` and `PRIVATE_UPSTREAM_ALLOWLIST` lift
the private-address check for provider calls — allowlisting a host is the documented way
to reach a local Ollama — but they **do not apply to image fetching**. A `base_url` is
typed by an editor; an image URL arrives from whoever holds an API key, so letting it
inherit that exemption would turn every allowlisted deployment into a way for a key holder
to probe the internal network. On top of that:

- `GET` only, with `Accept: image/*`, and only `http`/`https` URLs.
- The response's own `Content-Type` decides, against the same per-provider list an inline
  image is held to. The URL's extension is never consulted. The cache is keyed on the URL
  alone, so a hit is re-checked against the asking provider's list rather than trusted.
- `IMAGE_FETCH_MAX_MB` is enforced on `Content-Length` and again while reading, so a
  response that declares nothing (or lies) cannot exceed it.
- Images are fetched one at a time, at most `IMAGE_FETCH_MAX_PER_REQUEST` per chat
  request, each within `IMAGE_FETCH_TIMEOUT`.
- Across requests, `IMAGE_FETCH_MAX_CONCURRENT` caps the fetches the whole process runs
  at once and the connections it will hold to one host. A per-request limit bounds one
  request and nothing at all when many arrive together, which is what would make the
  gateway a useful amplifier for a target an API key chose. **While another project is
  queueing, no single project takes more than half those slots**: capping the process
  alone closed the outward problem and opened an inward one, where a few requests aimed at
  a slow image host filled every slot and an unrelated project waited out its whole
  timeout.
- **Takes, not holds.** A slot already taken is never reclaimed. A project that filled the
  gateway while nobody was waiting keeps those downloads running and can sit above its
  half until they finish; what changes the moment somebody else queues is that it takes
  nothing further until it is back inside the share. Cutting a running download short to
  rebalance would throw away a fetch that is already part paid for, which is not what this
  ceiling is for. So the share is a floor under everybody else rather than a ceiling on
  one tenant: with nobody else waiting, a single project reaches the whole of
  `IMAGE_FETCH_MAX_CONCURRENT`, which is what the setting says it is.
- A fetch that cannot get a slot within two seconds (or `IMAGE_FETCH_TIMEOUT`, if that is
  shorter — the wait is not separately configurable) is answered `429` with `Retry-After`
  and `code: "image_fetch_saturated"`. The gateway ran out, which is a resource limit
  rather than the client getting anything wrong, and deliberately not a `5xx`, which would
  report a fault and count against availability when nothing is broken. `Retry-After`
  names `IMAGE_FETCH_TIMEOUT`, not the queue wait: a slot frees when a **download**
  finishes, so naming the wait sent clients that honour the header — the official SDKs do,
  for a `429` as much as for a `5xx` — straight back into the same full queue. The queue
  wait is its own budget, separate from `IMAGE_FETCH_TIMEOUT`, which starts once a slot is
  in hand; otherwise the same saturation came back sometimes as a `429` and sometimes as a
  download timeout blaming the image host.

Anything the client got wrong — a bad scheme, a non-image response, an oversized image,
a non-2xx from the image host, too many images — is a `400 invalid_request_error` naming
only the **host** of the offending URL, never the full URL (which may carry a signed query
string). A non-2xx is deliberately a `400` and not a `502`: the client chose the host.

Fetched images are cached in memory (`IMAGE_CACHE_ENTRIES`, `IMAGE_CACHE_MAX_MB`,
`IMAGE_CACHE_TTL`) because a
multi-turn conversation resends the same image part on every turn. The cache is keyed on
the URL alone and is not written to disk, so an image whose content changes within the TTL
keeps serving the old bytes until it expires; a response carrying `Cache-Control: no-store`
is never cached.

## Prompt caching

Providers cache a repeated prompt prefix and charge less for the cached part. Ragmux
passes a client's caching intent through where the provider has a field for it, and
always reports back what the provider said it cached. Both sides feed the
[cost estimate](/docs/api/#model-prices) shown on request logs.

### Response side: one accounting for every provider

`usage` gains OpenAI's breakdown objects, so the same fields mean the same thing on
every connection:

```json
"usage": {"prompt_tokens": 250, "completion_tokens": 500, "total_tokens": 750,
          "prompt_tokens_details": {"cached_tokens": 200, "cache_creation_tokens": 40},
          "completion_tokens_details": {"reasoning_tokens": 30}}
```

`prompt_tokens` **always includes** the cached and freshly written parts: the mapping
guarantees that. `prompt_tokens + completion_tokens == total_tokens` is weaker — it is
how each adapter assembles the block from an upstream that reports consistently, and it
is not checked, so a Gemini tool round (see [Tool calling](#tool-calling)) or any
upstream reporting a total that does not add up breaks it; the block is relayed as it
came. `cache_creation_tokens` has no OpenAI equivalent — OpenAI does not bill cache
writes, Anthropic does.

| Provider | What it reports | How it is mapped |
|---|---|---|
| `openai` | `prompt_tokens_details.cached_tokens` (already inside `prompt_tokens`) | passed through unchanged |
| `deepseek` | `prompt_cache_hit_tokens` / `prompt_cache_miss_tokens` at the top level (their sum is `prompt_tokens`) | the hit count is folded into `cached_tokens`; the top-level fields are still relayed |
| `anthropic` | `input_tokens` **excluding** `cache_creation_input_tokens` and `cache_read_input_tokens` | all three are added up into `prompt_tokens`; reads become `cached_tokens`, writes `cache_creation_tokens` |
| `gemini` | `usageMetadata.cachedContentTokenCount` (already inside `promptTokenCount`) | becomes `cached_tokens`; `thoughtsTokenCount` keeps counting as completion tokens and is also reported as `reasoning_tokens` |
| `ollama` | nothing; Ollama has no prompt cache | no cache fields |

> **Change from 0.3.x:** for a cached Anthropic request, `prompt_tokens` is now larger.
> Earlier releases reported Anthropic's `input_tokens` verbatim and so under-reported a
> cached prompt by the size of its cached prefix. Recorded token budgets and metrics for
> such requests move up accordingly.

### Request side: `cache_control`

**Anthropic** takes a `cache_control` marker in three places, and Ragmux relays each of
them verbatim (the gateway never inspects or invents one):

| Where the client puts it | What is sent upstream |
|---|---|
| on a content part: `{"type":"text","text":"…","cache_control":{"type":"ephemeral"}}` | copied onto the matching Anthropic content block |
| on a tool, at the top level *or* inside `function` | copied onto the Anthropic tool object |
| on a `system`/`developer` message's content part | the `system` field becomes an array of blocks, one per system message, carrying the marker |

With no `cache_control` anywhere, `system` stays the plain joined string it has always
been — a request that does not ask for caching is byte-identical to a 0.3.x one. When a
message has several marked parts the last marker wins, since Anthropic caches the prefix
up to and including the marked block.

**RAG context drops the system marker.** When a project's RAG store injects passages, the
context block is prepended to the first `system`/`developer` message (in front of the
project `system_prompt`, which is itself prepended to the client's own system text —
see [Context injection](/docs/rag/#context-injection)), and the
`cache_control` on that message's parts is removed. The block changes with every query
and comes first, so a breakpoint behind it would make Anthropic write a new cache entry
(billed above the plain input rate) on each request and never read one back. The parts
themselves and every other field are kept; markers on other messages, user content and
tools are untouched, and a project `system_prompt` on its own does not drop anything.
Placing the context after the client's cached prefix is planned for v0.6.

**OpenAI, DeepSeek and `custom_openai`: nothing is sent.** Their caching is automatic and
server-side; there is no request field to emit, so Ragmux emits none. A `cache_control` a
client embeds rides along untouched (message content is relayed raw) and OpenAI ignores
it. Only the response side matters here — this is not a gap.

**Gemini and Ollama: response side only.** Gemini's explicit caching needs a stateful
`cachedContents` resource created and referenced across requests, which does not fit a
stateless passthrough; its implicit caching happens automatically and is reported in
`cachedContentTokenCount`. Ollama has no prompt cache at all.

The project system prompt and the RAG context block the gateway injects are the stable
prefix most worth a breakpoint, but the gateway does not mark one: a project-level
`cache_prompt` switch needs a column and dashboard work of its own and is not in 0.4.

## Model field echo

A project without chat routes ignores the client's `model` for selection: the project's
connection decides. The gateway replaces the value with the connection's model name before
calling the provider and echoes the client's original value back in the response (and in
every stream chunk) so SDKs that compare the field stay happy. When the client sends an
empty `model`, the connection's model name is returned instead. `GET /v1/models` lists that
model name with `owned_by` set to the provider type.

## Routing

A project can define named routes (see [Routes](/docs/api/#routes)). Each route has one to
eight targets: a connection, an optional `model` override, a `weight` and a `priority`.
Once a project has at least one chat route, the client's `model` selects what runs:

1. A route name matches exactly: the route's targets are ordered (priority groups
   ascending, weighted random inside a group) and the first one is called.
2. Otherwise the default connection's model name selects the implicit route: the project's
   own connection, as before.
3. An empty `model` also selects the implicit route.
4. Anything else is `404` `model_not_found`.

The target's `model` replaces the connection's model name for the call; the provider,
pricing, request log and trace all see that effective model. The response `model` still
echoes the client's value (for an empty `model`, the connection's model name). In
`request_logs`, `model_name` is the effective model, `requested_model` is the client's
value and `route` is the route name, empty for the implicit route. Routes of kind
`embeddings` or `rerank` do not switch chat routing on.

The project page has a route editor for all of this: a route's kind (`chat`, `embeddings`
or `rerank`) and up to eight targets, each with its connection, optional `model`, `weight`
and `priority`.

### Before your first chat route

Without chat routes the gateway answers whatever `model` string a client sends (it echoes
it back). The first chat route ends that: any string that is neither a route name nor the
default connection's model name becomes `404 model_not_found`. Check what your clients
actually send before you create it:

```sql
SELECT requested_model, count(*) FROM request_logs
WHERE project_id = <project id> GROUP BY requested_model ORDER BY count(*) DESC;
```

Create a route for every value that matters, or point those clients at the default
connection's model name. Rolling back is deleting the route: with no chat routes left the
project echoes any `model` again.

## Retry and fallback

A chat call that fails before the first byte reached the client can be repeated on the
same target (retry) or handed to the route's next target (fallback). The order of targets
is the one [Routing](#routing) describes; a project without routes has one target, so only
retries apply. What happens is decided by the kind of failure:

| Failure | Retry | Fallback |
|---|---|---|
| The request never left: DNS, refused connection, TLS handshake | yes | yes |
| Connection reset or cut after the request was sent | no | yes |
| `429`, `503` (and Anthropic's `529`) with no `Retry-After`, or one of 2 s or less | yes | yes |
| `429`, `503` (and Anthropic's `529`) with `Retry-After` above 2 s | no | yes |
| `500`, `502`, other `5xx` | no | yes |
| `504` or a timeout | no | only if budget is left |
| `401`, `403`, `404` (the key or model is wrong for that target) | no | yes |
| `400`, `413`, `422` and other `4xx`, a client that went away | no | no |

A reset after the request was sent is never retried on the same target: the provider may
already have processed it, and a repeat could be billed twice. A `529` from a provider
other than Anthropic is not retried either; it only hands the request to the next target.
A failure to download an image URL the client sent is the client's, not the target's: it
is answered as it is, with neither retry nor fallback.

**Retries.** Each target is retried up to `UPSTREAM_MAX_RETRIES` times (default `2`, `0`
turns them off). The wait before a retry is exponential with full jitter: a random time
below 250 ms, 500 ms, 1 s and so on, capped at 2 s. When the upstream sent `Retry-After`
of 2 s or less, the longer of the two is waited. A longer `Retry-After` is not waited for:
the next target takes over, and when there is none the client gets the error with
`Retry-After` set to the upstream's value.

**Budget.** `UPSTREAM_TIMEOUT` is the budget of the whole request, not of one attempt.
Each attempt runs with what is left of it, and a wait that would use up the remainder is
skipped, so the attempts of a non-streaming request never add up to more than
`UPSTREAM_TIMEOUT`.
A streaming attempt is held to what is left of the budget only until its first chunk
arrives; once the stream is flowing `UPSTREAM_TIMEOUT` no longer limits it, and
`STREAM_MAX_DURATION` and `STREAM_MAX_BYTES_MB` do.
The rate-limit check, the system prompt and retrieval run once before the first attempt,
and usage is recorded once.

**Streams.** Retry and fallback apply only until the first chunk is sent, including an
upstream that answers `2xx` and fails before it sends anything. After the response headers
went out the stream is never restarted or joined to another one: a cut stream ends the way
it did before.

**Circuit breaker.** A connection that keeps failing is taken out of the candidate list for
a while, so retries and fallbacks do not spend the budget on a dead target. Every
connection has its own breaker, held in memory by each replica; it is never written to the
database, so replicas do not share it. A breaker has three states:

- `closed`: calls pass and consecutive failures are counted. A success resets the count.
- `open`: reached after `BREAKER_FAILURE_THRESHOLD` failures in a row (default `5`, `0`
  turns breaking off). The connection is skipped for `BREAKER_OPEN_DURATION` (default
  `30s`), and a target that opens in the middle of a request is not retried any more.
- `half-open`: once the duration has passed, one call goes through as a probe. If it
  succeeds the breaker closes; if it fails the breaker opens again for a full duration.
  Other requests keep skipping the connection while the probe is in flight, and a probe
  that ends in a way that says nothing about the upstream, such as a client that went
  away, frees the slot for the next request.

What counts as a failure is what the table above calls an upstream fault: the request
never left (DNS, refused connection, TLS handshake), a reset after it was sent, `429`,
`503` and `529`, other `5xx`, and `504` or a timeout. `401`, `403` and `404` do not count,
although they still hand the request to the next target, and neither do other `4xx` or a
client that went away. Such an attempt leaves the count where it was. The same holds for
a stream: a client that leaves before or during it, or an upstream that fails after its
first byte (the client already got a `200`), is neither a success nor a failure, so it
neither resets the count nor closes a half-open breaker.

An open connection is skipped when the route's targets are walked, so a request goes
straight to the next healthy one. When every target of a route is open, the first one is
called exactly once, without retries; there is no immediate `503`, so a recovered upstream
is noticed by the first request that reaches it. A project without routes has one target
and behaves as before. The state is per replica, so a fleet of `N` replicas sends up to `N`
probes to a bad upstream, and a restart starts every breaker `closed`. Opening and closing
are logged at `warn` and `info` level with the connection name, and
`ragmux_upstream_breaker_state` reports the state per connection
([Observability](/docs/observability/#the-metric-set)).

**What the client sees.** The error of the last attempt, in the format of
[Error relay and redaction](#error-relay-and-redaction). A fallback that succeeds is
invisible apart from the request log: one row per request, `attempt` is the number of
upstream calls in total, `fallback_from` is the connection id of the first target whenever
the request moved past it (also when no target answered), and `model_name` is the model of
the last target tried. A target whose adapter cannot be built is skipped without counting
as a call; on a route with several targets that includes the first one. Each failed
attempt is logged at `warn` level with its connection id and class. Retries and fallbacks
are counted in `ragmux_upstream_retries_total` and `ragmux_upstream_fallbacks_total`
([Observability](/docs/observability/#the-metric-set)); a failure that is neither retried nor
handed on, such as a `400`, reaches neither.

## Error relay and redaction

Provider errors are relayed with the upstream message (cut to 512 characters) in the
OpenAI error envelope. A `4xx` keeps its status, and so do `503` and `504`, which tell
the client to come back later; Anthropic's `529 overloaded` means the same and becomes
`503`. Every other upstream `5xx` becomes `502`. The upstream's
`Retry-After` (delay seconds or an HTTP date) is passed on unchanged; a value in neither
form is dropped. The gateway's own `429`s set their own `Retry-After` instead. Transport failures are never relayed verbatim:
they become `502 upstream_error` with one of `upstream unreachable`, `upstream TLS
handshake failed`, `upstream returned a non-HTTP response`, `upstream redirect
rejected: …`, the private-address message from
[Private upstreams](/docs/configuration/#private-upstreams), or `upstream request failed`;
timeouts become `504 timeout` / `upstream request timed out`. The raw cause is logged at
warn level with the upstream host. Streaming responses end with an error after
`STREAM_MAX_DURATION` or `STREAM_MAX_BYTES_MB`.

Before an upstream message reaches a client, the request log or the gateway's own log,
the connection's own API key and anything matching `Bearer …`, `Basic …`, a JWT
(`eyJ…`), `sk-…`, `sk-ant-…`, `AIza…`, `ya29.…`, `gsk_…`, `hf_…`, `xai-…` or credentials
embedded in a URL (`https://user:pass@host`) is replaced by `[redacted]`. This covers
Anthropic stream `error` events and Ollama error bodies as well.