Providers

A model connection is one provider type plus a base URL, an API key and a model name. Projects route chat requests through one connection; RAG stores use one for embeddings. Credentials are AES-256-GCM encrypted with SECRET_KEY and never leave the gateway. Connections are managed with POST /admin/api/models (see the API reference) or in the dashboard’s Models tab; GET /admin/api/provider-types lists the types with their defaults.

Provider types

Base URL handling: a trailing / is trimmed. For openai and deepseek a /v1 suffix is appended when the URL does not already end in one. For custom_openai the URL is used exactly as given (so include /v1 yourself, e.g. http://vllm:8000/v1). For ollama a trailing /v1 is removed because the native endpoints live at the root. When the gateway runs in Docker, local servers on the host are reachable as http://host.docker.internal:<port>.

A base_url must be a plain http:// or https:// URL with a host and without credentials, query string or fragment (azure_openai alone may carry an api-version query parameter, see Azure OpenAI). Hosts on private or local networks (the Docker host, a Compose service, localhost) are refused unless they are listed in PRIVATE_UPSTREAM_ALLOWLIST or ALLOW_PRIVATE_UPSTREAMS=true is set — see Private upstreams. model_name may contain letters, digits and . _ : / @ - (up to 128 characters, no leading /, no ..); for gemini it is sent as a single escaped path segment.

POST /admin/api/models/{id}/test with {"mode":"chat"} or {"mode":"embedding"} sends a ping through the connection and reports the reply, or the dimensions for embeddings, together with latency; the outcome is kept on the connection (last_test_at, last_test_ok, last_test_latency_ms, last_test_error). The same test runs on a connection that is not saved yet through POST /admin/api/models/test, which takes the create body plus mode and, with an empty api_key, a connection_id whose stored key is reused. Each connection also reports private_upstream: whether its base_url pointed at a loopback, private or link-local address when it was saved. See REST API reference.

Presets

Hosted providers that speak the OpenAI Chat Completions format need no adapter of their own: they are custom_openai connections with a known base URL. GET /admin/api/provider-presets lists the ready-made ones, each with the provider_type, base_url and model names to use, so a client can offer them as a one-step choice:

Every preset needs an API key (requires_api_key: true), sent as Authorization: Bearer. A preset is a convenience, not a stored type: send its provider_type and base_url, and a model_name of your choice, to POST /admin/api/models. The result is an ordinary custom_openai connection that is validated and tested like one typed by hand. The model names are suggestions; any name the provider serves works, and a provider that retires one does not break existing connections.

In the dashboard the same list is the Start from a preset select on the new-connection form: picking one fills in the provider type, base URL and a model name, and nothing is saved until you press Save connection. The preset itself is never sent.

Presets carry no price. custom_openai ships no built-in price rows, because the type names a URL rather than a vendor and a row keyed on it would price every self-hosted server like the hosted service. Add the models you use to the price table (provider type custom_openai) to get cost figures; until then usage is counted and a request is logged with cost_source: "none" and a cost of zero.

Capabilities

What each adapter can do with a provider type. These describe the adapter, not the upstream model: a model that cannot see images still shows vision for its provider type, because the translation exists — the model’s own refusal is relayed as it comes.

cohere_rerank and voyage_rerank sit outside that table: rerank is the only capability they have, and chat and embeddings are both no. chat is what the dashboard’s model picker filters on: a connection whose type has it can back a project or a chat route target.

¹ Tool calls still work in streams; they arrive as one complete delta per call instead of argument fragments, because that is how the upstream sends them — see Ollama and Gemini. ² The upstream cannot fetch an image URL, so the gateway downloads it and sends the bytes; see Images. ³ OpenAI and Azure OpenAI cache prompts automatically, so there is no marker to send.

The table gates nothing: an adapter answers for itself with a typed error when a request asks for something it cannot do, and a second copy of those rules in front of it would only drift. It is a description, for the dashboard and for this page.

GET /admin/api/provider-types reports it per type: the flat supports_embeddings, requires_api_key, supports_streaming and supports_tools fields, plus a capabilities object with the full table above.

Rerank providers (cohere_rerank, voyage_rerank)

These two types are connections that can do exactly one thing: reorder retrieved passages by relevance to a query. They cannot back a project and they cannot back a RAG store’s embeddings. provider.New and NewEmbedder reject them, the dashboard’s model and embedding pickers filter them out, and saving a project or a store that names one is refused with 400 (provider "cohere_rerank" cannot be used for chat, ... cannot be used for embeddings). The only place a connection of these types belongs is a RAG store’s rerank_connection_id or a kind = rerank route target (served by POST /v1/rerank) — see Reranking.

They are ordinary model connections in every other respect: the API key is AES-256-GCM encrypted at rest with the connection id as additional authenticated data, it is covered by ragmux rotate-key, it is masked in the API, the base_url goes through the same SSRF guard when it is saved, and the call itself travels the gateway’s one hardened outbound path — the netguard dialer, the redirect cap, the fixed transport-error messages, redaction and the response body cap. POST /admin/api/models/{id}/test does not apply to them: they answer neither a chat nor an embedding ping. Test them from the RAG store’s search panel instead, which shows whether reranking ran or was skipped.

Each passage is cut to 4000 characters before it is sent. Indexes the API answers with are validated against the list that was sent: anything out of range or repeated is dropped, and a reply with nothing usable left is a 502. Every failure — a timeout, a 429, a malformed body — leaves the fused retrieval order in place; the request is answered without reranking and the reason is logged.

OpenAI-compatible (openai, deepseek, custom_openai)

The client request is forwarded as-is apart from model (replaced by the connection’s model name), stream (set by the gateway) and stream_options (set to {"include_usage": true} on streams so token counts are recorded; removed on non-streaming calls).

Top-level request fields travel as sent (see Request field passthrough), but inside messages only role, content, name, tool_calls and tool_call_id are kept; the response is rewritten as well. Replies and SSE chunks are decoded into the gateway’s own OpenAI-shaped structures and written back out, so only the fields it knows come through:

• top level: id, object, created, model, choices and usage (with prompt_tokens_details and completion_tokens_details);
• per choice: index, finish_reason and a message (a delta on streams) holding role, content and tool_calls.

Anything else the upstream adds is not forwarded: reasoning_content, refusal, annotations, logprobs and system_fingerprint are all dropped, even when the request asked for them (logprobs: true reaches the upstream, as the passthrough below says, and its answer does not come back). A client that needs one of them cannot get it through the gateway in this release.

What the gateway asks the upstream for is not what it passes on: the usage-only trailer chunk reaches the client only when the client itself sent stream_options: {"include_usage": true}. The same rule covers anthropic, gemini and ollama, whose adapters all emit that trailer so token counts are recorded — see the streaming contract.

custom_openai covers vLLM, LM Studio, LiteLLM, text-generation-inference’s OpenAI route, Ollama’s /v1 shim and any other server speaking the Chat Completions format.

Prompt caching on these providers is automatic and server-side: nothing is sent for it, and prompt_tokens_details.cached_tokens (DeepSeek’s prompt_cache_hit_tokens) is surfaced back to the client. See Prompt caching.

Request field passthrough (Extra)

/v1/chat/completions parses the known fields (model, messages, stream, temperature, top_p, max_tokens, max_completion_tokens, stop, n, tools, tool_choice, response_format, stream_options, user) and keeps every other top-level field verbatim. For OpenAI-compatible providers those extra fields are sent to the upstream unchanged, so provider-specific options (seed, a request-side logprobs, reasoning_effort, vLLM’s guided_json, …) work without gateway support. For ollama a defined subset is mapped (below); anthropic and gemini ignore unknown fields.

Azure OpenAI

azure_openai talks to an Azure OpenAI resource. It uses the Chat Completions body and response, so streaming, tool calls, images and embeddings behave as they do for openai; what differs is how the request is addressed and authenticated:

POST {endpoint}/openai/deployments/{deployment}/chat/completions?api-version={version}
POST {endpoint}/openai/deployments/{deployment}/embeddings?api-version={version}
api-key: <api_key>

The key travels in the api-key header; no Authorization: Bearer header is sent. A model connection has no column for the version, so the four settings map onto the existing fields:

$AUTH is the bearer header from the login example in the API reference.

curl -X POST http://localhost:8765/admin/api/models \
  -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"name":"azure-gpt4o","provider_type":"azure_openai",
       "base_url":"https://my-resource.openai.azure.com",
       "model_name":"my-gpt4o","api_key":"<azure key>"}'

Without an api-version the gateway uses 2024-10-21. Pass another one when a feature you need (a newer model, a preview capability) requires it. The same connection can serve a RAG store’s embeddings when its deployment is an embedding model; chat and embedding deployments are separate connections.

The dashboard’s connection form shows these settings under their Azure names (endpoint, Deployment name, API version) and joins them into base_url and model_name on save. Leave the API version blank to use the default.

Validation on save: base_url is required and takes no credentials or fragment, and its only allowed query parameter is a single api-version (letters, digits, ., _, -); a pasted trailing / or /openai is accepted and removed. model_name is the deployment name: letters, digits, ., _ and -, starting with a letter or digit, up to 64 characters. The endpoint goes through the same private-address check as every other base_url; use PRIVATE_UPSTREAM_ALLOWLIST for a private endpoint such as an Azure Private Link address.

The key sits in the same encrypted api_key field as every other provider key: AES-256-GCM at rest, masked in the API, and re-encrypted by ragmux rotate-key along with the rest. Upstream errors are relayed and redacted like any other provider’s, so the key never appears in one.

Azure prices are not built in; add them to the price table with provider type azure_openai.

Anthropic

Requests are translated to the Messages API and responses back to the OpenAI schema:

• system and developer messages are removed from the message list, joined with blank lines and sent as the top-level system field (this is where the project system prompt and RAG context end up). With a cache_control on one of them the field becomes an array of blocks instead — see Prompt caching.
• max_tokens is required by Anthropic; when the client sends neither max_tokens nor max_completion_tokens the gateway uses 4096.
• temperature, top_p and stop are mapped; n, response_format, user and unknown fields are dropped.
• response_format of type json_schema or json_object is not an error: the request is still sent, without it, so the model is not held to the schema. The response carries Warning: 299 ragmux "response_format is not supported for anthropic connections; ignored" (set before the first byte, so streams have it too) and the gateway logs the event at info level with the format type only. Ask for JSON in the prompt instead.
• Consecutive messages with the same role are merged, as the Messages API requires.
• Tools: OpenAI tools[].function definitions become Anthropic tools (input_schema from parameters, {"type":"object","properties":{}} when absent); tool_choice "auto" → {"type":"auto"}, "required" → {"type":"any"}, a named function → {"type":"tool","name":…}, "none" removes the tools. Assistant tool_calls become tool_use blocks, tool messages become tool_result blocks, and tool_use in the reply becomes OpenAI tool_calls (also in streams, via input_json_delta).
• Image parts: image_url with a data: URL is sent as a base64 image block, any other URL as a URL image block — Anthropic fetches it itself, which is cheaper than relaying the bytes through the gateway. A Bedrock-style gateway in front of Anthropic that has no source.type: url needs the images inlined instead; that is the one constant anthropicInlineImages in internal/provider/images.go.
• Finish reasons: end_turn/stop_sequence → stop, max_tokens → length, tool_use → tool_calls, refusal → content_filter, model_context_window_exceeded → length; any other stop_reason (pause_turn, a value Anthropic adds later) → stop, logged at warn level with the value. Usage is mapped to prompt_tokens/completion_tokens, including on streams; see Prompt caching for how the cache counters are folded in.

Anthropic has no embeddings API; choose another connection for RAG stores.

Gemini

Requests are translated to generateContent / streamGenerateContent:

• system/developer messages become systemInstruction; assistant turns are sent with role model.
• temperature, top_p, max_tokens/max_completion_tokens and stop map to generationConfig (topP, maxOutputTokens, stopSequences); a response_format whose type starts with json sets responseMimeType: application/json.
• Image parts: a data: URL becomes inlineData. fileData.fileUri carries only what Gemini can resolve itself — a Files API object (https://generativelanguage.googleapis.com/v1beta/files/…) or a gs:// path. Every other image URL is downloaded by the gateway and sent as inlineData; with IMAGE_FETCH=false such a request is refused with 400. See Images.
• Finish reasons: STOP → stop, MAX_TOKENS → length, safety-related reasons (SAFETY, RECITATION, BLOCKLIST, PROHIBITED_CONTENT, SPII) → content_filter. A turn that called tools reports tool_calls whatever Gemini said, because Gemini has no distinct reason for it. usageMetadata is mapped to OpenAI usage, including cachedContentTokenCount — see Prompt caching. Explicit caching (cachedContents) is not supported.
• Embeddings use batchEmbedContents; the model name gets a models/ prefix if missing.

Tool calling

Gemini has function calling but neither call ids nor JSON Schema, so the translation does more work than elsewhere:

• tools[].function definitions become one functionDeclarations list; tool_choice maps to toolConfig.functionCallingConfig: "auto" → AUTO, "required" → ANY, "none" → NONE, a named function → ANY with allowedFunctionNames. Unlike the Anthropic path, "none" keeps the declarations — Gemini has an exact equivalent, so the model still knows the tools exist.

• Assistant tool_calls become functionCall parts; arguments that are not a JSON object are sent as {}.

• Results are correlated by function name, not by id. A tool message’s tool_call_id is looked up in the assistant tool_calls that preceded it and the recovered name is sent as a functionResponse with role user (Content.role only takes user or model; consecutive ones merge, which is what parallel results need). A result whose call is not in the message history is dropped, with a debug log line, rather than turned into an upstream 400. response must be a JSON object: the content is used as-is when it parses as one, otherwise wrapped as {"result": "<text>"}.

• In streams a functionCall always arrives complete inside one chunk, so each call is emitted as one tool_calls delta carrying the index, id, name and arguments together. No argument fragments are invented.

• Thought signatures round-trip through the client. A thinking model attaches an opaque thoughtSignature to the function call it makes, and expects it back when the tool result is sent. Ragmux relays it on each tool call as an extra field, ragmux_signature, in the response (the JSON reply and the streamed delta alike), and puts it back on the functionCall part when an assistant message carrying it comes in on the next turn. A client that appends the assistant message to the history unchanged — as the OpenAI SDKs do with the message object they received — sends it back without doing anything. Under parallel calling only the first call carries a signature. The field means nothing outside the connection that issued it: OpenAI-compatible connections have it removed from the request before it is sent.

  Known limit. A client that rebuilds the tool calls by hand — in a stream, collecting id, name and arguments from the deltas and ignoring other fields — drops the signature, and Gemini can then answer the next turn with a 400. Such a 400 is relayed as any other provider error (see Error relay and redaction). Copy ragmux_signature along with the other fields of each tool call. A signature that arrives in a stream chunk after the first call’s delta has already been sent is not relayed.

Known limit — tool-use tokens. On a tool round Gemini also reports usageMetadata.toolUsePromptTokenCount, which Ragmux neither reads nor maps. Gemini’s totalTokenCount is relayed as total_tokens as it arrives, so on those requests prompt_tokens + completion_tokens can be less than total_tokens, and whatever the difference covers is not priced: the cost estimate is built from the prompt and completion counts alone. Google’s own references disagree on what totalTokenCount sums — the REST reference says prompt + thoughts + candidates, the published generativelanguage protobuf says prompt + candidates — and neither states whether tool-use tokens are inside it, so Ragmux does not guess. Every other provider keeps prompt_tokens + completion_tokens == total_tokens. Read a Gemini tool-round bill from Google’s console, not from here.

Tool schema sanitising

Gemini’s parameters take an OpenAPI 3.0 subset and reject keywords they do not know, rather than ignoring them. Every OpenAI strict-mode tool carries "additionalProperties": false and most non-trivial ones carry $defs/$ref, so a schema forwarded unchanged fails with Invalid JSON payload received. Unknown name "additionalProperties" — an error that names something the caller never wrote by hand. Each schema is therefore rewritten before it is sent.

Kept: type, description, enum, items, properties, required, nullable, anyOf, minimum, maximum, minItems, maxItems, minLength, maxLength, pattern, and format only for enum/date-time on strings and float/double/int32/int64 on numbers.

Kept means kept as the type Gemini’s Schema declares, not kept as written: description and pattern must be strings, nullable a boolean, minimum/maximum doubles, minItems/maxItems/minLength/maxLength non-negative whole numbers (they are int64 there, so {"minItems":1.5} and {"maxLength":1e30} are dropped), and enum an array of strings, because Schema.enum is repeated string. A value of any other shape is dropped like an unknown keyword — {"type":"integer","enum":[1,2,3]} loses its enum, and so does an enum on a node that is not a string, or on an anyOf node, rather than travelling to Gemini as something it rejects.

Rewritten:

A $ref is matched on its full JSON Pointer, so #/$defs/Foo, #/definitions/Foo and a Foo nested under another node are three different definitions, and an external https://…#/definitions/Foo resolves to none of them (it is elided, since that document is never fetched). Properties and keywords are walked in sorted order, so the same schema always sanitises to the same bytes.

Dropped: $schema, additionalProperties, title, default, examples, $comment, patternProperties, not, if/then/else, unevaluated*, other format values and any keyword not listed above. The dropped names are logged once per tool at debug with provider=gemini.

$ref inlining is depth-capped at 8 and cycle-aware; a recursive or unresolvable reference becomes {"type":"string","description":"(recursive schema elided)"} for a cycle and "(unresolved schema reference elided)" for a definition this document does not contain, which since references are matched on their full pointer is every external one. A node that is not an object, or one nothing survived in, becomes {"type":"object"}, a node that carried no type takes the one its surviving keywords imply (an anyOf node stays untyped, because its branches carry the types) — that inference is a guess, and a lossy one: a node carrying only minimum/maximum becomes number, so a field the caller meant as an integer can come back with a fractional value, and one carrying only minItems becomes string, keeping an array keyword on a scalar. Gemini rejects an untyped node outright, so a guess beats the alternative; give a type to anything whose shape matters. A declaration whose sanitised schema has no properties left is sent without parameters — several model versions reject {"type":"object","properties":{}}. Sanitising never fails; anything Gemini still objects to comes back relayed verbatim.

This is lossy on purpose. A tool that relies on oneOf to discriminate between argument shapes, on additionalProperties: false to forbid extra keys, or on a recursive $ref behaves differently on Gemini than on OpenAI. If a tool depends on those, keep it on a provider whose schema support is wider.

Ollama (native API)

The ollama type talks to Ollama’s own /api/chat (one JSON object, or newline-delimited JSON objects when streaming) and /api/embed endpoints instead of the OpenAI shim, which gives access to Ollama-only options:

• temperature, top_p, max_tokens (→ num_predict) and stop are placed in options.
• Extra request fields are mapped: keep_alive is passed through as-is, num_ctx is added to options, and an options object is merged into options (any Ollama model option, e.g. {"options":{"num_gpu":1,"repeat_penalty":1.1}}).
• response_format json_object sets format: "json"; json_schema sends the schema as format.
• Tools use OpenAI’s {type:"function", function:{…}} shape natively, so tools are forwarded; assistant tool_calls and tool messages (with tool_name) are translated both ways. Ollama sends no call ids, so the gateway generates them (call_…).
• In streams each tool call arrives whole inside one NDJSON line, so it is relayed as one tool_calls delta carrying the index, id, name and arguments together. This is a property of the protocol, not a gap: Ollama genuinely produces the arguments atomically, splitting them into fragments would invent framing that carries no information, and every OpenAI-shaped client accumulates arguments by string concatenation. Calls spread over several lines keep distinct indexes.
• Images are sent to Ollama as inline base64. An image_url that is not a data: URL is downloaded by the gateway first (see Images); with IMAGE_FETCH=false it is rejected with 400.
• Finish reasons: stop → stop, length → length; usage is built from prompt_eval_count / eval_count. Ollama has no prompt cache, so no cache fields are reported and the shipped price table charges ollama models nothing.

Ollama itself is unauthenticated; if you set an api_key it is sent as Authorization: Bearer …, which is useful behind an authenticating reverse proxy.

Ollama usually runs on a private address, which the gateway refuses by default. Add its hostname to the allowlist in .env before creating the connection:

PRIVATE_UPSTREAM_ALLOWLIST=host.docker.internal,ollama   # Docker host, or a Compose service named "ollama"

Then use base_url http://host.docker.internal:11434 (Ollama on the host) or http://ollama:11434 (Ollama as a Compose service). ALLOW_PRIVATE_UPSTREAMS=true allows every private host instead; see Private upstreams.

To use Ollama’s OpenAI-compatible endpoint instead, create a custom_openai connection with base_url ending in /v1 (for example http://host.docker.internal:11434/v1). The native type is preferable for keep_alive, num_ctx and batch embeddings.

Images

A user message may carry OpenAI image parts: {"type":"image_url","image_url":{"url":"…"}}, either an inline data:image/png;base64,… URL or a remote one. Inline images are checked and then passed to every vision-capable provider as they are. Remote ones depend on the provider:

An inline data: URL is held to the same whitelist a downloaded image is, and refused with the gateway’s own 400 when it does not meet it — before anything is sent upstream:

• the URL must carry ;base64, so data:image/png,%89PNG… is refused rather than forwarded as if it were encoded (the data: scheme itself is matched case-insensitively);
• the media type must be one this provider accepts (see the table below), so data:text/html;base64,… no longer travels as an image;
• the payload must decode as standard base64, padding included, and must not be empty — an empty one used to vanish silently, leaving the client a 200 for a message the model never saw an image in.

These used to pass through untouched and came back as a provider 400 that named nothing the caller could act on. The check belongs to the translating adapters (gemini, ollama, anthropic); the OpenAI-compatible types relay the request body verbatim by design, so there the upstream is still the one that reads the part.

The accepted formats are the ones each upstream documents, and they are not the same set:

One shared list would be wrong in both directions — it would refuse the HEIC Gemini takes and promise Anthropic a format its API rejects — so each connection is held to its own, inline and downloaded alike.

The gateway’s own fetch (IMAGE_FETCH, on by default) goes out over the same hardened transport as provider calls but under a stricter policy: an image URL that resolves to a loopback, link-local or private address is refused, and a redirect to another host — or one that drops from https to http on the same host — is not followed.

The difference matters. ALLOW_PRIVATE_UPSTREAMS and PRIVATE_UPSTREAM_ALLOWLIST lift the private-address check for provider calls — allowlisting a host is the documented way to reach a local Ollama — but they do not apply to image fetching. A base_url is typed by an editor; an image URL arrives from whoever holds an API key, so letting it inherit that exemption would turn every allowlisted deployment into a way for a key holder to probe the internal network. On top of that:

• GET only, with Accept: image/*, and only http/https URLs.
• The response’s own Content-Type decides, against the same per-provider list an inline image is held to. The URL’s extension is never consulted. The cache is keyed on the URL alone, so a hit is re-checked against the asking provider’s list rather than trusted.
• IMAGE_FETCH_MAX_MB is enforced on Content-Length and again while reading, so a response that declares nothing (or lies) cannot exceed it.
• Images are fetched one at a time, at most IMAGE_FETCH_MAX_PER_REQUEST per chat request, each within IMAGE_FETCH_TIMEOUT.
• Across requests, IMAGE_FETCH_MAX_CONCURRENT caps the fetches the whole process runs at once and the connections it will hold to one host. A per-request limit bounds one request and nothing at all when many arrive together, which is what would make the gateway a useful amplifier for a target an API key chose. While another project is queueing, no single project takes more than half those slots: capping the process alone closed the outward problem and opened an inward one, where a few requests aimed at a slow image host filled every slot and an unrelated project waited out its whole timeout.
• Takes, not holds. A slot already taken is never reclaimed. A project that filled the gateway while nobody was waiting keeps those downloads running and can sit above its half until they finish; what changes the moment somebody else queues is that it takes nothing further until it is back inside the share. Cutting a running download short to rebalance would throw away a fetch that is already part paid for, which is not what this ceiling is for. So the share is a floor under everybody else rather than a ceiling on one tenant: with nobody else waiting, a single project reaches the whole of IMAGE_FETCH_MAX_CONCURRENT, which is what the setting says it is.
• A fetch that cannot get a slot within two seconds (or IMAGE_FETCH_TIMEOUT, if that is shorter — the wait is not separately configurable) is answered 429 with Retry-After and code: "image_fetch_saturated". The gateway ran out, which is a resource limit rather than the client getting anything wrong, and deliberately not a 5xx, which would report a fault and count against availability when nothing is broken. Retry-After names IMAGE_FETCH_TIMEOUT, not the queue wait: a slot frees when a download finishes, so naming the wait sent clients that honour the header — the official SDKs do, for a 429 as much as for a 5xx — straight back into the same full queue. The queue wait is its own budget, separate from IMAGE_FETCH_TIMEOUT, which starts once a slot is in hand; otherwise the same saturation came back sometimes as a 429 and sometimes as a download timeout blaming the image host.

Anything the client got wrong — a bad scheme, a non-image response, an oversized image, a non-2xx from the image host, too many images — is a 400 invalid_request_error naming only the host of the offending URL, never the full URL (which may carry a signed query string). A non-2xx is deliberately a 400 and not a 502: the client chose the host.

Fetched images are cached in memory (IMAGE_CACHE_ENTRIES, IMAGE_CACHE_MAX_MB, IMAGE_CACHE_TTL) because a multi-turn conversation resends the same image part on every turn. The cache is keyed on the URL alone and is not written to disk, so an image whose content changes within the TTL keeps serving the old bytes until it expires; a response carrying Cache-Control: no-store is never cached.

Prompt caching

Providers cache a repeated prompt prefix and charge less for the cached part. Ragmux passes a client’s caching intent through where the provider has a field for it, and always reports back what the provider said it cached. Both sides feed the cost estimate shown on request logs.

Response side: one accounting for every provider

usage gains OpenAI’s breakdown objects, so the same fields mean the same thing on every connection:

"usage": {"prompt_tokens": 250, "completion_tokens": 500, "total_tokens": 750,
          "prompt_tokens_details": {"cached_tokens": 200, "cache_creation_tokens": 40},
          "completion_tokens_details": {"reasoning_tokens": 30}}

prompt_tokens always includes the cached and freshly written parts: the mapping guarantees that. prompt_tokens + completion_tokens == total_tokens is weaker — it is how each adapter assembles the block from an upstream that reports consistently, and it is not checked, so a Gemini tool round (see Tool calling) or any upstream reporting a total that does not add up breaks it; the block is relayed as it came. cache_creation_tokens has no OpenAI equivalent — OpenAI does not bill cache writes, Anthropic does.

Change from 0.3.x: for a cached Anthropic request, prompt_tokens is now larger. Earlier releases reported Anthropic’s input_tokens verbatim and so under-reported a cached prompt by the size of its cached prefix. Recorded token budgets and metrics for such requests move up accordingly.

Request side: cache_control

Anthropic takes a cache_control marker in three places, and Ragmux relays each of them verbatim (the gateway never inspects or invents one):

With no cache_control anywhere, system stays the plain joined string it has always been — a request that does not ask for caching is byte-identical to a 0.3.x one. When a message has several marked parts the last marker wins, since Anthropic caches the prefix up to and including the marked block.

RAG context drops the system marker. When a project’s RAG store injects passages, the context block is prepended to the first system/developer message (in front of the project system_prompt, which is itself prepended to the client’s own system text — see Context injection), and the cache_control on that message’s parts is removed. The block changes with every query and comes first, so a breakpoint behind it would make Anthropic write a new cache entry (billed above the plain input rate) on each request and never read one back. The parts themselves and every other field are kept; markers on other messages, user content and tools are untouched, and a project system_prompt on its own does not drop anything. Placing the context after the client’s cached prefix is planned for v0.6.

OpenAI, DeepSeek and custom_openai: nothing is sent. Their caching is automatic and server-side; there is no request field to emit, so Ragmux emits none. A cache_control a client embeds rides along untouched (message content is relayed raw) and OpenAI ignores it. Only the response side matters here — this is not a gap.

Gemini and Ollama: response side only. Gemini’s explicit caching needs a stateful cachedContents resource created and referenced across requests, which does not fit a stateless passthrough; its implicit caching happens automatically and is reported in cachedContentTokenCount. Ollama has no prompt cache at all.

The project system prompt and the RAG context block the gateway injects are the stable prefix most worth a breakpoint, but the gateway does not mark one: a project-level cache_prompt switch needs a column and dashboard work of its own and is not in 0.4.

Model field echo

A project without chat routes ignores the client’s model for selection: the project’s connection decides. The gateway replaces the value with the connection’s model name before calling the provider and echoes the client’s original value back in the response (and in every stream chunk) so SDKs that compare the field stay happy. When the client sends an empty model, the connection’s model name is returned instead. GET /v1/models lists that model name with owned_by set to the provider type.

Routing

A project can define named routes (see Routes). Each route has one to eight targets: a connection, an optional model override, a weight and a priority. Once a project has at least one chat route, the client’s model selects what runs:

1. A route name matches exactly: the route’s targets are ordered (priority groups ascending, weighted random inside a group) and the first one is called.
2. Otherwise the default connection’s model name selects the implicit route: the project’s own connection, as before.
3. An empty model also selects the implicit route.
4. Anything else is 404 model_not_found.

The target’s model replaces the connection’s model name for the call; the provider, pricing, request log and trace all see that effective model. The response model still echoes the client’s value (for an empty model, the connection’s model name). In request_logs, model_name is the effective model, requested_model is the client’s value and route is the route name, empty for the implicit route. Routes of kind embeddings or rerank do not switch chat routing on.

The project page has a route editor for all of this: a route’s kind (chat, embeddings or rerank) and up to eight targets, each with its connection, optional model, weight and priority.

Before your first chat route

Without chat routes the gateway answers whatever model string a client sends (it echoes it back). The first chat route ends that: any string that is neither a route name nor the default connection’s model name becomes 404 model_not_found. Check what your clients actually send before you create it:

SELECT requested_model, count(*) FROM request_logs
WHERE project_id = <project id> GROUP BY requested_model ORDER BY count(*) DESC;

Create a route for every value that matters, or point those clients at the default connection’s model name. Rolling back is deleting the route: with no chat routes left the project echoes any model again.

Retry and fallback

A chat call that fails before the first byte reached the client can be repeated on the same target (retry) or handed to the route’s next target (fallback). The order of targets is the one Routing describes; a project without routes has one target, so only retries apply. What happens is decided by the kind of failure:

A reset after the request was sent is never retried on the same target: the provider may already have processed it, and a repeat could be billed twice. A 529 from a provider other than Anthropic is not retried either; it only hands the request to the next target. A failure to download an image URL the client sent is the client’s, not the target’s: it is answered as it is, with neither retry nor fallback.

Retries. Each target is retried up to UPSTREAM_MAX_RETRIES times (default 2, 0 turns them off). The wait before a retry is exponential with full jitter: a random time below 250 ms, 500 ms, 1 s and so on, capped at 2 s. When the upstream sent Retry-After of 2 s or less, the longer of the two is waited. A longer Retry-After is not waited for: the next target takes over, and when there is none the client gets the error with Retry-After set to the upstream’s value.

Budget. UPSTREAM_TIMEOUT is the budget of the whole request, not of one attempt. Each attempt runs with what is left of it, and a wait that would use up the remainder is skipped, so the attempts of a non-streaming request never add up to more than UPSTREAM_TIMEOUT. A streaming attempt is held to what is left of the budget only until its first chunk arrives; once the stream is flowing UPSTREAM_TIMEOUT no longer limits it, and STREAM_MAX_DURATION and STREAM_MAX_BYTES_MB do. The rate-limit check, the system prompt and retrieval run once before the first attempt, and usage is recorded once.

Streams. Retry and fallback apply only until the first chunk is sent, including an upstream that answers 2xx and fails before it sends anything. After the response headers went out the stream is never restarted or joined to another one: a cut stream ends the way it did before.

Circuit breaker. A connection that keeps failing is taken out of the candidate list for a while, so retries and fallbacks do not spend the budget on a dead target. Every connection has its own breaker, held in memory by each replica; it is never written to the database, so replicas do not share it. A breaker has three states:

• closed: calls pass and consecutive failures are counted. A success resets the count.
• open: reached after BREAKER_FAILURE_THRESHOLD failures in a row (default 5, 0 turns breaking off). The connection is skipped for BREAKER_OPEN_DURATION (default 30s), and a target that opens in the middle of a request is not retried any more.
• half-open: once the duration has passed, one call goes through as a probe. If it succeeds the breaker closes; if it fails the breaker opens again for a full duration. Other requests keep skipping the connection while the probe is in flight, and a probe that ends in a way that says nothing about the upstream, such as a client that went away, frees the slot for the next request.

What counts as a failure is what the table above calls an upstream fault: the request never left (DNS, refused connection, TLS handshake), a reset after it was sent, 429, 503 and 529, other 5xx, and 504 or a timeout. 401, 403 and 404 do not count, although they still hand the request to the next target, and neither do other 4xx or a client that went away. Such an attempt leaves the count where it was. The same holds for a stream: a client that leaves before or during it, or an upstream that fails after its first byte (the client already got a 200), is neither a success nor a failure, so it neither resets the count nor closes a half-open breaker.

An open connection is skipped when the route’s targets are walked, so a request goes straight to the next healthy one. When every target of a route is open, the first one is called exactly once, without retries; there is no immediate 503, so a recovered upstream is noticed by the first request that reaches it. A project without routes has one target and behaves as before. The state is per replica, so a fleet of N replicas sends up to N probes to a bad upstream, and a restart starts every breaker closed. Opening and closing are logged at warn and info level with the connection name, and ragmux_upstream_breaker_state reports the state per connection (Observability).

What the client sees. The error of the last attempt, in the format of Error relay and redaction. A fallback that succeeds is invisible apart from the request log: one row per request, attempt is the number of upstream calls in total, fallback_from is the connection id of the first target whenever the request moved past it (also when no target answered), and model_name is the model of the last target tried. A target whose adapter cannot be built is skipped without counting as a call; on a route with several targets that includes the first one. Each failed attempt is logged at warn level with its connection id and class. Retries and fallbacks are counted in ragmux_upstream_retries_total and ragmux_upstream_fallbacks_total (Observability); a failure that is neither retried nor handed on, such as a 400, reaches neither.

Error relay and redaction

Provider errors are relayed with the upstream message (cut to 512 characters) in the OpenAI error envelope. A 4xx keeps its status, and so do 503 and 504, which tell the client to come back later; Anthropic’s 529 overloaded means the same and becomes 503. Every other upstream 5xx becomes 502. The upstream’s Retry-After (delay seconds or an HTTP date) is passed on unchanged; a value in neither form is dropped. The gateway’s own 429s set their own Retry-After instead. Transport failures are never relayed verbatim: they become 502 upstream_error with one of upstream unreachable, upstream TLS handshake failed, upstream returned a non-HTTP response, upstream redirect rejected: …, the private-address message from Private upstreams, or upstream request failed; timeouts become 504 timeout / upstream request timed out. The raw cause is logged at warn level with the upstream host. Streaming responses end with an error after STREAM_MAX_DURATION or STREAM_MAX_BYTES_MB.

Before an upstream message reaches a client, the request log or the gateway’s own log, the connection’s own API key and anything matching Bearer …, Basic …, a JWT (eyJ…), sk-…, sk-ant-…, AIza…, ya29.…, gsk_…, hf_…, xai-… or credentials embedded in a URL (https://user:pass@host) is replaced by [redacted]. This covers Anthropic stream error events and Ollama error bodies as well.