Ragmux as an OpenAI-compatible API gateway

Most of what makes Ragmux useful for an OpenAI-compatible provider isn’t translation — openai, deepseek and custom_openai connections already speak the shape Ragmux exposes, so there’s nothing to rewrite. What a gateway adds instead is everything around the request: one endpoint regardless of which of those three actually answers, budgets and rate limits enforced before the provider is ever called, and a passthrough contract precise enough that provider-specific parameters keep working without Ragmux having to know about each one by name.

Adding the connection

POST /admin/api/models creates it. provider_type: openai and provider_type: deepseek default to their own hosted APIs and only need an api_key and a model_name; provider_type: custom_openai is the general case — vLLM, LM Studio, LiteLLM’s own proxy, text-generation-inference’s OpenAI route, Ollama’s /v1 shim, or literally anything else speaking the Chat Completions format — and requires an explicit base_url, used exactly as given rather than having a suffix appended the way openai and deepseek get, so the /v1 has to be included yourself (http://vllm:8000/v1, for instance). A host on a private or local network — a Compose service name like vllm, localhost, the Docker host — is refused by default; add it to PRIVATE_UPSTREAM_ALLOWLIST, or set ALLOW_PRIVATE_UPSTREAMS=true, before the connection will save. See Private upstreams and the Ollama & vLLM gateway page for the full mechanics. An api_key is optional for custom_openai, matching that most self-hosted servers don’t require one. POST /admin/api/models/{id}/test with {"mode": "chat"} sends one bounded ping — the model is asked to reply with the word pong, capped at 256 tokens — and records last_test_ok, last_test_latency_ms and, on failure, a redacted last_test_error on the connection itself, so a broken key or an unreachable server shows up on the connection’s own status rather than in front of a real caller.

What gets forwarded exactly as sent

/v1/chat/completions only actually parses the fields it needs to route and bill a request — model, messages, stream, temperature, top_p, max_tokens (or max_completion_tokens), stop, n, tools, tool_choice, response_format, stream_options, user. Every other top-level field in the request body is kept verbatim and, for an OpenAI-compatible connection, sent upstream unchanged. That’s what lets a seed, a logprobs flag, an OpenAI reasoning_effort value, or vLLM’s guided_json keep working through Ragmux with no gateway-side support for that specific field — the passthrough is the feature, not a gap waiting to be closed. The same mechanism carries one gateway-specific addition the other direction: an optional top-level ragmux object is read and consumed by the gateway itself, never forwarded upstream, and {"ragmux": {"include_context": true}} is what asks a RAG-backed project to return the retrieved sources on a non-streaming response.

Three fields are the exception, because the gateway itself needs to control them: model is handled by the gateway and replaced with the effective model name before the provider is called. On a project without chat routes the project’s connection decides which model answers, and whatever value you send is only echoed back in the response. Once the project has chat routes, model selects one: a route’s exact name, the default connection’s model name, or an empty value; anything else is a 404 with code model_not_found. The response still carries the value you sent. stream reflects what the client actually requested. And stream_options is rewritten to {"include_usage": true} against the upstream on every streamed request, regardless of what the client sent, because the request log, budgets and cost estimate all need token counts — but see below for what that means for what the client itself receives back.

The streaming usage trailer is opt-in, on every provider

Ragmux always asks its upstream for usage on a streamed response, but it only relays the final trailer chunk — one data: line carrying an empty choices array and a usage object, just before data: [DONE] — to a client whose own request set stream_options: {"include_usage": true}. This isn’t specific to OpenAI-compatible connections; the same rule holds for Anthropic, Gemini and Ollama, whose adapters all produce that trailer internally regardless. Leave stream_options off, and a stream ends at [DONE] with no trailing empty-choices frame to trip up code that indexes choices[0] on every chunk without checking its length first. Ask for it, and the last chunk before [DONE] is exactly what OpenAI’s own streaming usage feature would send.

That said, a choices: [] frame isn’t always Ragmux’s own trailer — some upstreams emit frames of their own with an empty array, such as Azure’s content-filter chunk. Those pass through untouched rather than being dropped, since they carry information the gateway doesn’t own the right to discard. Code that guards array access before indexing handles both cases the same way.

Errors: the provider’s 4xx, a mapped 5xx, or a specific Ragmux code

A malformed request — bad JSON, a missing messages field, or no user message for an Anthropic or Gemini connection — is a 400 before anything reaches a provider, as is an image URL that a Gemini connection can’t inline. Tools are not a source of such errors: every chat provider type supports them, and the Gemini adapter sanitizes a tool schema and logs the keywords it drops rather than refusing it. Budget and rate-limit rejections are 429 with a code naming which tier and which limit (rate_limit_rpm, budget_daily, and so on) so a client can distinguish “slow down” from “this month’s ceiling is spent” programmatically instead of parsing a message string. A provider’s 4xx is relayed with its own status and message, credentials redacted. Upstream 5xx responses are mapped instead: 503 and 504 pass through, Anthropic’s 529 becomes 503, and every other 5xx becomes 502, so a client can tell “the upstream is overloaded or slow, retry” from “the upstream broke”.

GET /v1/models lists what the project can actually answer

An OpenAI SDK that lists available models before choosing one still works against Ragmux, but it’s worth knowing what it returns: {"object": "list", "data": [...]} in OpenAI’s own shape, describing what the calling project can reach, not the provider’s full catalogue.

A project without chat routes lists its single connection’s model, and {id} in GET /v1/models/{id} is ignored, because there’s only one answer it could give. A project with chat routes lists each chat route’s name, plus the default connection’s model name unless a route already uses it; with more than one entry, {id} has to match one of them or the request is a 404. A sk-user-… key that spans several projects and names none lists one entry per granted project’s connection instead, deduplicated by model name. Both endpoints need the models scope on a sk-user-… key; the project’s sk-proj-… key carries no scopes and reaches them without one.

This follows from how selection works: Ragmux doesn’t send a request to whichever model the client names, so listing every model the upstream provider offers would describe capabilities the project can’t reach. A client that wants an end user to pick between gpt-4o and gpt-4o-mini defines one route per choice, each with its own target, and the listing then shows those route names.

Swapping the provider behind an existing integration

Because the client only ever holds a project key and calls /v1/chat/completions, moving a project from openai to custom_openai — pointing an existing integration at a self-hosted vLLM server instead of a hosted API, say — is a change to the model connection, not to any application code. The same is true in the other direction: a project can start on a local server during development and move to a hosted provider before launch without either environment’s calling code ever mentioning which one is active. That’s also what makes tracking spend per project hold up across the swap — the request log, budgets and cost estimate all key off the project, not the provider answering it, so a mid-life provider change doesn’t fragment the history into two things that need reconciling by hand.

Full reference: /docs/api/.