What is an AI gateway?

An AI gateway is a service that sits between your applications and the model providers you actually call — OpenAI, Anthropic, Gemini, a local Ollama server, or anything else that speaks (or can be made to speak) a chat-completions style API. Instead of every application holding its own provider credentials and handling that provider’s specific request and response shape, it talks to one gateway, in one API, and the gateway does the rest.

The problem it solves

Skip the gateway, and “call three model providers, with retrieval” tends to look like this in practice, not in theory. Three SDKs get integrated separately, each with its own request and response shape a developer has to learn once per provider. Three sets of API keys end up wherever the service that happens to call that provider stores its secrets — an environment variable here, a secrets manager entry there — with no single place that holds all of them, and no way to rotate one without a deploy of whichever services embedded it. Swapping one model for another, or trying a cheaper one for a single feature, means changing the model name and adjusting for that provider’s specific request shape everywhere the call happens, not in one place.

A retrieval step, if there is one, gets bolted onto whichever call site needs it, usually built slightly differently each time because there’s no shared implementation to reach for. And when someone asks “how much did the support bot spend last month, separately from the internal tool that happens to share the same API key,” the honest answer is often that nobody can say — the provider’s own billing dashboard knows what was spent in total, not which of an organization’s applications spent it.

A gateway exists to turn each of those from a repeated, per-integration decision into a property of the infrastructure instead: one place credentials live and get rotated, one schema every application codes against regardless of which provider actually answers, one place a model can be swapped without touching the code that calls it, and one place every request is logged with enough detail to answer “who spent what” after the fact instead of guessing.

What a gateway typically adds

The pattern shows up under different names — LLM gateway, model proxy, AI router — but the recurring pieces are the same:

• One API surface. Applications integrate once, against a schema the gateway guarantees, instead of once per provider.
• Centralized credentials. Provider API keys live in the gateway, not scattered across every service that happens to call a model.
• Routing and translation. A request comes in against one schema and is adapted to whichever provider is actually configured behind it — including the parts that don’t map cleanly, like tool/function calls.
• Control. Rate limits, spend budgets, audit logging and access control enforced in one place instead of reimplemented per integration.

Not every gateway does all four. Some are routing-only; some add caching; some add observability and stop there.

What a gateway is not

“AI gateway” gets applied loosely enough that it’s worth saying what the term excludes, not only what it includes.

A reverse proxy forwards bytes from one address to another, unaware of what’s inside the request it’s forwarding. Point one at a model provider, and every application behind it still has to speak that provider’s exact schema, still needs its own credentials for that provider, and the proxy has no opinion on whether a given request should be allowed to run at all — no per-application budget, no swapping the model without touching client code, none of that comes free from routing packets. A gateway does the opposite of unaware: it parses the request specifically in order to translate it, and it decides whether the request may proceed before forwarding it anywhere.

An SDK wrapper — a client-side library that presents several providers behind one function call — solves the schema problem from the calling application’s side instead of the infrastructure’s. Every application that imports it still holds every provider’s credentials itself, wherever that application already keeps its secrets, and there’s no shared place to enforce a rate limit or a budget across applications that don’t share a process — each import is its own copy, with its own view of how much has been spent. A gateway centralizes exactly what an SDK wrapper leaves distributed across every application that imports it.

A model router — something that picks the best or cheapest model for a request among several candidates — solves a real but different problem: which model answers, not where credentials live or what happens to a request on the way there. Routing logic can sit on top of a gateway, or be the whole of what a narrower tool does; a gateway with no routing logic at all, just one fixed connection per application, is still doing the schema-translation, credential-centralizing and budget-enforcing job the term “AI gateway” is usually used for. Ragmux, for instance, routes only among the targets an operator lists on a named route, by weight and priority, with fallback to the next target on a failure — it does not pick the best or cheapest model among candidates on its own, and which candidates exist is an explicit decision of the operator.

What actually happens to a request

Strip away the branding, and a request passing through an AI gateway goes through the same handful of stages, whichever gateway it actually is:

1. Authenticate the caller and resolve what it’s allowed to do. The key on the request maps to an application-level identity — a project, in Ragmux’s case — that carries its own model connection, its own budgets, and whichever document store, if any, it’s attached to.
2. Check limits before doing anything expensive. Rate limits and token budgets are checked against that identity before the request goes any further. Ragmux’s own docs are explicit that this happens before retrieval runs specifically so a throttled request doesn’t still cost an embedding call on the way to being rejected.
3. Augment the request, if the gateway does that. A gateway with retrieval built in embeds the query, searches a document store and injects the matched passages at this stage, before anything is sent to a provider — see Where retrieval fits below.
4. Translate to the provider’s own schema. Field names, required parameters (Anthropic requires max_tokens where OpenAI doesn’t), tool- and function-call shapes and streaming formats all differ by provider; this is the step that makes “one schema in, whichever provider out” actually true rather than aspirational.
5. Forward the request and relay the response, normalizing it back to the schema the caller expects — including turning a provider-specific streaming format back into the one shape the caller’s own SDK already knows how to read.
6. Record what happened. Tokens used, latency, whether the response streamed, whether retrieval ran and what it found, and — if the gateway estimates cost — what that single request is reckoned to have cost, all attributed to the identity resolved in step 1.

None of these six steps is optional in the sense that skipping it silently breaks something; a gateway that only does steps 1, 4 and 5 is a legitimate, narrower kind of gateway — see “What a gateway typically adds” above — it simply isn’t doing the budget-enforcement or retrieval jobs some others do.

Where retrieval fits

A narrower slice of gateways — Ragmux among them — treat retrieval-augmented generation (RAG) as part of the request path rather than a separate system an application has to call first. A project can be attached to a document store; when a request comes in for that project, the gateway embeds the query, searches the store and injects the matched passages into the system prompt before the request goes to the provider. The calling application does not orchestrate a separate retrieval step — it sends one request and gets back an answer that is already grounded in the documents it was given.

That convenience isn’t free. Putting retrieval in the gateway means the gateway now depends on wherever the documents and their embeddings live — for Ragmux that’s the same PostgreSQL database as everything else, so there’s no second system to stand up, but there is a database to operate that a routing-only gateway wouldn’t need at all. It also turns the embedding model into a permanent choice for a given store: Ragmux fixes a document store’s embedding connection the moment it has its first chunks, and rejects any later attempt to edit that connection outright. Reprocessing re-embeds a store’s existing documents with the same connection — it’s how a changed chunk_size gets picked up, not a way to switch embedding models; moving to a different one means a new store, re-uploaded documents, and every project that used the old one repointed at the new one. A team that wants routing without the database dependency or that commitment should reach for a narrower, retrieval-free gateway instead — see Self-hosted AI gateway comparison for what that trade looks like across several projects.

What Ragmux specifically is

Ragmux is a single, self-hosted Go binary that is both of the things above at once:

• It exposes /v1/chat/completions, /v1/models and /v1/embeddings in the OpenAI-compatible schema — JSON and SSE streaming alike, on the chat endpoint — plus /v1/retrieve and /v1/rerank, and translates Anthropic and Gemini requests and streams to and from that schema, tool calls included, so the official OpenAI SDKs work against it unchanged.
• It can attach a project to a document store: PDFs, DOCX, HTML, Markdown and plain text are parsed, chunked, embedded and indexed with pgvector, then searched with a hybrid of vector similarity and Postgres full-text ranking whenever that project is called.
• Everything it needs to do both — users, model connections, projects, documents and their vector embeddings, and the request log — lives in one PostgreSQL database. There is no separate vector database, cache or queue to run alongside it.
• Provider credentials are AES-256-GCM encrypted at rest and never leave the gateway; each project’s key, rate limits, token budgets and access are enforced before a request reaches a provider, and every login and management action is recorded to an audit log.

What Ragmux is not: a model. It doesn’t train or host inference itself — it routes to and grounds requests against the providers you already have accounts or local servers for, including your own vLLM, LM Studio or LiteLLM deployment through the custom_openai provider type.

When running your own gateway is the wrong choice

None of this is free to operate, and a self-hosted gateway isn’t the right answer for every setup. It’s probably the wrong choice when:

• Exactly one application calls exactly one provider. With one service and one model, a gateway centralizes credentials that were never actually scattered and translates a schema that was never actually going to change out from under the calling code — the provider’s own SDK already does that job, with nothing extra to run.
• Nobody actually owns keeping it running. A gateway is a service with its own uptime; Ragmux specifically needs a PostgreSQL database kept backed up and a SECRET_KEY kept safe, real operational surface rather than a config file dropped into an existing app. A team of one shipping a side project may reasonably decide that surface isn’t worth taking on yet.
• The provider already gives you everything a gateway would add. Some providers ship their own spend controls, multi-model access and usage dashboards directly; if the entire reason to add a gateway was one of those, and the provider already covers it, the gateway would be solving an already-solved problem.

The case tends to get stronger the moment any of that changes: a second application, a second provider, or a real need to answer “what did this actually cost, broken down by project” — each one is an integration that either gets its own hand-rolled credential handling and cost tracking, or gets centralized once. See Tracking LLM spend per project for what the second option looks like in practice.