Putting Ollama or vLLM behind a gateway

Ollama and vLLM both let a model run on hardware you already control instead of a provider’s — the difference between them is mostly which API they expose. Ollama has its own native protocol with a few options no OpenAI-shaped API has room for; vLLM speaks OpenAI’s schema directly. Ragmux connects to either, but the two paths aren’t identical, and there’s a network-security step neither one skips.

The step every local server needs: the allowlist

A base_url field is filled in by whoever can edit model connections, and Ragmux treats that as an untrusted destination, not a trusted config value. By default, a connection whose host resolves to a loopback, link-local, multicast or private address — the 10/8, 172.16/12, 192.168/16 ranges a local Ollama or vLLM server almost always lives on — is refused outright, with the gateway resolving the hostname itself and dialing the checked address so a DNS answer can’t change between the check and the actual connection. A rejected save reports exactly why: upstream host "…" resolves to a private or local address; set ALLOW_PRIVATE_UPSTREAMS=true or add it to PRIVATE_UPSTREAM_ALLOWLIST.

Getting past that for a real deployment means one of two things in .env:

• PRIVATE_UPSTREAM_ALLOWLIST=host.docker.internal,ollama names the specific hostnames allowed to resolve privately — the Docker host, or a vllm/ollama service in the same Compose network. IP literals like 127.0.0.1 aren’t matched; it has to be a hostname.
• ALLOW_PRIVATE_UPSTREAMS=true disables the check for every connection, which only makes sense when everyone who can edit a model connection is also someone who may reach every host the gateway can — a much wider grant than allowlisting one hostname.

The same policy applies to the connection test endpoint, so a ping against an unallowlisted host fails the same way a real request would, rather than misleadingly succeeding.

Ollama’s native API, and what it gets you

provider_type: ollama talks to /api/chat and /api/embed directly rather than through an OpenAI-compatible shim, which is what makes a few Ollama-only options available: keep_alive (how long the model stays loaded in memory after a request) is passed straight through, num_ctx and any other Ollama model option — num_gpu, repeat_penalty, and so on — land in an options object the native endpoint expects. temperature, top_p, max_tokens and stop are translated into that same options object automatically, so ordinary OpenAI-shaped calls keep working unchanged; the extra fields are there for callers who want them, not a requirement for callers who don’t.

Streaming tool calls behave differently here than with a token-by-token provider, and it’s a property of the protocol rather than a limitation: Ollama emits a tool call’s arguments whole, inside one NDJSON line, so Ragmux relays it as one complete tool_calls delta rather than splitting it into fragments that would carry no real information. Ollama also has no concept of a tool call id, so the gateway generates one; several calls spread across separate lines each keep their own index. Images that aren’t already data: URLs are downloaded by the gateway and sent to Ollama as inline base64, subject to the same IMAGE_FETCH setting every other provider respects. Because Ollama has no metered billing, usage is still reported — built from prompt_eval_count and eval_count — but the shipped price table charges these requests nothing.

vLLM, and when to use custom_openai instead

vLLM doesn’t have a native protocol the way Ollama does — it speaks OpenAI’s chat and embeddings schema directly, so it’s connected as provider_type: custom_openai with a base_url ending in /v1 (http://vllm:8000/v1 for a Compose service, or http://host.docker.internal:8000/v1 on the Docker host). Unlike openai or deepseek, custom_openai uses the URL exactly as given rather than appending a suffix, so the trailing /v1 has to be there. An api_key is optional for custom_openai, matching that vLLM and most local OpenAI-compatible servers don’t require one by default.

Ollama itself can also be reached this way — its own OpenAI-compatible endpoint exists specifically for clients that expect that shape. The trade-off is what you’d expect: the native ollama type is preferable whenever keep_alive, num_ctx or batch embeddings matter, since none of those have an OpenAI-shaped equivalent to travel through; the custom_openai path is the simpler choice when a deployment already treats every provider uniformly and doesn’t need Ollama-specific controls.

Local embeddings for retrieval

Both paths support embeddings, not just chat, which matters for anyone building private document Q&A that shouldn’t leave the building. Ollama’s native /api/embed is used directly by an ollama connection; custom_openai uses whichever OpenAI-shaped embeddings route the server exposes, which vLLM provides for embedding-capable models. Either way, a document store’s embedding connection is fixed the moment it processes its first chunks — the same rule that applies to a hosted provider applies here too — so the choice between a local embedding model and a hosted one is worth making deliberately before documents are uploaded, not after.

Running both the chat model and the embedding model locally, with PRIVATE_UPSTREAM_ALLOWLIST scoped to exactly the hosts running them, is what makes a fully local retrieval setup possible: a query never leaves the machines you control, from the moment it’s embedded to the moment an answer comes back. That’s a stronger claim than “the chat model is local” — plenty of setups run inference locally but still call out to a hosted embedding API without noticing that’s the one request still leaving the network.

After it’s connected

A project backed by a local Ollama or vLLM connection is otherwise identical to one backed by a hosted provider: rate limits, token budgets, the audit log and request log all key off the project. That symmetry is what makes an air-gapped deployment practical in the first place — the same gateway, the same /v1/chat/completions contract, whether the model behind it is a hosted API or a server on hardware that never reaches the public internet at all.

Full reference: /docs/providers/#ollama-native-api.