Migrating from LiteLLM to Ragmux

Moving off LiteLLM is a config-translation problem more than a code-rewrite one, provided the calling application already talks to an OpenAI-compatible /chat/completions endpoint — which LiteLLM’s own proxy also exposes, so most applications already do. What changes is where a deployment, a key and a budget live: LiteLLM keeps them in a YAML file and a database; Ragmux keeps them as model connections and projects reachable through its own admin API. This walks through that translation piece by piece, including the parts that don’t map cleanly, so a migration doesn’t quietly drop a limit or a fallback nobody meant to remove. For the “should I move at all” question rather than the “how,” see Ragmux vs LiteLLM — this page assumes that decision is already made.

What a LiteLLM config actually holds

A LiteLLM Proxy deployment is driven by a config.yaml built around one model_list, each entry pairing a model_name — the alias callers request — with litellm_params naming the real provider and model, plus credentials:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-team
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 100

The os.environ/VAR syntax reads a credential from the proxy’s own environment rather than storing it in the file, and the proxy is started against that file directly — litellm --config /path/to/config.yaml, listening on http://0.0.0.0:4000 by default. Two entries can share one model_name, which is how LiteLLM does load balancing and fallback: a request for that alias is routed across every deployment registered under it, by routing_strategy, and rpm/tpm on each entry cap what that specific deployment can take before the router moves on to the next.

The mapping, piece by piece

One litellm_params entry becomes one model connection. POST /admin/api/models takes a provider_type (openai, anthropic, gemini, deepseek, ollama or custom_openai), an api_key, and a model_name — the same three pieces of information litellm_params already carries, just under Ragmux’s own field names. A deployment routed through LiteLLM itself, or any other server speaking the Chat Completions format, becomes provider_type: custom_openai with an explicit base_url — used exactly as given, so the /v1 suffix has to be included yourself; everything hosted directly by a provider Ragmux has a native adapter for skips the base_url field entirely. A LiteLLM proxy is typically reached at a Compose service name, localhost, or another private-network address, and Ragmux refuses those by default — add it to PRIVATE_UPSTREAM_ALLOWLIST, or set ALLOW_PRIVATE_UPSTREAMS=true, before the connection saves. See Private upstreams. The os.environ/VAR indirection has no equivalent — a connection’s api_key is stored, encrypted, on the connection itself, not read from the gateway process’s environment at request time — so a migration script reading the same environment variables LiteLLM used and passing their values straight into the API call is the direct replacement.

One model_name alias becomes one project, not one connection, because a project is also where Ragmux’s own limits and retrieval attach. POST /admin/api/projects with model_connection_id pointing at the connection from the previous step creates it, and the response’s api_key — an sk-proj-… value — is shown exactly once; save it before moving on, the same discipline a LiteLLM virtual key demands. If several LiteLLM deployments shared one alias for load balancing or fallback, the Ragmux equivalent is a route inside that project: one connection per deployment, each added as a target of the route, and the alias becomes the route’s name so the client’s model value stays the same. See Routing.

Limits translate, but the unit changes. LiteLLM’s rpm/tpm on a deployment and its dollar-denominated key and user budgets both express a ceiling; Ragmux’s rate_limit_rpm, rate_limit_tpm, budget_daily_tokens and budget_monthly_tokens on the project body express the same idea in requests and tokens, with 0 meaning unlimited. A requests-per-minute or tokens-per-minute figure carries over as-is. A dollar budget has to be converted using the provider’s own per-token price for that model, and re-derived if that price changes — Ragmux has no dollar-denominated budget field to set directly.

Routing strategy maps only in part. LiteLLM’s routing_strategy (least-busy, latency-based, and similar) spreads a request across whichever deployment under an alias looks healthiest at that moment. A Ragmux route orders its targets by priority (groups ascending, weighted random by weight inside a group) and moves to the next target on a failure that allows fallback (a 500, a reset connection or a 401 hands over without being retried; see Retry and fallback), so weighted splits and ordered fallback carry over, while strategies that measure latency or load per request do not.

What’s genuinely different underneath

LiteLLM’s own README describes it as giving “a single, unified interface to call 100+ LLM providers… using the OpenAI format” and ships as both a Python SDK (uv add litellm) and the separately-run Proxy (uv tool install 'litellm[proxy]', then litellm --model gpt-4o for a quick single-model start). That breadth is the thing a migration can’t recreate: Ragmux ships six provider types including custom_openai, which covers whatever an OpenAI-compatible server exposes, so a model_list entry naming a provider Ragmux has no adapter for has no destination to migrate to — it either stays on LiteLLM, moves behind a custom_openai connection if that provider exposes an OpenAI-compatible endpoint of its own, or needs a provider Ragmux doesn’t support yet. Licensing is the other asymmetry worth naming plainly during a migration decision: LiteLLM’s core SDK and proxy are MIT, but code inside its enterprise/ directory carries a separate commercial license required for production use — the license text doesn’t enumerate which features that covers, so treat it as a licensing question to raise with LiteLLM directly rather than a specific feature list — while Ragmux is AGPL-3.0-or-later throughout, with no gated tier to hit later.

What a migration gains, rather than loses, is retrieval as part of the same request path. A LiteLLM deployment doing retrieval-augmented generation today is calling a separate vector database and injecting context itself, before the request ever reaches the proxy. Attach a rag_store_id to a Ragmux project instead, and that same request gets its query embedded, searched and the matched passages injected automatically — one less system a migrated application has to keep operating.

Teams, keys and the database underneath

LiteLLM’s proxy separates a master key from per-user and per-team virtual keys, with budgets and rate limits attachable at any of those levels, and it can run without a database at all in a simpler mode that shares one key across every caller — a database becomes required specifically once virtual keys, spend tracking or budgets enter the picture. Ragmux’s access model maps onto that only partly: a project’s member_user_ids is its team, PUT /admin/api/projects/{id}/members is how that set changes later, and every member who is also an editor can rotate the project’s key or change its limits — there’s no separate concept of a master key above the project level, because there’s no shared key meant to span every project the way a LiteLLM master key spans every team. And Ragmux needs PostgreSQL unconditionally, for a single evaluation instance as much as for production, so a LiteLLM deployment that had been running database-free with a shared key needs a Postgres instance provisioned as part of the move, not as an optional upgrade taken later.

That database is also where the two systems diverge on what a migration inherits versus starts over. LiteLLM’s spend tracking and request logs live in its own tables, and nothing about creating a Ragmux connection or project reads them — a migrated deployment starts its request log, budgets and cost estimates at zero on the day it cuts over, with no historical spend carried across. Plan the cutover date accordingly if a monthly budget or a spend report needs to span the transition cleanly: closing out LiteLLM’s numbers for the old period and starting Ragmux’s for the new one, rather than expecting one continuous history, avoids a reporting gap nobody budgeted time to reconcile by hand.

Testing the cutover before it’s permanent

Because nothing about creating a Ragmux connection touches the LiteLLM deployment it was copied from, the two can run side by side against the same upstream provider keys for as long as the comparison takes. Run the same set of requests — the ones that exercise tools, streaming and whatever provider-specific parameters the application actually sends — through both, and check for a mismatch before pointing production traffic at the new project key. POST /admin/api/models/{id}/test catches a bad key or a mistyped base_url before that; it doesn’t catch a subtler behavioral difference, like a routing_strategy that used to mask a flaky deployment and behaves differently under a route’s priority and fallback rules. Once the comparison holds and every application config has moved from the LiteLLM proxy’s URL and virtual key to Ragmux’s URL and project key, the LiteLLM deployment itself is safe to decommission — right down to the API keys it held, which are worth rotating rather than reusing indefinitely across two systems.

Full reference: /docs/configuration/.