# Running more than one replica > Ragmux keeps no durable state in the process. Source: https://ragmux.com/docs/scaling/ Updated: 2026-10-10 Project: https://github.com/ragmux/ragmux (AGPL-3.0-or-later) --- Ragmux keeps no durable state in the process. Everything that outlives a request lives in PostgreSQL, so scaling out is a matter of pointing several gateway processes at the same database, putting a plain round-robin proxy in front, and setting `SECRET_KEY` so every replica reads the same credentials. This page covers what that takes, what the gateway does differently with more than one replica, and what is deliberately not replicated. ## What is already shared | Concern | Where the state lives | Replica-safe | |---|---|---| | Sessions and API keys | `sessions`, `projects.api_key_hash` | yes - a cookie issued by one replica authenticates on any other | | Rate limits and budgets (RPM, TPM, daily/monthly) | `project_usage`, reserved atomically per UTC window | yes - the counter is a row, not a process variable | | Login throttling and lockouts | `login_attempts` | yes | | Request logs, audit log, metrics | `request_logs`, `audit_logs` | yes | | Provider credentials | `model_connections.api_key_enc`, AES-256-GCM | yes, **with a shared `SECRET_KEY`** (see below) | | Documents, chunks, embeddings | `documents`, `chunks`, `chunk_embeddings_` | yes | | Document ingestion queue | `documents.status` + claim columns | yes - see [Ingestion across replicas](#ingestion-across-replicas) | | Retention job | leader-elected with a session advisory lock | yes - see [The retention leader lock](#the-retention-leader-lock) | | Schema migrations | `schema_migrations` under an advisory lock | yes - replicas may start together | | Streaming completions (SSE) | nothing; a per-request passthrough | yes - see [Streaming](#streaming) | | First-run setup / `ADMIN_PASSWORD` bootstrap | `users`, guarded so exactly one wins | yes | ## What you must set ### SECRET_KEY `SECRET_KEY` is the AES-256 key that encrypts provider credentials at rest. With it unset, a replica generates one into `DATA_DIR/secret.key` - and with several replicas, **each generates its own**, so the second one cannot read what the first wrote. Set it before you scale: ``` SECRET_KEY=$(openssl rand -hex 32) ``` Since 0.4 a wrong or diverged key is a boot failure rather than a puzzle. On start, the gateway seals a fixed value into the `instance_settings` table (the first replica to reach an empty table wins; the others verify against it) and reads it back before it touches a single credential. When it does not decrypt, the start fails with the key's source named: ``` SECRET_KEY does not match the one this database was written with (key source: env): the canary stored in instance_settings does not decrypt. Start with the original key, or run "ragmux rotate-key" with it in the environment to move the database to a new one. ``` That catches the cases operators actually hit: a second replica falling back to its own key file, a dump restored onto a host that has a different key, a rotation applied to one node only, a wiped data volume. It costs nothing on a single instance. `ragmux rotate-key` re-seals the canary in the same transaction as the credentials, so a rotation needs no extra step. See [Rotating SECRET_KEY](/docs/backup-restore/#rotating-secret_key). ### An external PostgreSQL The default all-in-one layout (`docker-compose.yml`) bundles PostgreSQL in the same container and **cannot** be scaled: a second replica would start a second database on its own volume. Use `docker-compose.split.yml`, or any managed PostgreSQL 14+ with the `vector` extension, and point every replica at it. ### Connection pooling Each replica opens its own pool of `DB_MAX_CONNS` connections (default 10), so the fleet needs `DB_MAX_CONNS × replicas` connections plus headroom for backups and psql sessions, and the server's `max_connections` must cover that. Either lower `DB_MAX_CONNS` per replica, or put a pooler in front. If you use **PgBouncer**, two things matter: - **Session mode** keeps everything working, including the retention leader lock. A session advisory lock belongs to a connection, and transaction mode hands that connection to somebody else between statements, so the lock cannot be held. Nothing breaks - every step of the retention pass is idempotent - but several replicas will each run a pass. - **Lazy DDL is safe in either mode.** The two things this gateway builds on demand - a `chunk_embeddings_` table for a new embedding width, and the BM25 index of the `pg_search` backend - are not idempotent the way the retention pass is: `CREATE INDEX IF NOT EXISTS` raced by two sessions fails on one of them. Both take `pg_advisory_xact_lock` in the same transaction that runs the DDL, which is the one unit transaction mode keeps on a single server connection, so neither depends on session mode. - **pgx v5 prepares statements** and caches them per connection. Under transaction mode a cached statement can be sent to a connection that never prepared it. Either use session mode, or disable the cache in the connection string: ``` DATABASE_URL=postgres://user:pass@pgbouncer:6432/ragmux?default_query_exec_mode=exec&statement_cache_capacity=0 ``` ## Ingestion across replicas Uploaded documents are not processed by the replica that received them. They are a queue in the `documents` table, and every replica works it. - **Claim.** One dispatcher goroutine per replica (not one per worker, so the database sees one poll per replica) takes the next claimable row with `FOR UPDATE SKIP LOCKED`: a document that is `pending`, or one that is `processing` but whose owner stopped renewing its lease. Skipping locked rows is what lets every replica poll at once without queueing behind each other. - **Lease.** The claim writes `claimed_by`, `claimed_at` and `lease_until`. While the job runs, the worker pushes `lease_until` out every third of `INGEST_LEASE`. If a replica is killed, nothing renews the lease and the document becomes claimable again once it expires - at worst `INGEST_LEASE` after the crash. - **Heartbeat.** When a renewal finds the claim gone (another replica took the document over), the job's context is cancelled, the worker stops, logs `lost ingestion lease; another replica took over`, and deliberately writes **no** status: the row belongs to its new owner now. - **Attempt cap.** Each claim increments `attempts`, and a document at `INGEST_MAX_ATTEMPTS` is skipped by the claim query. Without that cap, a document that reliably takes the process down becomes a cluster-wide crash loop as replicas take turns on it. The retention job gives those rows their final `failed` status. A successful ingest and an explicit retry (upload, reprocess) both reset the counter, so a healthy document never accumulates. - **Clean shutdown.** A replica that stops in an orderly way puts whatever it still owned straight back into the queue instead of leaving it to the lease, and gives the attempt back. A single-instance restart therefore picks its work up immediately, exactly as it did before the queue moved into the database. **Honest about the failure mode:** a lease that expires while the replica is alive but slow - a long GC pause, a database blip that ate the heartbeats - can have one document processed twice. That costs embedding money, not data integrity: `ReplaceDocumentChunks` is a single transactional delete-and-insert with the status write inside it, so the last writer wins cleanly and no half state is ever visible. The heartbeat, and the context it cancels the moment the claim is gone, is what keeps it rare. ### Settings | Variable | Default | What it trades | |---|---|---| | `INGEST_LEASE` | `2m` | How long a crashed replica's document stays untouchable. Lower means faster failover and more heartbeat traffic; it is renewed, not sized to the job (a job may run 15 min). | | `INGEST_POLL_INTERVAL` | `5s` | How quickly a replica notices work queued by another replica. Uploads on the same replica kick the dispatcher immediately. | | `INGEST_MAX_ATTEMPTS` | `5` | How many claims a document gets before the retention job fails it. | | `MAX_PENDING_DOCUMENTS` | `1024` | The backlog an upload is still accepted into. | ### The 503 on upload changed meaning `MAX_PENDING_DOCUMENTS` replaced the old in-process channel of 1024. A `503` on `POST /admin/api/rag-stores/{id}/documents` used to mean *this replica's channel is full* - a number that said nothing about the fleet, and that a second replica would have happily accepted. It now means *every replica together is this far behind*, counted as `SELECT count(*) FROM documents WHERE status='pending'` and cached for about a second. That is genuine cross-replica backpressure and the number an operator can act on. ## The retention leader lock The retention job (request logs, audit entries, login attempts, expired sessions, usage counters, exhausted documents) runs on every replica, an hour apart. Before each pass a replica takes `pg_try_advisory_lock`; one gets it and does the work, the others log at `debug` and return. The first pass is jittered over an extra minute so a fleet restarted together does not wake up together. Every step of the pass is an idempotent `DELETE ... WHERE created_at < cutoff`, so **the lock is an optimisation against duplicated work, not a correctness requirement**. That is also why a connection pooler in transaction mode - where a session lock cannot be held - costs a duplicated pass and nothing else. The alternative, a `leader_election` table holding a lease row, was rejected: it needs schema, a renewal loop, and a window in which a dead leader's lease has not expired yet. `pg_try_advisory_lock` needs no schema at all, releases the moment the connection dies, and matches how the codebase already serialises migrations. The lock key is scoped to the current schema, so two Ragmux instances sharing one database in separate schemas elect their own leader. ## Streaming **Streaming needs no session affinity.** `POST /v1/chat/completions` with `"stream": true` is a passthrough: the gateway opens the upstream request, copies SSE chunks to the client as they arrive, and writes one `request_logs` row when it is done. There is no stream id, no resume token, no server-side buffer to reattach to - so there is nothing for a second request to be routed back to. Configure your proxy for plain round robin; `ip_hash` or a sticky cookie only costs you an unbalanced fleet. What this means in practice: - Two clients using the same project key can stream from two replicas at the same time. Each replica writes its own log row, and both count against the same shared RPM/TPM budget, because that budget is a database row. - If the replica serving a stream dies, the client sees the connection drop. The replica records the interrupted request with `status_code = 499` (nginx's convention for a client that went away) and the upstream call is cancelled. There is no state to clean up. **The client retries, and any replica can serve the retry.** A retry is a new completion, not a resumption - the same as a dropped stream against any provider API. - `internal/e2e/streaming_replica_test.go` is that behaviour as a test: two full router stacks over one database, concurrent streams on one key, a severed replica mid-stream, and the shared rate limit. On the proxy, the only requirements are the ones a single instance already has: do not buffer responses (`proxy_buffering off`) and keep the read timeout above `STREAM_MAX_DURATION` (default `30m`). See [Behind a reverse proxy](/docs/configuration/#behind-a-reverse-proxy). ## Rolling restarts and shutdown On `SIGINT`/`SIGTERM` a replica shuts down in this order: 1. Stop accepting HTTP and drain in-flight requests - **15 s**. A streaming completion still running at the end of that window is cut off and logged as `499`. On the default configuration `/metrics` is served by this router, so it stops answering here. 2. Wait for running ingestion jobs - **30 s**, then their context is cancelled. 3. Put every document this replica still owned back into the queue, and give the cancelled attempt back - **10 s**, on its own budget after step 2. 4. Flush queued spans to the collector - **5 s**, skipped when tracing is off. Spans that end after this are counted in `ragmux_tracing_spans_dropped_total` and logged. 5. Close the separate metrics listener - **5 s**, and only when `METRICS_LISTEN` is set; it is kept open this long so the dropped-span count is final before the endpoint goes away. 6. Stop the retention job (releasing the leader lock with its connection) and close the pool. Those budgets are sequential, so the worst case is **65 s** - and it is a worst case, not a typical one: an idle replica walks the whole list in milliseconds, because every step above is a deadline rather than a wait. So the worst case for the fleet is: a replica is `SIGKILL`ed before step 3 runs, and the documents it was working on wait out `INGEST_LEASE` (2 min by default) before another replica claims them. Nothing is lost; a document is only ever `ready` after one transactional write. - Give the container room for the whole sequence: `stop_grace_period: 90s` in Compose, `terminationGracePeriodSeconds: 90` in Kubernetes. A shorter grace period turns every deploy into the lease-expiry case, and the defaults are far shorter - 10 s in Docker, 30 s in Kubernetes - so this has to be set explicitly. - Roll one replica at a time. Migrations are serialised by an advisory lock, so a new replica starting against a database an old one is still using is safe as long as the schema change is backwards compatible - which is the rule these migrations follow (`ADD COLUMN IF NOT EXISTS`, new tables, no drops). ### `/readyz` during a rolling upgrade There is one shared database and one schema, so the moment the first new replica starts, it migrates the database out from under every old replica still serving. `/readyz` compares the applied schema version against the one the binary embeds, and the comparison runs in **one direction only**: | Applied vs. embedded head | `/readyz` | Why | |---|---|---| | behind (`applied < head`) | `503`, `"status":"migrating"` | The tables this build expects do not exist yet. It cannot serve. | | equal | `200` | The normal case. | | ahead (`applied > head`) | `200` with a `degraded` entry | Older code, newer schema. It keeps serving. | Failing readiness on a newer schema would be the stricter reading, and it was rejected because of what it costs: with `maxSurge: 1` the first new pod migrates the database and *all N old replicas* go unready at once, so a deploy designed to be seamless empties the fleet instead - `maxUnavailable: 0` does not help, because the old pods take themselves out. Rolling back is worse: the previous image would never become ready against the schema the new one left behind. **This depends on migrations staying additive *and* non-narrowing.** Old code running against a newer schema is only safe while both hold. A migration breaks the replicas this behaviour deliberately keeps in rotation if it: - drops or renames a column or table, - narrows a type or adds `NOT NULL` to an existing column, - **adds a constraint or a unique index to an existing column.** This last one is not hypothetical: `0008_users_ci_audit.sql` added `CREATE UNIQUE INDEX ... ON users (lower(username))`. During the upgrade window an old replica still allows creating `Alice` next to an existing `alice` - its code predates the rule - and the write now fails on a constraint it does not know about. Narrow and admin-only here, but the shape is general: a new constraint makes previously valid writes fail in old code. Any of those needs the staged expand/contract treatment - ship the additive half, roll every replica onto code that respects the new rule, then add the constraint or drop the column in a later release - rather than a single migration. If one has to ship as a single migration, this readiness behaviour is not enough on its own and the upgrade needs a maintenance window. The `degraded` entry is informational and `/readyz` still returns `200`, so nothing pages on it by itself. It is expected for the length of a deploy; alert on it *persisting*, which means a replica was left behind. ## The Compose example ```bash # .env needs SECRET_KEY, POSTGRES_PASSWORD and RAGMUX_DB_PASSWORD docker compose -f docker-compose.split.yml -f docker-compose.scale.yml \ up -d --scale ragmux=3 ``` `docker-compose.scale.yml` is an overlay on the split layout. It removes the `container_name` and the fixed `127.0.0.1:8765` host port (either of which alone stops `--scale` from working), adds an nginx front end (`docker/nginx/ragmux.conf`) that resolves the `ragmux` service name per request so it picks up a scale change without a reload, and sets `stop_grace_period: 90s`. Check that all three replicas are working the queue: ```bash docker compose -f docker-compose.split.yml -f docker-compose.scale.yml ps docker compose -f docker-compose.split.yml -f docker-compose.scale.yml \ logs ragmux | grep 'ingestion ready' ``` Each replica logs its own `owner` (host, pid and a random suffix), which is what appears in `documents.claimed_by`. ## Kubernetes - A plain `Deployment` with `replicas: N`; no `StatefulSet`, no volume. The only mounts a replica needs are its configuration. - `SECRET_KEY` and `DATABASE_URL` from a `Secret`. `SECRET_KEY_FILE` and `DATABASE_URL_FILE` read a mounted file instead, if you prefer that. - `terminationGracePeriodSeconds: 90` or more - see above. - Liveness on `GET /healthz`, which pings the database. Readiness on either: `/healthz` keeps the 0.3 behaviour (ready as soon as the process listens and the pool answers), `/readyz` additionally holds a replica back until the schema it needs is applied - see above for how it behaves mid-upgrade. Give readiness a short period. - A `Service` of type `ClusterIP` and any ingress controller. **Do not set session affinity.** Turn response buffering off for the ingress (`nginx.ingress.kubernetes.io/proxy-buffering: "off"`) and raise `proxy-read-timeout` past `STREAM_MAX_DURATION`. - Scale ingestion with `INGEST_WORKERS` per replica rather than with replicas alone: workers are cheap, replicas each cost a connection pool. - A `PodDisruptionBudget` of `maxUnavailable: 1` keeps a node drain from emptying the fleet into the lease-expiry case at once. ## What is not replicated None of these need to be, but it is worth knowing why: - **The PDF worker subprocess.** PDFs are parsed by re-executing the gateway binary as `ragmux pdf-extract`, so a malformed file cannot take the server down with it. That subprocess is local to the replica running the ingestion job and lives for the length of one parse. It holds the document's claim through the parent, so nothing else touches the row meanwhile. - **In-process caches.** Each replica keeps the set of `chunk_embeddings_` tables it has already seen, and an HTTP transport with idle upstream connections. Both are derived, not authoritative: the worst a cold replica does is one extra `CREATE TABLE IF NOT EXISTS` or one extra TLS handshake. Nothing invalidates across replicas, and nothing has to. - **Circuit breaker state.** Each replica keeps the breaker of every model connection in memory ([Retry and fallback](/docs/providers/#retry-and-fallback)). A replica opens a breaker from the failures it saw itself, so replicas notice a bad upstream at different moments, and a fleet of `N` replicas sends up to `N` probes to a recovering one. A restarted replica starts with every breaker closed. `ragmux_upstream_breaker_state` is per replica; read it per instance, not as a fleet-wide value. Sharing the state would put a write on the request path for what is only a hint, so it is deliberately not shared. - **The one-second ingestion backlog count.** Each replica caches the pending count for about a second, so a burst of uploads arriving at different replicas within the same second can overshoot `MAX_PENDING_DOCUMENTS` by the size of the burst. The ceiling is a coarse guard against an unbounded queue, not a quota. - **The `secret.key` fallback file.** Deliberately per-replica and deliberately unusable for a fleet - which is the whole point of the canary above.