Backup and restore

Ragmux keeps all of its state in one PostgreSQL database plus one secret. Back up both, and you can rebuild a gateway on any machine. This guide covers what to back up, the bundled scripts, scheduled backups, physical backups, the restore runbook and how to rehearse a disaster.

What is state

With the split layout the gateway container itself is stateless: it can be deleted and recreated at any time. With the all-in-one layout the container is disposable too, as long as the ragmux-data volume stays.

Since 0.4 a wrong key stops the start instead of every provider call. The database holds a canary in instance_settings that only the key it was written with can open, so a dump restored onto a host with a different SECRET_KEY fails at boot with SECRET_KEY does not match the one this database was written with (key source: …) rather than looking healthy until the first chat request. Put the matching key back — see SECRET_KEY.

Without the key at all, the fastest way back is a database whose canary you own: restore the dump, drop the canary row (DELETE FROM instance_settings WHERE key = 'secret_key_canary') so the next start seals a new one, and then re-enter every provider API key in the dashboard (Models -> Edit) — users, projects, documents and vectors all come back, only the encrypted credentials are lost. Keep the key in a secret manager (or your deployment’s env store) and in the backup bundle, encrypted at rest or in a separate location from the dumps. scripts/backup.sh can write it next to the dump with INCLUDE_SECRET_KEY=1 (see below).

GET /admin/api/system reports what a backup will contain, with sample values — migrations_version and version are whatever the running build reports:

{
  "backup": {"tables": 1, "documents_bytes": 1048576, "last_migration_at": "2026-09-18T12:34:41Z"},
  "database": {"postgres_version": "17.11", "pgvector_version": "0.8.6", "migrations_version": N, "size_bytes": 8787635},
  "secret_key_source": "env",
  "version": "<x.y.z>"
}

backup.tables is the number of chunk_embeddings_<dims> tables, documents_bytes the sum of the stored document contents; the same numbers are shown on the dashboard overview. ragmux -version prints the binary version.

Logical backups with pg_dump

The recommended format is pg_dump -Fc (custom format): compressed, verifiable with pg_restore --list, and restorable selectively. By hand, with the default all-in-one container (Postgres is reached over the container-local unix socket as postgres):

docker compose exec -T -u postgres ragmux pg_dump -Fc --no-owner --no-privileges -d ragmux \
  > backups/ragmux-$(date +%Y%m%d-%H%M%S).dump

and with the split stack:

docker compose -f docker-compose.split.yml exec -T postgres \
  pg_dump -Fc --no-owner --no-privileges -U ragmux -d ragmux \
  > backups/ragmux-$(date +%Y%m%d-%H%M%S).dump

Notes:

• The dynamically created chunk_embeddings_<dims> tables are ordinary tables and are included automatically; so are their HNSW indexes (rebuilt on restore, which is the slowest part of a large restore).
• The dump contains CREATE EXTENSION vector. The target server must have pgvector available (pgvector/pgvector:pg17 does; on managed Postgres enable it first).
• pg_dump runs in a consistent snapshot, so you can take it while the gateway is running. Documents that are being ingested at that moment are restored in their pending/processing state; the gateway claims them again after a restart, once any lease recorded in the dump has expired.
• Use the same major version of pg_dump/pg_restore as the server (17). Running the tools inside the container that holds Postgres (ragmux in the all-in-one layout, postgres in the split one) guarantees that.

Restoring a ParadeDB dump onto a plain pgvector server

A dump taken from one of the ParadeDB variants carries pg_search objects that a pgvector/pgvector:pg17 target cannot create. pg_restore reports each one and exits 1, even though every ragmux table restores:

ERROR:  extension "pg_search" is not available
ERROR:  relation "paradedb._typmod_cache" does not exist
ERROR:  access method "bm25" does not exist

Note what that list means: dropping idx_chunks_bm25 before the dump is not enough on its own. The BM25 index is only one of the entries — the paradedb schema, the CREATE EXTENSION pg_search, its comment and the extension’s own _typmod_cache table are dumped too, and a restore that only lost the index still fails.

The recipe that works is a TOC filter. pg_dump has no --exclude-index, so list the archive, drop the entries the target cannot create, and restore through the edited list.

First find them. Do not assume the list is the ParadeDB ones: every extension present in the source database is in the dump, and the split ParadeDB image (docker-compose.paradedb.yml) ships more than pg_search — postgis, postgis_topology, postgis_tiger_geocoder, pg_ivm, pg_stat_statements and fuzzystrmatch are all installed in its default database, none of which a pgvector/pgvector:pg17 target has. The all-in-one :<version>-paradedb image installs only vector and pg_search, so there the ParadeDB filter alone is complete.

pg_restore -l ragmux.dump | grep 'EXTENSION -'

Compare that against the target (SELECT name FROM pg_available_extensions;) and filter on what is missing. For an all-in-one ParadeDB dump that is:

pg_restore -l ragmux.dump > full.list
grep -viE 'pg_search|paradedb|bm25' full.list > filtered.list
pg_restore -U ragmux -d ragmux --no-owner -L filtered.list ragmux.dump

For a split ParadeDB dump, add the rest (postgis|pg_ivm|pg_stat_statements|fuzzystrmatch| tiger|topology|spatial_ref_sys). Check the filter before running it: grep matches anywhere in the TOC line, so a pattern like bm25 or paradedb would also drop a table or index of your own whose name contains it, silently. diff full.list filtered.list shows exactly what is being dropped.

A filtered restore finishes with no errors and exit 0. The database is complete: chunks.content and chunks.tsv are what the lexical half reads, and idx_chunks_bm25 is a derived object that ragmux rebuilds lazily — on a ParadeDB server, on the first hybrid search of a store using the pg_search backend.

Nothing else needs changing. A rag_stores row keeps search_backend = "pg_search" through the restore and the gateway falls back to pgvector for it — hits still come back, the search response names the backend that really ran, and one warning per store is logged (see Search backends). Restoring the same dump back onto a ParadeDB server later finds the setting still there and rebuilds the index; on a large corpus that first search is the one that pays for the build, so consider building it ahead of time (see the upgrade notes in CHANGELOG.md).

Roles are cluster-global and are never in a database dump, so create ragmux_app on the target before restoring a split-layout dump, whichever recipe you use.

Scripts

Both scripts live in scripts/, run with bash, print -h help, exit 0 on success, 1 when the dump/restore itself fails and 2 for usage or configuration errors. They never print DATABASE_URL, passwords or SECRET_KEY, run with umask 077 so every file they create is private to the invoking user, and never put a credential on a command line (restore.sh sends the login body through stdin and the bearer token through a private header file, so neither shows up in ps or shell history).

Where they connect

1. If DATABASE_URL is set and pg_dump/pg_restore are on PATH, the scripts use the local tools against that URL (external or managed Postgres, or the binary without Compose).
2. Otherwise they use docker compose. Run them from the directory holding docker-compose.yml, or set COMPOSE_FILE (COMPOSE_FILE=docker-compose.split.yml for the split stack). The Compose project is chosen the usual way: -p NAME, COMPOSE_PROJECT or the standard COMPOSE_PROJECT_NAME environment variable, which docker compose honours by itself. The layout is detected from the project (LAYOUT=auto): when it has a <POSTGRES_SERVICE> service (postgres) the scripts run docker compose exec -T postgres … as before (split); otherwise they treat the <RAGMUX_SERVICE> container (ragmux) as the all-in-one image and run the tools inside it as the postgres user over the unix socket (aio). LAYOUT=split|aio overrides the detection.

scripts/backup.sh

scripts/backup.sh                       # -> ./backups/ragmux-YYYYmmdd-HHMMSS.dump
BACKUP_DIR=/mnt/backups KEEP_DAYS=30 INCLUDE_SECRET_KEY=1 SECRET_KEY_DIR=/mnt/keys scripts/backup.sh
make backup                             # same as scripts/backup.sh

What it does: pg_dump -Fc --no-owner --no-privileges into <name>.dump.partial, pg_restore --list on the result (a corrupt or truncated file fails here and is deleted), rename to <name>.dump, print the size, then rotate. The final line on stdout is the path of the new dump, so it composes with other tooling:

f=$(scripts/backup.sh) && aws s3 cp "$f" s3://my-bucket/ragmux/

scripts/restore.sh

scripts/restore.sh backups/ragmux-20260918-153455.dump          # prints the plan, exit 2
scripts/restore.sh --yes backups/ragmux-20260918-153455.dump    # runs it
ADMIN_USER=admin ADMIN_PASSWORD=… scripts/restore.sh --yes …    # also reports migrations_version
make restore FILE=backups/ragmux-20260918-153455.dump YES=1

Steps, in order:

1. Verify the file with pg_restore --list (in the all-in-one layout through a throwaway docker compose run --rm --entrypoint pg_restore ragmux --list, before anything is stopped).
2. Stop the gateway (docker compose stop ragmux; in DATABASE_URL mode run STOP_CMD or warn). This matters: --clean drops tables and blocks on open connections. All-in-one: stopping the ragmux service stops the embedded Postgres too, so the script then starts a one-off container on the same volume that runs Postgres alone: docker compose run -d --rm --no-deps ragmux postgres-only, and waits for pg_isready inside it (up to 90 s).
3. pg_restore --clean --if-exists --no-owner --no-privileges -d ragmux, through docker compose exec -T postgres (split) or docker exec -i -u postgres <one-off> (aio). Every object in the dump is dropped and recreated, so the target does not need to be empty.
4. Hand the restored tables to the application role (APP_ROLE, default ragmux_app), see Ownership below.
5. Start the gateway (docker compose up -d ragmux; START_CMD in direct mode). All-in-one: the one-off container is stopped first (docker stop, a clean pg_ctl stop -m fast), then the normal service comes up. Migrations run at startup.
6. Poll /healthz and, if admin credentials are set, print the system info.

When the restore fails the gateway is left stopped (and, all-in-one, the one-off container is stopped as well) so you can investigate before migrations run on a half-restored schema. Done by hand, the all-in-one sequence is:

docker compose stop ragmux
cid=$(docker compose run -d --rm --no-deps ragmux postgres-only)
until docker exec -u postgres "$cid" pg_isready -q; do sleep 1; done
docker exec -i -u postgres "$cid" pg_restore --clean --if-exists --no-owner --no-privileges -d ragmux < <dump>
docker stop "$cid"          # clean Postgres shutdown; the container removes itself
docker compose up -d ragmux # the entrypoint re-runs 01-ragmux.sql: tables handed to ragmux_app

Ownership. pg_restore runs as the superuser (POSTGRES_USER) with --no-owner, so every restored table belongs to the superuser, while the gateway connects as the least-privilege role ragmux_app (see Database privileges). Reads would fail and the next migration’s ALTER TABLE too, so right after the restore the script changes the owner of every table and sequence in public to APP_ROLE (default ragmux_app) and grants it ALL on them. The step is skipped with a note when the role does not exist (a deployment that still uses the superuser DATABASE_URL), and in DATABASE_URL mode it needs psql on PATH (otherwise a warning tells you to run it by hand: ALTER TABLE ... OWNER TO ragmux_app for each table, or simply re-run docker/postgres-init/01-ragmux.sql, whose last block does the same). The all-in-one entrypoint runs that SQL on every start anyway, so a restore done by hand there is fixed up by the next docker compose up -d.

Extension errors. The dump contains DROP EXTENSION IF EXISTS vector and CREATE EXTENSION vector. On the bundled Postgres (either layout) the restore runs as the superuser, so both succeed silently. On servers where the extension was created by a superuser and the restore runs as a plain role (managed Postgres, shared clusters) the drop fails and the create then reports extension "vector" already exists. pg_restore marks any failed statement with a non-zero exit, so the script reads its stderr instead of using --exit-on-error: errors mentioning extension are printed as ignored, any other pg_restore: error: line fails the restore with exit 1, and the gateway is left stopped so you can investigate without it re-running migrations on a half-restored schema.

Leftover tables. --clean only drops what the dump contains. If the target already had a chunk_embeddings_<dims> table for a width the dump does not include, that table survives, empty of anything the restored stores reference. It is harmless; drop it by hand if you want a tidy schema.

Scheduled backups (Compose profile)

Both Compose files ship a backup service under the backup profile that runs prodrigestivill/postgres-backup-local:17:

docker compose --profile backup up -d          # starts ragmux (and postgres) plus backup
docker compose --profile backup exec backup /backup.sh   # trigger a run right now

With the default all-in-one file the backup container reaches the embedded Postgres over the unix socket shared through the ragmux-pgsocket volume (POSTGRES_HOST=/var/run/postgresql, user postgres, trust authentication; the POSTGRES_PASSWORD the image insists on is a placeholder). With the split file it connects over TCP with POSTGRES_PASSWORD.

It dumps ragmux on SCHEDULE (default @daily, cron syntax accepted) into ./backups/{last,daily,weekly,monthly}/ragmux-<stamp>.sql.gz and keeps BACKUP_KEEP_DAYS=7, BACKUP_KEEP_WEEKS=4, BACKUP_KEEP_MONTHS=6 of history. Override these and BACKUP_SCHEDULE in .env. The image’s own healthcheck listens on 8081 inside the container (not published).

These files are plain SQL, gzip-compressed, not the custom format restore.sh expects. Restore one with psql after stopping the gateway; split stack:

docker compose -f docker-compose.split.yml stop ragmux
gunzip -c backups/daily/ragmux-20260918.sql.gz \
  | docker compose -f docker-compose.split.yml exec -T postgres psql -v ON_ERROR_STOP=0 -U ragmux -d ragmux
docker compose -f docker-compose.split.yml up -d ragmux

All-in-one (a one-off Postgres on the volume, as restore.sh does it):

docker compose stop ragmux
cid=$(docker compose run -d --rm --no-deps ragmux postgres-only)
until docker exec -u postgres "$cid" pg_isready -q; do sleep 1; done
gunzip -c backups/daily/ragmux-20260918.sql.gz \
  | docker exec -i -u postgres "$cid" psql -v ON_ERROR_STOP=0 -d ragmux
docker stop "$cid"
docker compose up -d ragmux

The scheduled dumps use CREATE EXTENSION IF NOT EXISTS, but they do not DROP existing tables, so restore them into an empty database (docker compose down -v first, or a fresh CREATE DATABASE). Point the backups directory (or an rsync/object-storage job reading it) at storage that is not on the same disk as the database volume.

Physical backups and point-in-time recovery

Logical dumps give you an RPO equal to the dump interval. If losing more than a few minutes of request logs, uploads or key rotations is unacceptable, add continuous WAL archiving with pgBackRest or wal-g: a periodic base backup plus every WAL segment, restorable to any second. This is worth it when the database is large (dumps take long and rebuild HNSW indexes on restore), when ingestion is continuous, or when the gateway’s request log is your audit trail. It needs a sidecar or a custom Postgres image with the tool installed and access to the data directory; keep SECRET_KEY in the same disaster-recovery bundle exactly as with dumps.

Managed Postgres (Amazon RDS/Aurora, Cloud SQL, Azure Database, Neon, Supabase, Crunchy Bridge…): use the provider’s automated snapshots and PITR; they cover the chunk_embeddings_* tables like any other. Still take a periodic pg_dump for off-provider copies and for moving between providers, and still keep SECRET_KEY outside the provider. Enable the vector extension on the target before restoring.

Restore runbook

Prerequisites: a dump, the matching SECRET_KEY, the Ragmux image version you intend to run, and a Postgres 17 with pgvector (the all-in-one image brings its own).

1. Stop the gateway so nothing writes during the restore: docker compose stop ragmux (all-in-one: this stops the embedded Postgres too; restore.sh brings a Postgres-only container up for the restore).
2. Restore: scripts/restore.sh --yes <dump> (it stops and starts the gateway for you in either layout), or by hand as shown under scripts/restore.sh: split stack docker compose exec -T postgres pg_restore --clean --if-exists --no-owner --no-privileges -U ragmux -d ragmux < <dump> followed by the ownership hand-over the script performs (re-running docker/postgres-init/01-ragmux.sql as in Database privileges does it); all-in-one, the postgres-only sequence, after which the entrypoint’s own start does the hand-over.
3. Start with the right SECRET_KEY in .env: docker compose up -d ragmux. Migrations run automatically under an advisory lock.
   • Restoring an older dump into a newer Ragmux is supported: schema_migrations comes from the dump and only the missing versions are applied on start.
   • Restoring a newer dump into an older Ragmux is not: the old binary does not know the newer schema. Upgrade the image first.
4. Verify (or let the script do the first three):
   • curl -fsS http://localhost:8765/healthz returns {"status":"ok",…}.
   • Log in to /admin/ with a known user.
   • GET /admin/api/system: database.migrations_version matches the app’s expected version, backup.tables and backup.documents_bytes look like the source.
   • Models tab: api_key_masked is shown and Test chat succeeds for one connection (this proves SECRET_KEY is right).
   • RAG store: run a search in the dashboard and get hits.
   • Projects: the api_key_prefix values are the ones your clients use; make one real POST /v1/chat/completions with an existing sk-proj-… key.
5. Take a fresh backup of the restored system so the chain of dumps continues.

Disaster-recovery drill

Rehearse on a scratch project at least quarterly and before every major upgrade:

• scripts/backup.sh on production (or fetch the latest scheduled dump).
• On a clean machine or with a different Compose project name (COMPOSE_PROJECT_NAME=ragmux-drill, different ports:), docker compose up -d (a fresh ragmux-data volume; the dump may come from either layout).
• scripts/restore.sh --yes <dump> with the production SECRET_KEY.
• Run the verification checklist above; record how long the restore took (your RTO).
• Restore once without the key to see the decryption error, so the team knows what “lost the key” looks like and where the key lives.
• docker compose down -v the drill stack.

Rules of thumb:

• Back up before every upgrade. docker compose pull && scripts/backup.sh && docker compose up -d. Migrations are forward-only.
• Keep at least 7 daily, 4 weekly and 6 monthly dumps, and one copy off the host.
• Alert when the newest dump is older than twice the schedule.
• Keep dumps from before a SECRET_KEY rotation together with the old key.

Moving between layouts

The all-in-one volume (ragmux-data, /data/pg) and the split stack’s pgdata volume have different names and directory structures, and nothing converts one into the other: moving between the layouts (including from the 0.3.0 two-container docker-compose.yml to the default single-container one) is a dump and a restore.

1. Take a dump on the old layout: scripts/backup.sh (it detects the layout; add -f docker-compose.split.yml / COMPOSE_FILE when the split file is the old one).
2. Stop the old stack: docker compose [-f docker-compose.split.yml] down (keep the volume until the new layout is verified).
3. Start the new layout on a fresh volume with the same SECRET_KEY in .env: docker compose [-f docker-compose.split.yml] up -d (the split file also needs POSTGRES_PASSWORD and RAGMUX_DB_PASSWORD).
4. scripts/restore.sh --yes <dump>, then the verification checklist above.
5. Once verified, remove the old volume (docker volume rm <project>_pgdata or <project>_ragmux-data).

A deployment that used the secret.key fallback (all-in-one without SECRET_KEY) must copy /data/ragmux/secret.key out of the volume first (docker compose cp ragmux:/data/ragmux/secret.key ./secret.key) and set its content as SECRET_KEY on the new layout.

Rotating SECRET_KEY

Rotate the key when it may have leaked, when someone who knew it leaves, or on a schedule. ragmux rotate-key re-encrypts every stored provider API key in one transaction, verifies each row decrypts with the new key before committing and prints the number of rows changed. It needs the current key in the environment (SECRET_KEY, SECRET_KEY_FILE or the secret.key fallback) and DATABASE_URL.

1. Take a backup (scripts/backup.sh) and keep the old key with it.

2. Stop the gateway so nothing writes a credential with the old key meanwhile: docker compose stop ragmux.

3. Generate the new key and run the rotation with the old one still configured:

   NEW_KEY=$(openssl rand -hex 32)
   docker compose run --rm ragmux rotate-key --new "$NEW_KEY"
   # rotate-key: re-encrypted 3 model connection(s); start the gateway with the new SECRET_KEY

   In the all-in-one layout that one-off container starts the embedded Postgres on the volume for the duration of the command and shuts it down cleanly afterwards (the stopped service must not be running at the same time, hence step 2).

   With the binary: SECRET_KEY=<old> DATABASE_URL=… ragmux rotate-key --new "$NEW_KEY".

4. Put the new key into .env (or the secret file) and start the gateway: docker compose up -d ragmux.

5. Verify: Test chat on a model connection succeeds. If the gateway was started with the wrong key it refuses to start at all (SECRET_KEY does not match the one this database was written with); put the right key back, nothing was lost.

Rows are only rewritten when every one of them decrypts with the current key and re-decrypts with the new one; otherwise the command exits 1 and the database is unchanged. The same transaction re-seals the instance_settings canary, so the first start after a rotation accepts the new key and refuses the old one. Dumps taken before the rotation still need the old key.

Since 0.2.3 each stored key is also bound to its connection id (key_version = 1 in model_connections), so a ciphertext moved to another row does not decrypt. Dumps taken with an earlier version restore fine: the gateway re-seals every key_version = 0 row with the same SECRET_KEY on its first start and logs how many it changed. That start fails with upgrade stored provider keys when the key does not match the dump; put the right key back and start again.

Security notes

Dumps contain the encrypted provider API keys, every uploaded document, password hashes, the audit log and request logs (which include model names, token counts and error messages, but not prompts or completions). Treat them like the database itself: encrypt at rest (age, gpg, or an encrypted bucket), restrict who can read the backups directory (backup.sh creates it with mode 700 and every file with 600), and do not store <name>.key next to the dump on the same unencrypted disk unless the whole disk is encrypted: point SECRET_KEY_DIR somewhere else. When you copy the key file elsewhere, keep its permissions.