Observability

Ragmux exports Prometheus metrics at /metrics and OpenTelemetry spans over OTLP/HTTP. Both are off by default, both are implemented in the standard library (internal/metrics/, internal/tracing/), and both are deliberately small: the gateway takes no Go module dependency for either.

• Metrics
  • Turning it on
  • Why the endpoint is authenticated
  • Scrape configuration
  • The metric set
  • Cardinality checklist
• /readyz and /healthz
• Tracing
  • Turning tracing on
  • The spans
  • Sampling
  • The trust gate
  • No span carries content
  • What a hand-rolled exporter does not get
  • Running a Collector

Metrics

Turning it on

METRICS_ENABLED=true
METRICS_TOKEN=$(openssl rand -hex 24)

or, on a host where Prometheus runs alongside the container:

METRICS_ENABLED=true
METRICS_LISTEN=127.0.0.1:9090

METRICS_ENABLED=true with neither a token nor a loopback METRICS_LISTEN stops the start with

config: METRICS_ENABLED without METRICS_TOKEN would publish per-project usage and spend
unauthenticated; set METRICS_TOKEN or bind METRICS_LISTEN to a loopback address

Why the endpoint is authenticated

The metric set names every project and every model in the installation and reports per-project token counts and spend. That is the same data /admin/api/projects/{id}/metrics serves, and that endpoint requires a session. An unauthenticated /metrics would be the one place the product gives it away for free: anyone who can reach the port learns what your customers are called, which models they use, how much they send and what it costs you.

So the endpoint is authenticated, and there are exactly two ways to satisfy that:

• A bearer token. Authorization: Bearer <METRICS_TOKEN>, compared in constant time so a wrong token cannot be recovered a byte at a time. Responses carry Cache-Control: no-store.
• A loopback bind. With METRICS_LISTEN=127.0.0.1:9090 the endpoint is only reachable from the machine itself, and a token adds nothing.

METRICS_LISTEN does more than move the port. While it is set, /metrics is never registered on the main router, so a reverse proxy configured to forward everything, a path-prefix mistake, or a future route that shadows a deny rule cannot expose it. There is no route to reach.

The endpoint is also excluded from the JSON request log, the way the dashboard’s static assets are: a fifteen-second scrape interval would otherwise be four log lines a minute about nothing, and the scrape is already counted in the metrics it fetches.

Scrape configuration

# prometheus.yml
scrape_configs:
  - job_name: ragmux
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets: ["ragmux:8765"]
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/ragmux-metrics-token

With METRICS_LISTEN=127.0.0.1:9090, point the target at 127.0.0.1:9090 and drop the authorization block.

Alert rules

deploy/prometheus/alerts.yaml is a ready set of Prometheus rules built on the metrics below: scrape target down, /readyz failing, a high 5xx rate, a high upstream error rate, a stuck ingestion backlog, metric series being dropped at METRICS_MAX_SERIES, a project close to its token budget, and the retention job lagging. Load it next to the scrape configuration:

# prometheus.yml
rule_files:
  - /etc/prometheus/ragmux-alerts.yaml

The rules assume the job is named ragmux, as in the scrape configuration above; change the job matcher of RagmuxDown if yours differs. The thresholds are starting points — tune them to your traffic before paging anyone on them. The file can be checked with promtool check rules deploy/prometheus/alerts.yaml.

The metric set

Every name is prefixed ragmux_. Durations are seconds, as Prometheus expects.

HTTP. Recorded by a middleware installed straight after the request-id middleware, so it measures everything below it — including a request that panics.

Gateway. Recorded at the single deferred chokepoint of a finished chat completion, which the success, error, mid-stream-failure and client-disconnect paths all reach.

tokens_estimated_total is a separate metric rather than a estimated="true" label on tokens_total on purpose: a token series that silently mixes estimated and reported counts is a trap for anyone doing cost maths. Divide the estimated request count by ragmux_gateway_requests_total to see how much of a model’s spend figure is guesswork.

Limits.

The denied request itself is still counted by ragmux_http_requests_total with status="429".

Retrieval.

used differs from the store’s configured search_backend when the configured one is not available on this server and the search fell back to pgvector. Comparing the two is how a silent fallback becomes visible:

sum by (store) (rate(ragmux_rag_searches_total{used="pgvector"}[5m]))
  and on() ragmux_search_backend_available{backend="pg_search"} == 0

Ingestion.

ragmux_documents is a SELECT status, count(*) FROM documents GROUP BY status with its own three-second timeout, cached for 30 seconds. Backlog depth is a database property now that the queue lives there, and without the cache a scrape storm — several Prometheus replicas, or a dashboard refreshing — would become a query storm against the same table ingestion is writing to. A scrape that cannot reach the database serves the previous values rather than zero: an alert would read a zero as “the backlog drained”.

Database pool. All read from pgxpool.Stat(), which costs no query.

Self.

Cardinality checklist

The metric set is designed so that the number of series grows with the size of the installation — projects, models, RAG stores, routes — and never with traffic. Every label added to this repository has to pass this list:

• Route labels come from the chi pattern, read after next.ServeHTTP returned (chi fills it during routing), never from r.URL.Path. That single decision is the primary guard: /admin/api/rag-stores/{id}/documents is one series, the path would be one per document id. A request no route matched is labelled unmatched, so a 404 sweep for /wp-admin/... collapses into one series. Dashboard static assets collapse into /admin/*.
• Model labels come from the model connection (conn.ModelName), resolved server-side. Never the model string the client sent: it is attacker-controlled and an unbounded label value is a memory bug, not a cosmetic one.
• Project and store labels are numeric ids. Not names — a name can be renamed, which orphans a series, and a name is user-supplied text.
• Failure reasons are constants from a closed set. rerank_failures_total takes timeout|upstream|parse|unavailable, never err.Error(), which quotes upstream bodies and model replies. limits_denied_total takes the limits.Reason* constants.
• Values the client or the upstream chooses are mapped onto a closed set plus other. Two labels look bounded and are not, so both go through an explicit allowlist in internal/obs:
  • method is whatever token the client wrote on the request line. GET POST PUT PATCH DELETE HEAD OPTIONS pass through; every other verb — PROPFIND, a lowercase get, a per-request UUID — becomes other.
  • errors_total{type} is provider.Error.Type, which upstreamError copies out of the upstream response body. The upstream therefore picks the value, and an upstream that returns an id per response would mint a series per response. Known types (Ragmux’s own upstream_error, invalid_request_error, timeout, plus the documented OpenAI and Anthropic vocabularies) pass through; anything else becomes other. The error is still counted — only the label is collapsed.
• Never a label taken from a header, an error message, a filename, a query string, a user agent or an IP address.

The same rule binds span attributes, which leave the process for a third-party collector: http.request.method on http.server goes through the same mapping.

ragmux.request_id is generated, never read from the request. Ragmux installs its own obs.RequestID in place of chi’s middleware.RequestID, because chi’s starts from the client’s X-Request-Id header and only generates an id when it is absent — which would put attacker-chosen bytes on a span.

Validating the header instead does not make that safe, which is worth stating plainly: a charset-and-length filter cannot tell a generated id from a credential, because they are the same shape. A Ragmux gateway key is sk-user- plus 43 characters from an unreserved alphabet — 51 bytes that pass any such filter — and so are an AWS AKIA… key, an OpenAI sk- key and every hex or base64url token short enough to fit a length cap. For spans and metrics the header is therefore not read at all, which makes the guarantee structural rather than a question of how good the filter is.

/v1 is the one deliberate exception, and it is kept away from telemetry. A client may send its own X-Request-Id to a /v1 endpoint (1 to 128 characters of A-Z a-z 0-9 . _ : -; anything else is replaced, never rejected). That value goes to exactly three places that stay inside your own boundary: the X-Request-Id response header that echoes it back, the request_logs.request_id column, and a client_req_id log field. req_id in the same log lines stays the generated id, so the gateway’s lines and the http access line of one request join on req_id; the access line also carries client_req_id, which is how a client’s id finds it. The client’s value is kept in the gateway’s own context key; the ragmux.request_id span attribute and every metric label still come from the generated id. A client that puts a credential in the header can therefore expose it to itself and to whoever reads your database or your logs, not to the collector. /admin/api is unchanged: its X-Request-Id is generated.

If a proxy in front of Ragmux generates X-Request-Id and you correlate on it, that correlation ends at this hop — Ragmux answers with an id of its own. Configure the proxy to emit traceparent as well and set TRACING_TRUST_INCOMING=true, and the two sides share a trace id in your tracing backend. Read the trust gate first, because that setting also lets a client choose the trace id and the sampled flag.

Be clear about what that does and does not recover. It joins traces. It does not join logs: Ragmux log lines carry req_id, which is the generated request id (and client_req_id when the client sent its own), and no log line carries a trace id, so a proxy log line and a Ragmux log line still have no shared field to match on. And the proxies that typically mint X-Request-Id — nginx’s $request_id, HAProxy’s unique-id — do not emit traceparent unless you configure it, which usually means an OpenTelemetry module rather than a log-format change.

The registry enforces a backstop regardless: at METRICS_MAX_SERIES new combinations are refused, counted in ragmux_metrics_series_dropped_total and logged at error (at most once a minute, so a hot loop of refusals cannot become the log itself).

Treat a non-zero drop counter as an incident, not a note. The cap has no eviction: once it is full, every new legitimate series is refused from then on — a project created today, a model added today, that project’s ragmux_gateway_cost_usd_total series. Existing series keep updating, so nothing looks broken; the new ones simply never appear. Hitting the cap almost always means a label went unbounded, so look for the cause first and raise METRICS_MAX_SERIES only once you know the cardinality is legitimate.

Eviction was considered and rejected. Dropping the least recently used series would keep legitimate metrics from freezing, but it costs a write on every observation — a lock or an atomic store on the scrape path — and an evicted counter that comes back starts at zero, which Prometheus reads as a counter reset and turns into a silently wrong rate(). Data that is visibly missing is preferred to data that is quietly wrong.

The request log

A /v1 call that gets past authentication and request validation writes one row to request_logs, whatever happened to it afterwards: a rate-limit or budget denial is a row with its 429, an upstream failure a row with its error. A call answered earlier writes no row: a 401 or 403, a 400 for a malformed body or a missing messages, project_required, and a route or provider that could not be resolved. Those responses still carry an X-Request-Id, but looking it up in request_logs finds nothing. One request is one row — a retry or a fallback changes attempt, it does not add rows. Besides the usage and cost columns, a row identifies the request:

request_id is a column, never a metric label (see the cardinality checklist). The columns are added by migrations 0016 and 0017; the index is built CONCURRENTLY, so upgrading does not lock a large request_logs table against writes. Replicas that start together take turns on that build (a session advisory lock, polled rather than waited on), and an interrupted build repairs itself: PostgreSQL leaves an INVALID index behind, the next start drops it and builds it again, and the migration is only recorded once the index is valid. If the build fails, startup fails with the error instead of recording a broken index.

/readyz and /healthz

Two probes with two different jobs. Neither needs authentication and neither reports anything about projects or usage.

/healthz is deliberately untouched, so every existing probe and the image’s HEALTHCHECK keep their meaning.

In the bodies below N stands for a migration number, not a literal: migrations is whatever MAX(version) in schema_migrations is and expected_migrations is the head the running binary embeds. Only the relation between the two decides the answer, and both numbers move with every release, so they are written symbolically here rather than pinned to whatever the repository happened to be at.

/readyz answers 200:

{"status":"ok","version":"<x.y.z>","migrations":N,"expected_migrations":N}

and 503 with "status":"migrating" while the applied version is behind the binary’s: the tables this build expects do not exist yet, so the replica cannot serve.

A schema ahead of the binary is not a readiness failure. The comparison is deliberately one-directional. In a maxSurge rolling upgrade the first new pod migrates the shared database; from that moment every old replica — all of them still carrying traffic — sees a schema newer than its own. Failing readiness there would take the entire fleet out of rotation in the middle of a deploy that is supposed to be seamless, and a rollback to the previous image could never become ready at all. Ragmux migrations are additive and non-narrowing (ADD COLUMN IF NOT EXISTS, new tables, no drops), so the older code keeps working against the newer schema. It answers 200 and says so:

{
  "status": "ok",
  "version": "<x.y.z>",
  "migrations": N+1,
  "expected_migrations": N,
  "degraded": ["the database schema is at migration N+1, ahead of the N this binary embeds; this replica is running older code against a newer schema"]
}

Because /readyz returns 200, nothing alerts on this by itself — that is the operator’s to wire up. It is expected during a deploy and unexpected afterwards, so alert on the degraded entry persisting rather than on its appearance. This rests on migrations staying additive and non-narrowing; dropping a column, narrowing a type, or adding a constraint or unique index to an existing column would break the old replicas this decision keeps in rotation, and has to be staged instead — see /readyz during a rolling upgrade for the full list and the expand/contract recipe.

A degraded search backend is not a readiness failure either. A RAG store configured for pg_search on a server without the extension keeps answering — the search falls back to pgvector — so taking the replica out of rotation would turn a degraded-but-working condition into an outage. It is reported instead as an informational field alongside the 200:

{
  "status": "ok",
  "version": "<x.y.z>",
  "migrations": N,
  "expected_migrations": N,
  "degraded": ["pg_search is not installed on this server; 2 rag store(s) configured for it fall back to pgvector"]
}

Alert on the degraded field or on ragmux_search_backend_available; do not gate traffic on either.

Tracing

Turning tracing on

Configuring an endpoint is what turns tracing on; there is no separate switch to remember. TRACING_ENABLED=false is the override for an environment where a sidecar injects the OTEL_* variables and you want this one process quiet.

While tracing is off, a span site costs one atomic load and one branch, and allocates nothing.

The spans

There are eight span sites and no more. They are hand-written, so each one is a decision rather than a by-product.

Request span names

The request span follows the GenAI convention, {operation} {model}, and takes its name once the request has been routed, because that is when the model is known:

A request refused before a model was chosen (a bad key, a malformed body, an unknown route) keeps the fixed names gateway.chat_completion, gateway.embeddings, gateway.retrieve and gateway.rerank. There is no model to put in the name yet.

On a route with fallback targets the name carries the model that finally answered, or the last target tried when none did; the individual targets are the gateway.attempt spans beneath it.

Attempts

Every upstream call of the walk (see Retry and fallback) is its own gateway.attempt child span, whether it is a retry of the same target or the first call to a fallback target. Provider spans (provider.chat, rag.embed_query, …) hang beneath the attempt that made them, so a trace reads top to bottom as request, attempt, upstream call.

The request span repeats the outcome: ragmux.attempts is the number of attempts made (when more than zero), ragmux.route the route, and ragmux.fallback.from_connection the first target when a fallback served the request.

The gen_ai.* names follow the OpenTelemetry GenAI semantic conventions:

• gen_ai.provider.name replaces gen_ai.system. Both are emitted in 0.4.2 so a dashboard can move over; gen_ai.system is removed in a later release. The value is the provider type with two names mapped to the convention’s own: gemini becomes gcp.gemini and cohere_rerank becomes cohere. Every other type passes through unchanged.
• gen_ai.response.model is the model the upstream says answered, which can be a dated snapshot of the one requested. Adapters that echo the requested model back fall back to the connection’s model name. The value is cut at 128 bytes.
• gen_ai.response.finish_reasons is a closed set: stop, length, tool_calls, content_filter, function_call, and other for anything else the upstream sends. A stream reports every distinct reason it saw.

Two details worth knowing:

• A streaming provider.chat span ends when the response body closes, not when the headers arrive, so it covers the whole stream. A span that stopped at the header phase would report a forty-second completion as a two-hundred-millisecond call — precisely the number you opened the trace to find.
• ingest.document is a trace root with no inbound parent. A document reprocessed from the dashboard would otherwise hang a job that may run for minutes off the HTTP request that only queued it, and that request’s span closes in milliseconds.

internal/provider/http.go is also the single place traceparent (and tracestate) is written outbound, since every chat, embed, stream and rerank call funnels through doRequest. Propagation is one code path rather than one per adapter.

Sampling

Head-based, at OTEL_TRACES_SAMPLER_ARG (default 0.05). The decision is made once when a trace starts and inherited by every span beneath it, so a sampled request is complete or absent, never half there. One request in twenty is enough to see the latency structure of the gateway without paying to store every chat completion.

An unsampled trace still propagates its context downstream, so a service you call can sample its own side into the same trace id.

The trust gate

TRACING_TRUST_INCOMING is false by default, which means an inbound traceparent header is ignored and every request starts a new trace here.

That is a deliberate refusal of a convenience. The gateway is reachable by anyone holding an API key, and a client that can set traceparent can pin every request it makes into a single trace id and set the sampled flag on all of it. That is a cheap way to flood the collector from outside and to make one trace unreadable. Turn the gate on only when the callers are your own services behind your own boundary — which is exactly the case where a continued trace is worth something.

The tracestate header is forwarded verbatim but bounded at 512 bytes: it goes out to every upstream, and an unbounded client-supplied value does not belong there.

No span carries content

One rule: no attribute value ever carries message content, the retrieval query, passage text, a filename, the system prompt, or a credential. A span is exported to a third-party collector; it is metadata, not payload.

Two consequences that are easy to get wrong and are handled in the code:

• An upstream failure is recorded on a span by its bounded error type, not its message. A provider that quotes the offending request back inside a 400 would otherwise export prompt text to the collector. The full message still reaches the client and the request log, which stay inside your own boundary.
• rag.retrieve says which store was searched, how, and how many passages came back. It never says what was searched for.

internal/e2e enforces this with a test that pushes five distinct markers through the stack — message content, passage text, a filename, the system prompt and a provider credential — flushes the exporter into a stand-in collector, and greps the captured OTLP payload for each of them. The same test asserts that all eight span names did arrive, so it cannot pass by exporting nothing.

What a hand-rolled exporter does not get

internal/tracing is about three hundred lines of standard library. That buys a dependency-free binary and costs the following, all of which you would get from the OpenTelemetry Go SDK:

• No auto-instrumentation. Only the spans above exist. pgx queries, chi routing, DNS resolution and TLS handshakes produce no spans at all, so a slow query shows up as unexplained time inside its parent span rather than as a span of its own. This is the largest single difference and the one to remember when reading a trace.
• No metrics or logs signal. OTLP carries three signals; this exports one. Metrics go to Prometheus at /metrics, logs to stdout as JSON.
• OTLP/gRPC is minimal. It is opt-in, see Exporting over gRPC, and does one thing: unary Export calls, uncompressed, with no service config and no load balancing.
• No compression. Payloads are uncompressed (JSON over HTTP, protobuf over gRPC). At a 5 % sample rate on a gateway this is small; at 100 % on a busy one it is not.
• No span links, events or baggage. Attributes and a status, nothing else.
• No resource auto-detection. Nothing is inferred about the host, container, Kubernetes pod or cloud region. Set what you need in OTEL_RESOURCE_ATTRIBUTES, or let the Collector’s resourcedetection processor add it.
• No tail sampling, no retry queue. One retry after a second, then the batch is dropped and counted in ragmux_tracing_export_failures_total. A backlog of retries would compete with the requests that produced it.

By default the exporter speaks OTLP/HTTP with the JSON encoding, which is what an OpenTelemetry Collector accepts on port 4318 with no extra configuration. Vendor-direct endpoints vary and several are protobuf-only, so pointing Ragmux straight at a vendor may or may not work depending on the vendor. The supported configuration is to point Ragmux at a Collector and let the Collector talk to your backend — which is also where you add compression, retries, tail sampling and resource detection, all of the things in the list above.

Exporting over gRPC

Some Collectors and vendor agents only open the gRPC receiver (port 4317). Set the protocol to switch the exporter to OTLP/gRPC:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

• HTTP stays the default. http/protobuf is accepted so a sidecar that injects it does not break start-up, but the body is still the JSON encoding.
• The scheme picks the security of the connection. http://host:4317 is plaintext HTTP/2; https://host:4317 and a bare host:4317 are TLS. A plaintext link is something you opt into by writing http://. The OpenTelemetry way to the same end, OTEL_EXPORTER_OTLP_INSECURE=true (or OTEL_EXPORTER_OTLP_TRACES_INSECURE, which wins), makes a bare host:4317 plaintext; an explicit http:// or https:// always decides for itself, and the HTTP transport ignores the variable.
• A path in the endpoint (/v1/traces) is ignored: a gRPC target names a server.
• OTEL_EXPORTER_OTLP_HEADERS are sent as gRPC metadata. Batching, the one retry and ragmux_tracing_export_failures_total behave as over HTTP, and a collector that answers with a non-zero grpc-status counts as a failed export.
• An endpoint that cannot be parsed, or an unknown protocol, is a start-up error.

The span contents are identical on both transports. The gRPC client is built on the standard library’s HTTP/2 rather than google.golang.org/grpc, so the binary gains no dependency.

Running a Collector

docker-compose.otel.yml and otel-collector.yaml in the repository root are a copy-paste starting point:

docker compose -f docker-compose.yml -f docker-compose.otel.yml up -d

The overlay adds an otel-collector service on the gateway’s network and sets OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 on the ragmux service. Out of the box the Collector logs the spans it receives, which is enough to confirm the wiring; otel-collector.yaml has commented exporters for Jaeger, Tempo and an OTLP vendor endpoint to replace the debug one with.

The Collector address is operator-configured, like DATABASE_URL, so the outbound SSRF guard that governs provider calls does not apply to it: http://otel-collector:4318 is a private address on purpose and does not need ALLOW_PRIVATE_UPSTREAMS.