Ragmux

Use cases

RAG over Turkish content: store language, metadata filters and condensed queries

Set up Ragmux retrieval for Turkish documents: the store language for stemmed full-text search, a metadata filter for mixed-language stores, and condense for follow-ups.

Updated: Last reviewed: Verified against Ragmux 0.6.0

Turkish is the case where a retrieval stack’s defaults quietly disagree with the text. Words change shape by suffix, so the form in a question is often not the form in the document; users write follow-ups that mean nothing without the previous turn; and a company often keeps Turkish and English versions of the same policy side by side. Ragmux has three settings aimed at exactly those, and none of them is Turkish-only: they are the ordinary retrieval controls, and this page walks through them with Turkish content as the example.

Store language: what it changes and what it doesn’t

A store’s language is simple by default. Set it to tr and the store’s chunks are indexed with PostgreSQL’s turkish text search configuration, and the query is parsed with the same one. Because both sides use one setting, they cannot disagree about what a word’s stem is. That was not always true: in earlier releases the query configuration could be chosen separately while the index was built with simple, so a turkish query looked for stems the index did not contain and lost matches without saying so.

What it affects:

  • The full-text half of a hybrid search. The default hybrid mode fuses vector similarity with full-text ranking, and language shapes the second. In vector mode it has nothing to act on.
  • Token estimates. Where Ragmux has to estimate tokens, it divides characters by 4 for simple and by 3.5 for Turkish, a ratio the documentation says was measured against the o200k_base tokenizer. The estimate feeds chunk sizing, the embedding tokens of an upstream that reports none, and the retrieved context in the per-minute token limit. Any count the upstream reports replaces the estimate.

What it does not affect:

  • The vector search. How well the embedding model represents Turkish decides that, and nothing in the store setting changes it. Try your own questions through the store’s search test before trusting it.
  • A pg_search store. The BM25 index is global and tokenised once by PG_SEARCH_TOKENIZER, so the store language is ignored there.
  • Mixed-language content. There is no language detection. One store has one language.

Changing the language of a store that has documents is a background job: the store is marked rebuilding, chunks.fts is rewritten in batches, searches keep working on simple meanwhile, and fts_status turns ready when it ends. It costs one more tsvector and index entry per chunk. See Store language.

Mixed Turkish and English: filter, don’t merge

A store that holds both languages should stay on simple, which indexes words as written. To keep a Turkish question from retrieving the English copy of the same policy, tag documents when you upload them:

metadata={"lang": "tr", "product": "gateway"}

and narrow the request:

{"model": "…", "messages": […], "ragmux": {"filter": {"lang": "tr"}}}

The filter is an object whose keys are ANDed; a value is a scalar matched by equality (and sensitive to type, so 2026 does not match "2026") or {"in": [...]}. There are no ranges, no or and no not. It is applied to each leg of the search before that leg’s LIMIT, so a selective filter still returns a full top_k when that many matching chunks exist. A request can only narrow what the project would do; a filter outside the grammar fails with 400 instead of silently widening the answer. See Metadata and filters.

Metadata edits are cheap: PATCH /admin/api/documents/{id} replaces the object in one row update, with no chunk rewritten and nothing reprocessed.

Follow-up questions: condense

“Peki yarı zamanlı çalışanlar için?” retrieves nothing useful on its own. With query_strategy: condense, a chat model receives the last few turns and the new message and writes one standalone search query, in the language of the last message. That query, not the raw follow-up, is what gets embedded and searched.

The behaviour worth knowing before switching it on:

  • It only runs when there is history. A request with a single user message is used as it is, with no extra call.
  • One attempt, five seconds, never fatal. On an error, a timeout or an unusable reply the last user message is used and the failure is logged.
  • It costs. The call adds to the request’s latency, and its tokens are added to the request’s row and counted against the project’s limits.
  • It sees the conversation. The text goes to the provider of the condense connection, as it does for the chat model. Where that connection runs matters if the conversation is sensitive; the on-prem data-flow page lists the places.

See Query strategy. A related setting, retrieval_policy: auto, skips retrieval for a bare greeting or acknowledgement with a rule-based classifier that knows Turkish and English phrases. “Merhaba, iade nasıl yapılır?” has words the classifier does not know, so it retrieves; and in a condense project a bare “evet” is not skipped, because the condensed query turns it into a real question. See Retrieval policy.

A worked example: a returns-policy assistant

A retailer keeps its returns policy in Turkish and English, as PDFs, one set per product line. The store is on simple because it holds both, each document is uploaded with {"lang": "tr"|"en", "product": "..."}, and the Turkish storefront’s project sends "filter": {"lang": "tr"} on every request through its own backend. The project uses condense with a small local chat connection, so “ya kargo ücreti?” after a question about returns becomes a question about shipping costs on returns.

The retailer then runs ten real customer questions through the store’s search test and reads which passage ranked first for each, before pointing the storefront at it. That check is the part no setting replaces: it is how a team finds out whether its embedding model and its chunk size suit its own documents.

If the retailer later keeps only Turkish content in the store, it sets language to tr, waits for fts_status to read ready, and gets stemmed matching without re-uploading anything.

Upgrading a store that had fts_config

If a store had a fts_config other than simple, its queries now parse with simple: stemmed matching stops until language is set (tr for turkish). Ragmux logs a startup warning for each affected store on the pgvector backend, naming the language to set. Saving the store with a language, even simple, silences it.

Where this doesn’t fit

  • Scanned documents. There is no OCR, so a scan without a text layer is rejected. See Private document Q&A.
  • One store, many languages, stemmed. A store has one language. If Turkish and English must each be stemmed, they need two stores and two projects.
  • ParadeDB with several languages. A pg_search installation tokenises once for the whole database, so it cannot stem per store.

Set up retrieval for Turkish documents

  1. Create the store with language set to tr

    POST /admin/api/rag-stores with language set to tr, next to embedding_connection_id. The language decides how chunks are tokenised for full-text search and how the query is parsed against them, so the two always agree. A store that already holds documents is switched by sending PUT /admin/api/rag-stores/{id} with its full current body minus fts_config (read it with GET first; fts_config is deprecated, and sending it next to a different language is a 400) and the new language, or with the dashboard's language selector; that starts a background rebuild.

  2. Upload documents with a metadata object

    POST /admin/api/rag-stores/{id}/documents as multipart/form-data with a metadata field such as {"lang": "tr", "product": "gateway"}. It applies to every file of that request, and PATCH /admin/api/documents/{id} replaces it later without reprocessing anything.

  3. Check the result with the store's search test

    POST /admin/api/rag-stores/{id}/search with a real Turkish question and read the hits. The response reports whether the language was applied to that search, which it is not for a vector-only search or while a rebuild is running.

  4. Turn on condense for conversational projects

    PUT /admin/api/projects/{id} with the project's full current body (read it with GET first; only the RAG fields keep their value when left out, the others do not) and query_strategy set to condense, or switch it in the dashboard. history_turns_for_query (1 to 10, default 3) says how many earlier turns the rewrite sees, and query_model_connection_id names a small chat connection, or null for the project's chat connection (the first route target when it has routes).

  5. Narrow a chat request with ragmux.filter

    Add "ragmux": {"filter": {"lang": "tr"}} to POST /v1/chat/completions. A filter outside the grammar is refused with 400 and code invalid_ragmux rather than ignored.

Frequently asked questions

Which value of language do I use for Turkish?
tr. It selects PostgreSQL's built-in turkish text search configuration for indexing and for parsing queries. The accepted values are simple and a fixed list of ISO 639-1 codes; a code outside the list is refused with 400.
Can one store hold both Turkish and English documents?
Yes, but keep its language at simple. Ragmux does not detect the language of a document, and a store has one language, so a mixed store indexes words as written, without stemming or stop words. Tag each document with metadata such as lang and filter on it at query time.
Does the store language change the vector search?
No. It governs the full-text half of a hybrid search and the token estimates. Vector similarity is decided by the embedding model the store uses, so whether it handles Turkish well is a property of that model, which the search test lets you check against your own questions.
Does language work with the ParadeDB backend?
No. The pg_search backend ignores the store language, because its BM25 index is global and tokenised once by PG_SEARCH_TOKENIZER. The store response reports language_applies_to_search as false in that case.
What happens to searches while a language change is being applied?
They keep working. The store reports fts_status rebuilding and is searched with simple until the background job finishes, so results stay valid but unstemmed. Plan for minutes on a large store.
Does condense add cost and latency?
Yes, one chat completion before the search, with a five-second budget. Its tokens are added to the request's row and count against the project's limits. If it fails, times out or returns something unusable, retrieval uses the last user message and the request still succeeds.
Does the greeting skip understand Turkish?
retrieval_policy auto uses a rule-based classifier for Turkish and English that skips a message made only of known greeting or acknowledgement phrases. A message that mixes a greeting with a question, such as Merhaba, iade nasıl yapılır?, still retrieves.

Arrow keys to move, Enter to open.