In a previous post, I covered what Retrieval-Augmented Generation is and how to prepare data for ingestion. A companion post on the ingest pipeline walks through the data cleaning techniques that get content into the vector store. This post picks up where retrieval begins.

Ingesting documents into a vector database is only half the problem. The other half is what happens when someone types a question: understanding the query, ranking results, validating citations, and handling failures along the way.

Query handling

User input can contain control characters, excessively long text, or prompt injection attempts. Before the query reaches any LLM or embedding model, three layers of sanitization run:

• Control character removal: strip everything except newlines and tabs.
• Prompt injection mitigation: regex-based detection of patterns like “ignore previous instructions” or “disregard system prompt.” Matched patterns are stripped before the query reaches any model.
• Length truncation: queries are capped at 500 characters for classification and topic extraction (4000 for full content extraction).

Even though the LLM system prompts include their own guardrails, sanitizing at the boundary prevents entire categories of attacks from reaching the model.

Intent classification

Once the query is clean, the next question is: does it even need the RAG pipeline?

A four-way intent classifier runs before any retrieval:

A regex fast-path catches common greetings before the LLM classifier runs. The classifier uses temperature=0.0 for deterministic routing, so identical queries always get the same intent. If the classifier fails entirely, the system defaults to factual_query so search is never blocked by a transient error.

In typical usage, 15-25% of messages are greetings or acknowledgments. Routing them correctly saves 200-400ms and $0.001-0.01 per message by skipping embedding generation, vector search, and reranking.

Search and retrieval

With a confirmed factual_query, the system needs to find relevant content. This happens in several coordinated steps.

Topic-based filtering

Searching the entire vector space for every query produces noisy results. A question about cargo damage should not return chunks about engine maintenance just because they share vocabulary.

Topics are extracted from the query and matched against the corpus topic ontology using cosine similarity. The default mode is embedding-only: the query embedding is matched directly against topic embeddings, skipping an LLM extraction step and saving ~300ms per query.

Because the topic ontology is fragmented (median chunk count per topic is 1, concepts span ~7 synonymous topic rows), the matcher returns top-K closest topics per name rather than a single winner, pooling fragments into effective clusters.

If topic-filtered results return fewer chunks than a threshold, the filter retries with a looser similarity threshold. If still too sparse, the topic filter is dropped entirely while structural filters (like vessel metadata) are preserved.

Vessel metadata filtering

Fleet operators frequently need results for a specific vessel, type, or class. These are structural metadata filters applied at the database level before vector similarity: exact, not probabilistic. When an operator asks about a specific vessel, they get results only from that vessel.

Vessel names are resolved with fuzzy matching against a cached vessel list, and whole-word regex auto-detects vessel types and classes mentioned in the query text. Active filters persist per conversation, so follow-up questions inherit vessel context.

Hybrid vector + BM25 search

Pure vector similarity misses exact keyword matches (part numbers, IMO numbers, incident codes). Pure keyword search misses semantic meaning. The system combines both:

• Vector search via HNSW index (pgvector) with cosine similarity. hnsw.ef_search = 100 (up from the default 40) trades a few milliseconds of latency for 5-10% recall improvement.
• Optional BM25 scoring boosts results that match query keywords exactly.
• A similarity threshold (default 0.3) discards low-confidence results before reranking.

This catches both semantic matches (“propulsion system failure” matching “engine breakdown”) and exact matches (“IMO 9876543” matching only that specific number).

Thread digest retrieval

Broad queries like “any engine issues last month?” need to match content spanning an entire email thread. Individual message chunks may contain only the problem or only the resolution.

During ingest, multi-message threads get a synthesized digest chunk (200-400 words) that captures the full problem-investigation-resolution arc. At search time, this digest competes alongside regular message chunks in vector similarity. A query like “how was the cylinder crack resolved?” might match the resolution message chunk poorly (it mentions “replaced liner” without context), but the digest chunk matches well because it contains the full narrative.

Reranking

After initial vector retrieval returns ~40 candidates, a cross-encoder model (Cohere rerank) scores each candidate against the query. Cross-encoders process the query and document together rather than encoding them separately, and are 20-35% more accurate than bi-encoder similarity alone.

Before reranking, each document is prefixed with its email subject and source title. This helps the reranker distinguish between chunks from different incidents that have similar technical content.

Results below a rerank score of 0.2 are filtered out, but at least one result is always returned to prevent empty responses. If the reranking API fails, the pipeline falls back to unranked vector results with a warning log.

In practice, reranking made the biggest difference to answer quality. The LLM attends most to the first chunks in its context, so getting the order right matters more than retrieving a few extra candidates.

Building the LLM context

At ingest time, each attachment is classified into one of three embedding modes (full, summary, or metadata_only). This classification carries through to how context is built for the LLM.

In full mode, a context summary is prepended to the chunk text, separated by ---. The LLM sees both document-level context and the specific passage. In summary mode, only the context summary is emitted, since the chunk content is already a synthesized summary and repeating it wastes context window. metadata_only emits just the filename and basic metadata.

Each search result also gets a structured citation header:

[CITE:chunk_id | title:"Cargo Report..." | subject:"RE: Vessel Inspection..." | date:2024-01-15 | from:captain@... | page:5]

Only the chunk ID is exposed as an identifier, no doc_id or source_id to confuse the model. Title is capped at 50 characters, subject at 40.

When multiple results come from the same email thread, they are grouped by root email with an incident summary. The LLM sees “3 incidents found, 12 relevant passages total” rather than a flat list, so it can piece together the timeline of an incident from multiple chunks.

Citations and trust

The LLM generates inline [C:chunk_id] markers, which then go through a validation pipeline:

• Regex extraction captures citation markers (tolerating whitespace variations).
• ID validation confirms each chunk ID is exactly 12 hex characters.
• Deduplication collapses repeated citations of the same chunk.
• Archive verification concurrently checks that each referenced file exists in storage, running in parallel so verification takes max(times) instead of sum.
• If more than 50% of citations are invalid, the response is flagged. Invalid markers are stripped from the response text.

On the frontend, citations render as numbered badges with hover previews. When the data-citations SSE frame arrives mid-stream, raw [C:hex] markers are replaced with full HTML tags containing metadata, source title, page number, and download URI. A 3-second fallback ensures citations still render even if the validation pipeline is slow.

Users can click a citation badge to see the original email, the highlighted passage, and download the original PDF. Citation detail endpoints validate chunk IDs at the API boundary (12-char hex format check) before any database query runs.

For stored conversations, the citation payload is persisted alongside each assistant message. When a user returns days later, the same rendering function produces identical results: working links, accurate source references, same visual treatment.

Reliability

The search pipeline depends on several external services: embedding API, vector database, reranking API, LLM provider. Any of them can fail transiently. Every stage has a fallback:

Error messages that reach the user are sanitized: Bearer tokens, API keys, and database connection strings are stripped before display.

LangGraph agent checkpoints can also contain interrupted tool calls (the LLM requested a search, but the response was never received). Before each LLM call, the message history is scanned and orphaned calls are filtered out. Without this, a single interrupted request would make the entire conversation unusable.

Prompt caching reduces latency on follow-up messages by 40-60%. The system prompt is marked as a stable prefix for caching; dynamic content (vessel context, intent-specific instructions) is placed after conversation history so it does not invalidate the cache.

Observability

Without visibility into search quality, degradation goes unnoticed. Bad results do not produce errors; they produce silent user dissatisfaction.

Every search pipeline step is traced with Langfuse: session ID, user ID, intent classification result, topic filter decisions, retrieval count, rerank scores, generation output, and citation validation outcome. Users can submit thumbs-up/thumbs-down on each response, recorded as a Langfuse score linked to the specific trace.

Low-rated responses can be investigated by examining the full pipeline trace: which topics were extracted, which chunks were retrieved, what the reranker promoted, and what the LLM generated. If you do not have this kind of tracing, quality issues stay invisible until someone complains.

Citation detail endpoints validate chunk IDs before querying the database. Archive file serving rejects path traversal attempts. All endpoints have per-user rate limits (30/min for search, 100/min for citations) to protect upstream API budgets. Download URLs use pre-signed links with 300-second expiry rather than open storage access.

What held up

Reranking was the biggest quality improvement: a cross-encoder examining query and document together was 20-35% more accurate than embedding similarity alone. Topic filtering and metadata filters improved precision without sacrificing recall, because the loosening fallback catches edge cases.

I did not expect citation validation to matter as much as it did. A few dead links in an otherwise good answer made users distrust the whole system. Verifying every citation against the source archive before it reaches the user is worth the added latency.

The ingest pipeline gets data in. The search pipeline gets the right data out. Both need the same care around data quality, just at different points, and on the search side the user is watching in real time.