RAG Beyond the Basics: Advanced Retrieval Strategies

The RAG Reality Check

Retrieval-Augmented Generation (RAG) has become the go-to pattern for building AI applications that need access to private data. The basic setup is simple: embed your documents, store them in a vector database, retrieve relevant chunks, and feed them to an LLM.

But basic RAG hits walls quickly. Users ask questions that span multiple documents. Context windows fill up with irrelevant chunks. Answers miss crucial information that was "close but not quite" similar enough to retrieve.

Here's how to build RAG systems that actually work.

Understanding Retrieval Failure Modes

Before optimizing, understand why retrieval fails:

Semantic mismatch - The user's query uses different terminology than the documents. "How do I get reimbursed?" vs. documents that talk about "expense claims."

Context fragmentation - Relevant information is spread across multiple chunks that don't get retrieved together.

Recency blindness - Vector similarity doesn't understand time. The most relevant answer might be the most recent, not the most similar.

Specificity problems - Generic questions retrieve generic content, missing the specific answer buried in detailed documents.

Multi-Stage Retrieval

Single-step retrieval rarely performs well on complex queries. We use multi-stage approaches:

Stage 1: Broad Retrieval

Cast a wide net. Retrieve more documents than you'll ultimately use (top 50-100 instead of top 5-10).

Stage 2: Reranking

Use a cross-encoder reranker to score each candidate against the query. This is slower but much more accurate than embedding similarity alone.

Stage 3: Contextual Filtering

Apply business logic filters:

• Recency (prefer newer documents)
• Source authority (prioritize official docs over comments)
• Access control (user permissions)

Stage 4: Context Assembly

Don't just concatenate chunks. Structure the context intelligently:

• Group by source document
• Maintain document hierarchy
• Include metadata (dates, authors, document types)

Query Transformation

The user's query often isn't the best query for retrieval. Transform it:

Query expansion - Generate multiple phrasings of the same question. Retrieve for each and merge results.

Hypothetical Document Embedding (HyDE) - Have the LLM generate a hypothetical answer, then use that to retrieve. Often more effective than querying with the question directly.

Decomposition - Break complex questions into simpler sub-questions. Retrieve for each and synthesize.

Chunking Strategies

Default chunking (split by tokens or characters) is rarely optimal.

Semantic chunking - Split at natural boundaries (paragraphs, sections) rather than arbitrary token counts.

Hierarchical chunking - Create multiple chunk sizes. Retrieve at the appropriate granularity for each query.

Overlapping chunks - Include context from adjacent chunks to preserve continuity.

Metadata enrichment - Attach document structure (headers, section titles) to each chunk for better context.

Hybrid Search

Vector search alone has limitations. Combine approaches:

BM25 + Vector - Traditional keyword search catches exact matches that semantic search misses. Fuse results from both.

Structured + Unstructured - If your documents have structured metadata (dates, categories, authors), use SQL-style filtering alongside vector search.

Knowledge Graphs + Vectors - For complex domains, extract entities and relationships into a knowledge graph. Use graph traversal to find related concepts, then vector search within that subspace.

Evaluation and Iteration

You can't improve what you don't measure. Build evaluation into your RAG pipeline:

Retrieval metrics:

• Precision@K: Are retrieved documents relevant?
• Recall@K: Are all relevant documents retrieved?
• Mean Reciprocal Rank: Is the best document ranked first?

End-to-end metrics:

• Answer correctness (vs. ground truth if available)
• Answer groundedness (is the answer supported by retrieved context?)
• User satisfaction (implicit signals like follow-up questions)

Create a test set. 50-100 representative queries with known-good answers. Run it regularly to catch regressions.

Production Considerations

Caching - Cache embeddings, cache retrieval results for common queries, cache LLM responses where appropriate.

Latency - Optimize for perceived performance. Stream the LLM response while displaying retrieved sources.

Cost - Retrieval is cheap; LLM calls are expensive. Optimize context length. Consider smaller models for simple queries.

Monitoring - Log queries, retrieved documents, and generated answers. Build feedback loops for continuous improvement.

The Future of RAG

RAG is evolving rapidly:

• Agentic RAG - Agents that iteratively retrieve and reason
• Graph RAG - Combining knowledge graphs with retrieval
• Multi-modal RAG - Retrieving and reasoning over images, tables, and text together

The fundamentals matter most. Get retrieval right, and the rest follows.