RAG, From Crayons to PhD

Retrieval-Augmented Generation combines retrieved evidence with a model’s response. The retrieval source can be documents, a database, a search service or another authorized system. Its usefulness depends on finding relevant material and using it correctly.

A RAG system can fail in ingestion, retrieval, permission handling or generation. Diagnose those stages separately; this article does not establish an industry-wide production failure rate or a universal share attributable to retrieval.

This article walks the same idea up five floors. Floor one starts with the basics. Floor five is the cutting edge. By the time you reach the top you'll know what RAG is, when to use it, when to run, every tool worth caring about, the alternatives worth considering, and the hot takes most posts won't tell you.

Floor 1 — Crayons (ELI5)

Imagine asking a really smart friend a question. Instead of trying to remember the answer from memory, they walk to your bookshelf, find the right book, open it to the right page, and read it before answering. That's RAG. We taught computers the same trick: look it up first, then talk.

TL;DR (engineer level)

Floor 2 — What problem RAG actually solves

Large language models have three brutal constraints. RAG addresses all three at once.

A model’s training data does not reliably include a private or newly changed source. Retrieval can supply that material at runtime, provided it is indexed or queried with sufficient freshness and permission controls.

Second, context windows are finite and expensive. Even with Gemini's two-million-token window or Claude's one-million, dumping your whole corpus into every prompt is wasteful: you pay per token, latency scales with input, and the well-documented "lost in the middle" attention degradation means the model often ignores the part you most need.

Retrieved evidence can reduce unsupported answers, but a citation is not proof that the cited passage supports the claim. Evaluate source relevance, entailment, omissions and the model’s ability to abstain.

RAG's answer: keep the model frozen, keep your knowledge external in a searchable form, and at query time retrieve only the relevant slices. The model becomes a reasoning engine over fresh context, not a stale knowledge store.

The trade-off is the part most tutorials skip. You're now running a search engine and a generation engine in series. Search engines are the harder half of that equation. Most teams underestimate this and ship cosine similarity over a default chunker, then wonder why their bot hallucinates inside a system that was supposed to ground it.

Floor 3 — How it actually works

The pipeline has five real stages. Each one has non-trivial choices. Each one is a failure point most teams don't monitor.

1. Ingestion and chunking. Parse or otherwise represent the source, then choose retrieval units that retain useful context. Section-aware chunks, parent-document retrieval and contextual descriptions are options to evaluate. Preserve provenance and access metadata rather than treating text splitting as the whole ingestion problem.

2. Embedding, when used. An embedding model maps content and queries into a retrieval representation. Compare candidate models on the language, document type and queries you actually need; their contribution is one part of the pipeline.

3. Indexing. Select indexes appropriate to the data and workload. Vector indexes support similarity retrieval; keyword indexes help with exact identifiers. Hybrid retrieval combines evidence from both, but it should earn its complexity in evaluation.

4. Retrieval. Query rewriting, rank fusion and reranking are optional stages. Compare them with a simple baseline, including the latency and cost they add, and measure which relevant sources reach the final context.

5. Generation. Stuff the top chunks into a structured prompt — system instruction, context window, question, citation format — call the LLM, parse the citations, return the answer.

Retrieval recall and answer correctness are different measurements. An answer can be wrong even when the right source was retrieved. Test changes to chunking, fusion and reranking separately before attributing a gain to the database or model.

The tool landscape (opinionated, 2026)

Five layers, picks that are actually shipping in production today, brief honest take on each. If you want a comprehensive list go elsewhere — this is what I'd reach for.

Vector databases

Embedding models

Frameworks and re-rankers

Practical retrieval use cases

These examples describe applications to evaluate, rather than asserting the private architecture of a named product. Each combines an authorized evidence source with checks appropriate to its failure modes.

Should I use RAG? A decision tree

Ask these in order. Don't skip ahead. Most projects that fail do so because they answered question 6 first.

Anti-pattern check before building: if you can't articulate your evaluation metric in one sentence, you're not ready to build the system. Walk back to a notebook and figure out what "good" looks like.

Alternatives — when each one wins

RAG is one option within a larger application. A system can combine retrieval, tool use, MCP interfaces and direct SDKs; those choices address different requirements. Compare the following approaches against the problem being solved.

These categories can be combined. Long context is not limited to prototypes, and an MCP server can expose the retrieval component of a RAG system. Evaluate the whole answer path rather than choosing by label.

QMD: RAG without the infrastructure tax

If RAG is the production architecture for serving end users via an LLM, QMD is what the same idea looks like when the user is you and the corpus is your own knowledge.

QMD is one local Markdown-search option. A local index still requires ingestion, index maintenance, model resources where used, and a decision about which agents may read the results. Local retrieval does not mean a connected cloud model never receives excerpts.

For a bounded Markdown corpus, begin with the simplest retrieval service that meets the requirements. Add distributed serving or more complex indexing when scale, availability or access requirements justify it. No universal setup duration or latency cutoff is established here.

Sinapt proposes a shared retrieval layer using related ideas. Its scale and operating model require separate validation; a local tool’s behavior is not proof of a company-wide service’s performance.

Floor 4 — Advanced (the production-grade techniques)

These techniques address different retrieval failure modes. Evaluate the ones relevant to your workload against a simpler baseline, including their operating cost.

Hybrid retrieval (keyword + dense + RRF)

Hybrid retrieval combines lexical and vector candidates, often with rank fusion. pgvector provides vector capabilities in PostgreSQL; a complete hybrid pipeline also needs keyword retrieval and fusion logic. Other products package different portions of this stack. Measure their contribution rather than promising a fixed accuracy lift.

Re-ranking with cross-encoders

A cross-encoder scores a query and candidate together, adding work after initial retrieval. Reranking can improve ordering, but the effect on NDCG, answer quality, latency and cost depends on the corpus, model and candidate set.

Late interaction (ColBERT, ColPali)

Late-interaction approaches compare multiple representations per item rather than only one document vector. Page-image retrieval can preserve visual layout, while text extraction remains useful for exact text and other workloads. Compare the two on representative documents rather than assuming one replaces every PDF pipeline.

Query rewriting and HyDE

Query rewriting or HyDE can improve retrieval on some workloads. HyDE generates a hypothetical answer and uses its representation for retrieval; that answer is a search aid, not evidence. Generation and embedding add work. Prompt caching may reduce eligible repeated-input costs, so measure the complete query path against an unchanged baseline.

Parent-document and small-to-big retrieval

Embed small chunks for precise retrieval, but return the parent document for rich context. Decouples retrieval granularity from generation context. LlamaIndex calls this auto-merging. Sentence-window retrieval is the same idea at finer granularity: embed individual sentences, expand to surrounding window after retrieval.

Contextual retrieval (Anthropic, late 2024)

Anthropic’s September 2024 experiment combined Contextual Embeddings with Contextual BM25 and reported a 49% reduction in top-20 retrieval failures, from 5.7% to 2.9%. Adding reranking produced a reported 67% reduction, to 1.9%. Those are vendor-reported results for the described evaluation, not guaranteed gains from adding a summary to any embedding pipeline.

Floor 5 — PhD level (the cutting edge)

Vanilla RAG is a 2023 architecture. The frontier in 2026 is a small zoo of variants, each solving a different production failure mode.

GraphRAG (Microsoft, 2024)

GraphRAG extracts entities and relationships and can build summaries over groups in the graph. This supports some corpus-wide questions, but extraction errors and ingestion cost need evaluation. A graph is one way to aggregate evidence; it does not make non-graph aggregation impossible.

Self-RAG (Asai et al., 2023)

The model is fine-tuned to emit reflection tokens that decide whether to retrieve, whether the retrieved passages are relevant, and whether its own generation is supported by them. Retrieval becomes adaptive instead of always-on.

Corrective RAG (CRAG)

A lightweight retrieval evaluator scores each retrieval as Correct, Ambiguous, or Incorrect. Correct uses the chunks. Incorrect triggers a web-search fallback. Ambiguous does both and lets the generator pick.

Agentic RAG

Agentic retrieval adds planning, repeated queries and intermediate evaluation. It can help when one query does not collect enough evidence, while also adding latency, cost and opportunities to loop. Compare it with a bounded retry or direct retrieval baseline; no general reduction percentage is established here.

Multi-modal RAG

Multimodal retrieval can use text, page images or other representations. Select the representation by the task: chart interpretation, exact text lookup and layout-sensitive questions may have different requirements. Preserve a way to inspect the original source.

Evaluation

Production-grade RAG without evaluation is vibes-driven AI. The frameworks that actually catch regressions:

Production failure modes (the ones you'll actually hit)

Hallucination despite retrieval. Right context retrieved, model still invents an answer. Cause: weak instruction adherence, citation format not enforced, top-K too high causing dilution.

Lost in the middle. Even with the right chunk in context, attention degrades for middle positions. Mitigation: re-rank so the best chunk is first or last; keep top-K small (3-5) for the final prompt.

Retrieval failures can remain invisible without logging and evaluation. Record enough provenance to distinguish missing sources, ranking failures, outdated content and incorrect generation; do not infer prevalence from an unsourced industry percentage.

Distribution shift. General-purpose embeddings may underperform on specialized corpora. Compare them with domain-specific alternatives on representative queries, including cost and operating requirements.

Chunk boundary loss. The answer spans the boundary between chunks N and N+1 and neither contains it whole. Mitigation: 10-20 percent overlap, parent-document retrieval, or contextual retrieval.

Permission leaks. A chunk from a doc the user shouldn't see ends up in their answer. Filter at retrieval time, never at generation time. This is the actual hardest part of enterprise RAG.

Stale index. Docs change, the index doesn't update, the model cites a deleted policy. Versioning + TTL + re-embed pipelines.

When NOT to use RAG

Hot takes nobody else will tell you

These are the things that take a few production deployments before they become obvious. Skip the learning curve.

1. Your vector database is the least important choice in your stack

Investigate chunking, retrieval and ranking before assuming a database migration will fix answer quality. The relative effect of each component is empirical; this article does not establish a universal five-to-tenfold ordering.

2. Cosine similarity has no semantics

A 0.87 score doesn't mean "relevant." It means "similar in this embedding space, which was trained on a particular distribution." Two completely irrelevant chunks can score 0.85. A perfect match can score 0.62. Stop treating similarity scores as confidence. Calibrate, or re-rank.

3. Long context did not kill RAG

Long context and retrieval are both valid production options. Compare cost, latency, source access, freshness and answer quality for the workload. A system can retrieve a broad document set and then use a long context to analyze it.

4. Most "agentic RAG" is just RAG with retries

A bounded retrieval retry may be enough for some tasks; others need multi-step planning. Choose through representative evaluation rather than an unsupported percentage of all use cases.

5. ColPali is eating PDF RAG

Page-image retrieval is useful when visual structure carries relevant information. It still needs evaluation of tables, charts, small text, multilingual content and source attribution. Text extraction and hybrid representations remain legitimate alternatives.

6. The hardest part of enterprise RAG is permissions, not retrieval

Permission-aware retrieval aims to respect source-system access before material is disclosed to the model or user. Treat that as a control to implement and test, including permission changes, derived summaries and caches; a product’s design claim is not a no-bypass guarantee. Plan authorization from the beginning.

7. Evaluation is harder than the system

Evaluation needs representative questions, reviewed reference evidence and regression checks. The effort depends on the workload and failure cost; a fixed weekend-versus-six-month comparison is not a planning estimate.

The bigger picture

RAG includes a range of techniques for different failure modes: hybrid retrieval for mixed lexical and semantic queries, reranking for candidate ordering, contextual chunks for missing context, page-image representations for visual documents, bounded retries for incomplete retrieval, and graphs for some relational or corpus-wide questions. These are options to evaluate, not a mandatory production stack. Keep the simplest combination that meets the workload’s requirements.

A useful retrieval system makes relevant evidence available and helps the generator abstain when it is missing. It does not eliminate every reason a model might guess.

This is the bet behind Sinapt — the agent-first knowledge layer I'm building. RAG is the per-application retrieval pipeline. Sinapt is the company-wide queryable layer underneath it: one source of truth that every agent in your stack can query, instead of stitching ten separate RAG pipelines that each rebuild context from scratch. The deep dive is in Sinapt and the Queryable Company. The product itself is at sinapt.ai.

Primary research for the scoped contextual-retrieval figures: Anthropic’s September 2024 methodology and results.