LangChain RAG Implementation Patterns

Sequential RAG is exactly what it sounds like: embed the query, retrieve the top-k most similar chunks from a vector store, stuff those chunks into a prompt alongside the original question, and let the LLM generate a response. One linear pass, no feedback loops. In LangChain, this is a chain connecting retriever to prompt template to LLM to output parser, and it has fewer moving parts than anything else in this taxonomy.

That simplicity is valuable in the right context, and I've watched teams abandon it prematurely more than once. Stable, well-structured knowledge bases, internal wikis, product documentation, FAQs with predictable query patterns don't need sophisticated retrieval logic when the corpus changes infrequently and questions tend toward the concrete. When a single chunk typically contains the full answer, query transformation and agentic loops add latency without adding recall. Prototyping also demands exactly this kind of lean pipeline, because iteration speed matters more than precision when you're still figuring out whether the knowledge base is even structured correctly.

The failure modes are specific. Multi-hop questions that require synthesizing evidence across several documents break the single-pass assumption entirely; the pipeline physically cannot retrieve what it needs in one shot. Queries phrased in natural, idiomatic language often suffer from what retrieval researchers call the vocabulary mismatch problem: the correct document exists in the index, but the query vector doesn't land close enough to surface it. Stale knowledge bases compound both problems, because a frozen vector store cannot answer questions about anything that happened after its last index run.

Hallucination rates in production, across multiple benchmarks, span roughly 3% to 27% depending on domain and query complexity. That variance alone should give any team pause before deploying sequential RAG in a customer-facing or regulated context. It is a foundation layer, not a finished architecture. Every pattern that follows addresses one or more of these failure modes.

Advanced query transformations that close the gap between what users ask and what the index contains

The vocabulary mismatch problem is, in my experience, the most persistently underestimated source of retrieval failure. Users ask questions in natural language; the documents in the index were written in a different register, often more formal, more technical, or organized around concepts rather than questions. That semantic distance kills recall before the LLM ever sees the evidence. But what if the problem isn't the query itself, but the form in which it's sent to the index?

Three LangChain-supported transformation techniques address this from different angles.

HyDE: Hypothetical Document Embeddings

Rather than embedding the raw query directly, HyDE generates a hypothetical answer to the question first, then embeds that instead. The intuition is that a generated answer, even an imperfect one, lives in the same distributional space as the documents in the index. The original HyDE paper from Microsoft Research reported meaningful retrieval relevance improvements for conceptual queries, particularly in cases where the query and document registers diverge most sharply. The cost is a secondary LLM call before retrieval, which adds latency. On factual, narrow questions, HyDE adds overhead without much payoff; the trade-off is only favorable when the query distribution skews conceptual.

Multi-query retrieval

LangChain's MultiQueryRetriever rewrites the original query into several paraphrases, runs each against the index independently, then merges and deduplicates the results. The practical effect is broader coverage when the correct answer exists under multiple surface forms but none of them individually matches the user's phrasing. Token cost scales with the number of rewrite variants, which makes this a deliberate choice rather than a sensible default for all workloads.

Step-back prompting

Step-back prompting asks the LLM to identify the underlying principle or concept embedded in a specific question before retrieval runs. The logic is that a narrow, particular question may only be answerable by retrieving a document pitched at a higher level of abstraction. The relevant chunk is one step back from where the user's query literally points.

These three techniques operate at the chain level. The vector store and embeddings stay unchanged; only the query surface changes. Collectively they tend to improve accuracy over naive retrieval, though the exact lift depends heavily on domain and query distribution, and benchmarks like RAGAS or TruLens are useful for quantifying that gap in a specific application.

What they cannot solve is absence. If the right document isn't in the store, no query transformation will find it.

Routing-based retrieval for pipelines that draw from more than one data source

A single vector store is not the right retrieval path for every query, and pretending otherwise creates a class of failures that query transformation cannot fix. Some questions need current web data. Some need proprietary internal documents. Some need both. Routing is the architectural answer, and it's one of those design decisions that feels obvious in retrospect but takes a few production surprises to actually internalize.

In LangChain, a router component analyzes the incoming query and selects the retrieval path: local vector store, live web search, or a composite of both. The vector store path handles stable internal knowledge; the web search path fetches information not yet in the index; the composite path combines both when the answer requires internal context plus external freshness. The router itself is typically a lightweight LLM call or classifier. In LangGraph this logic is a conditional branch in the graph; in LCEL it's a runnable branch.

Choosing a live-search API for the web branch involves real trade-offs worth spelling out. Tavily is the most common choice in LangChain tutorials, offers structured results, and has a free tier at 1,000 requests per month, which suits prototyping but has limited customization for production workloads. Google Custom Search offers 100 free queries per day and then charges per additional thousand; its results lack the dense excerpts LLMs work best with.

One dimension most developers skip during prototyping: APIs that return source URLs alongside excerpts enable citation propagation through the pipeline. When a user asks where an answer came from, a pipeline that never captured provenance has no credible answer.

Routing also keeps latency manageable by design. Queries answerable from the vector store skip the web call entirely; only queries requiring freshness pay the network cost. The production hygiene this pattern demands, caching, error handling for failed web calls, token budget management when two sources return results simultaneously, is overhead that naive RAG encounters less often. Better to reckon with it before the first deployment than after.

Agentic RAG with LangGraph: retrieval loops that self-correct

The patterns above all execute a fixed sequence. Agentic RAG introduces something qualitatively different: the agent grades its own retrieval result and decides whether to answer, rewrite the query, or try a different source. The loop is the defining feature, and it changes the failure profile of the system in ways that matter.

The basic cycle: the agent receives a question, selects a knowledge base, retrieves documents, grades their relevance. If the documents are relevant, it generates and returns an answer. If they are not, it rewrites the original question and re-retrieves. That rewrite-and-retry cycle handles cases where the first retrieval attempt returns tangential results, which in complex domains is a meaningful fraction of queries, not an edge case.

LangGraph is the right tool here because standard LangChain chains are directed acyclic graphs: they cannot loop back. LangGraph models state as a graph that can cycle, which is exactly what rewrite-and-retry requires. State is explicit and inspectable at each node, which makes debugging tractable in a way that opaque chain execution does not. I've spent enough time staring at failed chain outputs with no visibility into intermediate state to appreciate why that distinction matters.

A grader subagent adds another layer. Configured with a rubric that specifies what counts as a grounded response, it evaluates whether the final output is actually derived from the retrieved source material before returning to the user. This shifts hallucination detection from post-hoc monitoring to in-loop enforcement, which is a meaningful architectural difference in regulated environments.

The latency cost is real. Each iteration of the loop adds time, and a loop that runs four times costs four times the compute. For research-grade or regulated applications where answer quality is paramount, that cost is often defensible. For sub-second chat interfaces, it requires deliberate scoping of which query types get routed through the agentic loop and which follow a faster path.

At the outer edge of this pattern, multi-agent orchestration distributes specialized retrieval tasks across parallel agents. A financial research application might run parallel agents over market data, regulatory filings, and historical knowledge, then synthesize the results into a single cited response. This mirrors production architectures like FinDebate (2025), which generates institutional-grade financial reports through a collaborative debate mechanism over domain-specific RAG systems.

Hybrid retrieval: combining dense and sparse search to raise the ceiling on recall

Dense vector search is not sufficient on its own. This becomes obvious fairly quickly in production, and it tends to surface in the worst places: a query containing a specific product code, a regulatory citation, or a proper noun that the embedding model treats as semantically similar to several other things. Keyword search handles exact-match terms trivially. That raises an important question: if each approach covers what the other misses, why not combine both signals?

Hybrid retrieval does exactly that. Dense retrieval via Chroma, Pinecone, Weaviate, or similar handles semantic similarity. Sparse retrieval via BM25 or TF-IDF handles lexical matching. Scores from both are fused, typically through Reciprocal Rank Fusion, so the final ranking reflects semantic and lexical relevance simultaneously.

Embedding model selection compounds the effect in ways that are easy to underestimate. Voyage-3-large outperforms OpenAI's text-embedding-3-large by a substantial margin on standard retrieval benchmarks, with a 32K-token context window versus 8K for OpenAI's model, which matters for long documents that would otherwise require forced truncation. Semantic chunking, which respects document structure rather than imposing fixed-size cuts, also improves recall over fixed-size chunking in controlled comparisons. Neither choice is exotic; both require deliberate selection rather than accepting defaults.

Reranking adds a second-pass filter. After initial retrieval, a cross-encoder reranker rescores the top candidates for relevance to the specific query. In precision-sensitive domains like legal or compliance work, the added latency, typically 50 to 100 milliseconds, is rarely a serious objection.

In LangChain, EnsembleRetriever combines multiple retriever types cleanly. Rerankers slot in as a post-retrieval step before the prompt is assembled. A useful working frame for benchmark targets: precision@k of at least 0.85 for regulated content, 0.75 for general knowledge work, and 0.65 for exploratory queries. These reflect where answer quality tends to degrade meaningfully in practice, not arbitrary thresholds.

Evaluating which pattern is working and catching regressions before users do

Evaluation is not optional, and the tooling situation is shakier than most teams assume going in. Cleanlab's 2025 benchmarks found that RAGAS failed on 83.5% of production examples and DeepEval on 58.9%. Those numbers should land with some weight: the most widely used automated detection tools are themselves substantially unreliable. One might argue that running multiple evaluation frameworks in parallel is overkill — but combining multiple evaluation signals is a necessity given the current state of the tooling, not a belt-and-suspenders luxury; it is the most defensible strategy available.

Two layers of metrics matter in practice. Retrieval quality metrics, precision@k, recall@k, MRR, nDCG, tell you whether the right chunks are reaching the prompt at all. Generation quality metrics, faithfulness, hallucination rate, citation coverage, tell you whether the LLM is staying grounded in what it was given. End-to-end metrics, correctness, factuality, latency, cost, safety, are what a product owner or compliance reviewer actually cares about. Teams that skip the retrieval layer often spend weeks debugging generation failures that are actually retrieval failures in disguise. It's a frustrating way to learn the lesson.

Each pattern surfaces its own evaluation concerns. Sequential RAG requires tracking retrieval precision carefully, since there is no reranking safety net. Agentic RAG requires logging every rewrite-and-retry cycle to detect runaway loops. Routing-based pipelines require auditing which branch is selected per query type to confirm the router is behaving as intended, and comparing answer quality across branches to verify the web path isn't introducing more noise than signal. Hybrid retrieval requires comparing precision@k before and after adding the sparse component to confirm that fusion is adding signal rather than diluting it.

LangSmith provides production-grade observability for LangChain pipelines: per-step latency, token counts, retrieval results, all inspectable. For debugging which step in a multi-hop chain degraded, it is close to non-optional.

The baseline question every team should answer before deploying is this: across a representative test set, does this pattern improve over naive RAG on the metrics that matter for this application? RAG consistently improves accuracy over unaugmented LLM baselines across domain-specific evaluations, regardless of base model. That establishes a floor. If a chosen pattern fails to clear it, something in the pipeline is broken, and it's worth finding before users find it first.

Data distributions shift. Knowledge bases evolve. Queries that were rare become common. The patterns that hold on launch day may not hold six months later. Evaluation infrastructure, not retrieval architecture, tends to be the thing that actually keeps a production system honest over time. It's unglamorous to build and nearly impossible to argue for in a sprint review when the retrieval pipeline is already "working." It's also the only early warning system you have.