Building a Custom Eval Suite for Search-Augmented LLMs

A car-safety rating tells you nothing about how an airplane handles turbulence, because the two machines fail in different places and for different reasons. The same mismatch applies to evaluating search-augmented language models with the metrics built for static, closed-book LLMs: those metrics were designed to measure what a model knows, not whether it searched well, found the right documents, and said something those documents actually support. This piece lays out what a purpose-built eval suite for search-augmented LLMs needs to measure, why the standard shortcuts (benchmark leaderboards, lexical metrics, ungrounded LLM judges) cannot substitute for that suite, and which tools and human review processes actually close the gap.

Why Standard LLM Eval Metrics Fail When Retrieval Enters the Loop

Standard LLM eval metrics were built on a premise that collapses the moment retrieval enters the picture: the model's output is treated as the only thing worth measuring. That premise ignores the step that produced the grounding context in the first place, and if the retrieval step pulled back the wrong documents, stale documents, or no documents at all, no amount of scrutiny applied to the final answer will catch the actual point of failure. Lexical-matching metrics like BLEU, ROUGE, Exact Match, and F1 compare a generated answer against a fixed reference answer, which works reasonably well when the correct answer is stable over time. It breaks down when the correct answer changes as the live web changes: yesterday's reference text can describe a stock price, a regulatory status, or a news event that is no longer accurate today.

When lexical metrics got too rigid, many teams turned to LLM-as-judge evaluation, but it has its own failure modes if it runs without external grounding. Each of these failure modes is invisible to a team that only looks at the judge's final score.

None of this is theoretical. Apple had to pull its AI news summary feature because it made up headlines. Each of these failures traces back to a system that produced fluent, confident output, because no one had built the evaluation infrastructure to catch what was wrong in the retrieval and synthesis steps that produced it.

What makes the retrieval loop distinct from a standard generation task is that it introduces two separate places a system can fail, not one. The query a system sends to search can be malformed, vague, or under-specified in a way that returns low-signal documents before generation even starts. Separately, the model's synthesis of whatever documents came back can be unfaithful to those documents or selectively cite only the parts that support a predetermined answer. Standard output-only metrics miss the first failure surface entirely, because they never look at the query that was issued. They catch the second only partially, because a lexical match to a reference answer says nothing about whether the generated text actually tracks what the retrieved sources said.

The four dimensions a search-augmented eval suite must measure

So you need to treat retrieval-augmented evaluation as four separable failure modes, each with its own metrics, rather than folding everything into one quality score. Retrieval faithfulness asks whether the synthesized answer stays within the bounds of what the retrieved documents actually say, and it is a distinct question from factual correctness: a response can be faithful to bad documents, just as easily as it can confabulate claims that go beyond good ones. The FACTS Grounding benchmark, built by Google DeepMind and Google Research, operationalizes faithfulness at scale by labeling a response "accurate" only if every claim in it is directly supported by the provided documents or simply doesn't require document support, with anything else labeled "inaccurate," and it averages its final scores across three independent LLM judges specifically to reduce the kind of individual judge bias the SAGE paper documents. SAGE itself approaches the same problem from a different angle, iteratively generating web queries, collecting information, summarizing findings, and refining its own evaluation through reflection, so that the judgment of whether a candidate answer is grounded rests on externally retrieved evidence rather than on the model's own parametric memory.

Citation accuracy asks something narrower and just as consequential: are the sources a system cites actually the sources its claims derive from? Citation hallucination is a plausible-looking reference that doesn't actually support the stated claim, and it's a distinct failure from answer hallucination, so a system can score well on faithfulness while still misattributing individual claims to the wrong document. Temporal freshness asks whether the retrieved content is current enough for what the query needs, because once a fast-moving domain has moved past a document, an answer built from it can be factually correct and still be the wrong answer. Finance sharpens this dimension better than almost any other domain: You.com's Finance Research API was built around exactly this pressure, combining live web intelligence with licensed structured financial data from providers like S&P Global and adding evidence reconciliation across sources that conflict with each other, because stale or unreconciled financial data carries direct regulatory and financial liability. Taken together, these four dimensions form the spine the rest of this framework hangs on: query quality governs what gets retrieved, faithfulness governs how it gets used, citation accuracy governs whether credit is assigned correctly, and freshness governs whether any of it was still true by the time it reached the user.

Why benchmark scores on standard leaderboards cannot substitute for this framework

A reasonable objection at this point is to ask why a team would need to build any of this from scratch when public benchmarks already exist. Benchmarks like MMLU are structurally compromised as proxies for search-augmented system quality, for two separate reasons that compound each other. The second is scope: MMLU and benchmarks like it measure static parametric knowledge, what a model learned during training, not live-web retrieval behavior, so a high score tells a developer nothing about whether the system can issue a good search query or reconcile two conflicting sources published an hour apart.

Prompt sensitivity compounds the problem further. Benchmarks built specifically around web retrieval make clear just how far the field still has to go: BrowseComp-zh, a set of 289 multi-hop questions spanning multiple domains, sees most models score in the single digits, and even the best published 2025 system falls well short of half marks. So a generic leaderboard result can't tell you much about how a specific domain-retrieval system will perform once it is actually deployed.

So the practitioner response that has emerged from this isn't to chase broader benchmarks, but to build domain-specific evals gated directly to CI/CD. None of these replace a custom suite. They complement it: a team gets a rough outside comparison while the real regression testing happens against its own data. That leaves the question of tooling as a practical one rather than a theoretical one: once a team accepts it needs a custom suite, which tools actually cover which layer of it?

Choosing the right evaluation tools for each layer of the retrieval-response loop

No single tool on the market today covers all four dimensions well, so if you treat selection as a decision tree rather than a ranking, you avoid the tool sprawl that comes from picking whatever is best known rather than what a given layer actually requires. The question at each branch is the same: which failure surface is being targeted right now?

For retrieval precision and recall specifically, RAGAS is purpose-built for retrieval-augmented systems, and you need the least configuration to get meaningful results from it, because its metrics map directly onto RAG failure modes rather than general-purpose LLM quality. For broader automated metric coverage, including the kind of custom criteria a citation-accuracy or freshness check requires, DeepEval offers more than 14 built-in metrics spanning hallucination detection, answer relevancy, contextual precision, contextual recall, faithfulness, toxicity, and bias, along with G-Eval, a customizable LLM-as-judge framework that lets a team write evaluation criteria in plain English and have it translated into a scored rubric. This is where domain-specific citation and freshness criteria tend to get operationalized in practice. Confident AI wraps DeepEval's open-source metrics into a platform with more than 50 research-backed metrics, cross-functional workflows that let product managers and QA staff run eval cycles without engineering involvement through HTTP-based connections, CI/CD integration, and production-to-eval pipelines that curate live traces into evaluation datasets automatically.

For CI/CD-gated regression testing specifically, Braintrust offers score history over time, diff views, an annotation interface, and a native GitHub Action, so that every production failure a human reviews becomes a permanent test case running in CI/CD, one that cannot ship again without triggering an alert. For LLM-as-judge faithfulness grounding at benchmark scale, the FACTS Grounding design, three independent frontier LLM judges with scores averaged to mitigate any single judge's bias, is the model worth following when building a custom judge for citation accuracy. None of these tools is a complete answer on its own. The point of the decision tree is that tool sprawl is avoidable only when a team is deliberate about which layer each tool is actually responsible for.

Adding human-in-the-loop review for the dimensions automated scorers cannot reach

Automated scorers handle the overwhelming majority of eval volume once a suite is built, but three tasks in that process cannot be delegated to automation at all: discovering what to measure in the first place, building the labeled data that automated scorers are calibrated against, and catching quality dimensions that resist formal specification no matter how the rubric is written.

Discovery has to come before automation, not after it. Before any scorer can be built, a human needs to review a sample of production outputs and notice what is actually going wrong: a tone failure here, a domain-specific citation error there, a freshness gap on a topic that moves faster than the eval suite anticipated. Only after that discovery step can those failure patterns get formalized into a metric a scorer can check for automatically. Consider a finance analyst reviewing a small sample of earnings-filing extractions each week, flagging the handful of cases where an automated scorer missed a misattributed citation or let a stale filing slip through as current. Those flagged cases don't just get corrected and discarded. They feed back into the eval dataset, becoming the examples future scorers are calibrated against.

That calibration data matters because scored examples from human reviewers become the golden dataset that automated scorers and LLM judges are measured against, and if you don't have that ground truth, you end up optimizing for a metric that no longer reflects actual output quality. Some dimensions resist automated scoring even with good calibration data behind them: a response can be faithful to its sources yet miss the point of what the user actually asked, or a financial answer can be technically accurate while citing a source that doesn't actually support the specific claim attached to it. Catching that kind of mismatch takes domain-expert judgment, because no matter how carefully you write a scored rubric, it cannot fully encode it. Rhesis AI addresses this with a collaboration layer built specifically for agent development, where domain experts annotate agent outputs and test cases through a no-code interface at the result, metric, or turn level, and engineers then turn those annotated requirements into single- and multi-turn test suites scored with built-in and custom metrics.

Scorer drift is the maintenance problem that follows from all of this. A scorer stops capturing current edge cases over time, even a well-built one, because the system it evaluates changes and the domain it covers keeps moving. Braintrust surfaces human-versus-judge disagreement on the same row of data, so drift is visible immediately rather than months after a scorer has quietly stopped measuring what it was built to measure.

Finance as the proving ground for citation accuracy and freshness eval

Finance exposes every weakness in a search-augmented eval suite faster than almost any other domain, because errors in citation accuracy and temporal freshness carry direct regulatory and financial liability, not just the risk of embarrassing a brand. That makes it the clearest model for how any high-stakes retrieval application ought to think about evaluation.

The eval challenge in finance is not just accuracy, but source reconciliation. Financial data arrives from multiple sources that frequently conflict with each other and that carry hard timestamps, so a response that was accurate as of one filing date can be wrong as of the next one, and an eval suite built for this domain has to be able to distinguish a response that is simply wrong from one that is merely outdated. The architecture behind You.com's Finance Research API, a unified data layer combining live web intelligence with licensed structured financial data and evidence reconciliation across conflicting sources, is itself a useful design pattern for how a finance eval suite ought to assess retrieval quality in the first place: not just whether the model found a source, but whether it reconciled conflicting sources correctly when they disagreed.

Other production systems show adjacent patterns, so you need to understand them on their own terms. Linkup, used in production by McKinsey, SNCF, and Cohere, runs an asynchronous deep research pattern through a dedicated research endpoint that conducts multi-step web investigation, synthesizes findings across sources, and returns cited, structured output. The eval challenge specific to that pattern is verifying that the cited sources actually support the synthesized claims attached to them, not simply confirming that citations are present somewhere in the output. Valyu takes a different approach, indexing more than 36 proprietary data sources spanning regulatory filings, peer-reviewed research, patents, clinical trials, financial market data, legal records, and life sciences, and returning parsed, structured, LLM-ready results. The eval implication there is that retrieval precision across that many heterogeneous source types demands source-type-aware evaluation, since a metric tuned for regulatory filings will not necessarily catch an error in a clinical trial record, and a suite built for one source type alone will miss exactly the failures the domain's breadth is most likely to produce.