Building an Internal Search API Eval Suite

A search API eval has to test a pipeline where at least three components can fail independently: the retriever can miss documents, the generator can misrepresent what it retrieved, and the whole system can return content that's simply out of date. Blend those into one score and the suite lies to you in a specific, predictable way, and most teams do exactly that because a single dashboard number is easier to report up the chain. This piece breaks down how to build an internal eval suite that catches each failure mode on its own terms.

Here's the failure mode in concrete terms. A legal research RAG system ships with a faithfulness score of 0.91 on its offline eval set, a number that looks strong by most standards. Three weeks later, one in six customer responses is missing a key statute. The dashboard still reads 0.91 faithfulness. Context recall, a metric nobody was watching closely, has quietly dropped to 0.62. What happened: the retriever started dropping the second statute on multi-hop questions, and the generator, doing exactly what it was built to do, answered coherently from whatever partial context it received. Faithfulness measures whether the answer is honest about its sources; whether the sources were the right ones to begin with is a question faithfulness never asks. Sit with that gap before getting to the mechanics of building something better, because it's the reason single-score evals fail in the first place.

The four dimensions every search API eval suite must cover

Retrieval quality, answer grounding, freshness, and latency: four dimensions, and they move independently of one another. A system can nail retrieval and still hallucinate. Grounding can be perfect while the documents underneath it are stale. Speed and correctness don't travel together either, so a system can be fast and wrong, or slow and right. The legal RAG case above shows grounding looking fine while retrieval quietly broke; the mirror failure, where retrieval is flawless but the generator adds claims nothing in the context supports, is just as common and just as invisible when only one number sits on the dashboard.

Retrieval quality asks whether the system surfaced the right documents, in a sensible order, with enough coverage of the question. Grounding asks whether the generated response actually reflects what got retrieved, with no claims invented along the way. Freshness asks whether what got retrieved is current enough to reason from correctly, which matters enormously in finance, news, or anything regulatory. Latency asks whether the whole thing runs fast enough for production, and whether that speed holds up under the query types that actually matter for the workload in question.

Worth pausing on a distinction that gets blurred constantly, and confusing it is where a lot of eval design goes wrong from the start: a benchmark and a metric do different jobs. A benchmark is a fixed dataset paired with a scoring protocol, built so different providers can be compared on equal footing. MMLU, SWE-bench, BrowseComp: these are benchmarks. A metric is a rubric run against a team's own data, built to catch a regression in a specific pipeline before it reaches a customer. An internal suite needs both. Benchmarks say which provider to pick; metrics say when the provider already picked has started to drift.

Agentic workloads sharpen every one of these four dimensions at once, and this is where the choice of retrieval strategy stops being a nuance and starts being decisive. Standard LLMs relying on basic keyword search score below 10% on complex multi-hop research benchmarks. Systems built around iterative retrieval, meaning search, reason, search again, score dramatically higher on the same tasks. Plain keyword search simply isn't built for multi-hop agent work, and the numbers say so plainly. The reason is mechanical: agents call search tools in loops, and a single stale or misretrieved result at step one propagates through every reasoning step that follows. A human skimming a results page can infer around a bad result; an agent reasons from exactly what the API hands back, with far less room to fill gaps intuitively. Precision requirements go up the moment a human stops sitting in the loop to catch the obvious errors.

Designing the test dataset before writing any eval code

Everything downstream depends on the test set. A weak dataset produces scores that look confident and mean nothing, because they don't transfer to what actually happens in production. Before any eval code gets written, the dataset needs a taxonomy that matches the query types the agent will actually see, chosen for that reason rather than for ease of writing.

That means covering factual lookups that need a single hop, multi-hop questions that require synthesizing across several retrieved documents, time-sensitive queries where the right answer changes depending on when you ask, ambiguous or underspecified queries that test how the system handles vagueness, and adversarial prompts built to see whether the retriever or the grounding layer can be misled. Five categories, and each tests something the others miss entirely.

Hard negatives deserve specific attention, and most teams skip this step entirely, which is a mistake. Include queries where a plausible-but-wrong document exists in the corpus and must not get ranked first. This is where retrieval systems tend to fail in practice: confidently surfacing something that reads as relevant and isn't, rather than missing obvious junk. A dataset without hard negatives will never catch that failure, no matter how many easy queries it contains.

Ground truth has to be established carefully, and differently, for each dimension. Retrieval needs a gold document set per query, so recall and precision can be computed against something real. Generation needs a reference answer per query, so faithfulness and answer relevance have a target to check against. Freshness needs a "correct as of" date annotation on each query, so a test can catch a stale result even when that result would otherwise look perfectly relevant.

Public datasets are a sensible place to start rather than build from scratch. Several open datasets cover factual lookup, multi-hop reasoning, and time-sensitive queries respectively; selecting a combination that maps directly to the query categories outlined above is a reasonable starting stack. Public benchmarks alone will miss the edge cases specific to a given workload, so domain-specific queries pulled from actual production traffic need to sit on top of that base. On sizing: big enough that scores don't bounce around from run to run, small enough to run in continuous integration without turning into a cost problem. Start curated, then grow the set as real regressions in production reveal gaps nobody thought to test for.

Measuring retrieval quality: the metrics that catch what faithfulness misses

Four metrics do most of the work here, and each catches something the others don't. Precision@K asks, of the top K results returned, what fraction are actually relevant; it catches noise and junk results the API is surfacing. Recall@K asks, of all the relevant documents that exist, what fraction show up in the top K. This is exactly the metric that would have caught the legal RAG regression, since a K of 10 is the standard cutoff used in the SEAL evaluation research. MRR, or Mean Reciprocal Rank, measures how high up the first relevant result lands, which matters directly for agent quality since a lot of pipelines only ever look at the top result. nDCG, Normalized Discounted Cumulative Gain, is a weighted ranking quality score that penalizes relevant results for being buried low in the list even if they technically appear somewhere.

The SEAL framework, which uses Recall@K and MRR metrics at K=10, offers a structured approach worth adopting even outside SEAL itself. It evaluates whether the system surfaced the correct sources, whether those sources were correctly used, and whether the final response fully addressed the query.

One rule that's easy to skip and shouldn't be: run retrieval metrics before the results ever reach the generator. That isolates a retrieval failure from a generation failure at the moment it happens, rather than trying to reconstruct which layer broke after the fact, which is precisely the diagnostic step missing in the legal RAG case. For agentic loops specifically, track retrieval quality per iteration rather than just on the first call. A retriever that's precise on iteration one but drifts by iteration three produces errors that cascade through everything downstream, and averaging across iterations hides exactly where that drift starts.

Measuring grounding and faithfulness without trusting a single score

Faithfulness is necessary but nowhere near sufficient on its own, and any team leaning on it alone is repeating the exact mistake from the legal RAG story. The generation side of an eval suite needs several metrics working together rather than one number carrying all the weight. Faithfulness itself checks whether every claim in a response traces back to something actually retrieved, flagging spans that aren't supported. Answer relevance checks something different: whether the response addresses the query itself, regardless of how faithful it stays to whatever context it happened to receive. Hallucination rate tracks the fraction of responses containing at least one unsupported claim. Attribution accuracy checks whether a citation, when the system provides one, actually maps to the passage it claims to be paraphrasing. Completeness checks whether every sub-question in a multi-part query got addressed, not just the easiest one.

Faithfulness and recall have to be tracked side by side, because a system can be faithful to incomplete context all day long without that ever showing up as a faithfulness problem. That's not a theoretical risk; it's exactly what a 0.91 faithfulness score sitting next to a collapsing 0.62 recall looks like in practice.

LLM-as-judge is a practical way to run faithfulness checks at scale, using a second model to evaluate the first. It works best with a precisely defined evaluator prompt, something like: does this claim appear verbatim or in paraphrase within the provided context? The risk is that the judge model hallucinates its own evaluations, so calibrating it against a human-labeled sample isn't optional if the scores are going to mean anything at all.

RAGAS is worth naming here as a concrete starting point: an open-source library with faithfulness, context precision, and context recall built specifically for RAG pipelines. As for where to set the pass/fail bar, enterprise tool-use benchmarks tracked by glean.com currently put even leading models around 70% accuracy, which argues against borrowing an industry number wholesale and calling it a threshold. Calibrate against a gold-standard dataset built for the specific domain instead. An imported number from someone else's benchmark tells you nothing about your own corpus.

Measuring freshness as a first-class signal

Freshness gets folded into relevance more often than it should, and that's a mistake worth correcting early. The two measure different failures entirely. A month-old article about a regulation that was just amended can score high on relevance and still be wrong. Stale data produces its own distinct failure mode: the agent reasons correctly from a premise that's simply no longer true, like a financial decision built on an interest rate figure that's since been superseded. Nothing about that chain of reasoning is broken. The input just belongs to a different moment in time, and no amount of relevance scoring surfaces that on its own.

Operationalizing this starts with annotating test queries with a correct-as-of date, the point past which an older result counts as a failure, full stop. For each retrieved result, the publication or last-modified date gets compared against that threshold. From there, freshness@K becomes a straightforward metric: the fraction of top K results that actually meet the date threshold. Purpose-built public benchmarks for time-sensitive queries are a sensible baseline before layering on domain-specific freshness tests of a team's own.

Freshness matters most for agentic finance and news workloads, where a single stale result early in a multi-step loop can invalidate everything the agent concludes downstream, no matter how sound the later reasoning steps look. One more thing worth doing in production, not just in the eval suite: log result publication dates as a standing practice. That turns freshness from something checked at test time into a signal a team can watch and alert on continuously.

Measuring latency and diagnosing where slowness lives

Latency belongs inside the eval suite itself, alongside the other metrics. Treating it as an afterthought is a mistake that compounds in the same way as the blended-score problem, and it's just as costly. Latency and accuracy trade off against each other in ways that only show up when they're measured together. A provider that returns faster but less complete results might score lower on recall, and a suite that tracks only one or the other will never surface that trade-off. Agentic loops raise the stakes further: a 500 millisecond overhead on a single API call sounds trivial until it compounds across dozens of iterations inside one agent run.

Three latency figures matter more than the average: p50, p95, and p99. P99 in particular is the number that catches tail behavior, and tail behavior is what agents actually run into under real load, a very different picture from the friendly median case that looks fine in a demo. Time-to-first-token versus full response latency is worth tracking separately too, especially for pipelines that stream results back to the agent rather than waiting for a complete response. Latency should also be segmented by query type rather than averaged across all of them; a complex multi-hop query has a different latency profile than a simple factual lookup, and blending them together hides both.

Isolate the API's own latency from whatever latency an orchestration layer adds on top. Measure the search API call by itself first, before measuring the end-to-end pipeline; that's the only way to know whether slowness lives with the provider or with the code sitting in front of it. Run these benchmarks under realistic concurrency rather than one request at a time, since production agents fire parallel tool calls constantly, and sequential testing won't catch what happens when several calls compete for resources at once. Set SLAs from these numbers before choosing a provider, not after signing a contract. A provider that looks perfectly fine at p50 can be disqualifying once p99 shows up.

Choosing evaluation frameworks and tooling for the suite

The framework chosen here determines what gets automated and what has to be built by hand, so it's worth treating as a real decision rather than a default pick made because a teammate mentioned a name in Slack.

DeepEval works like a pytest-style test suite built for LLM applications: test cases, reusable metrics, assertions, thresholds, a runner that plugs into CI/CD the same way any other test suite would. It covers RAG, agentic, and multi-turn evaluation inside one framework, which makes it a reasonable fit for teams that want eval folded into existing engineering workflows rather than living as a separate process off to the side. As of July 2026 it had close to 17,000 GitHub stars and more than 8 million PyPI downloads in the prior month, one data point worth weighing alongside how well it actually fits a given pipeline.

RAGAS, mentioned earlier for its faithfulness and context recall implementations, is the better fit when retrieval-side metrics are the primary concern rather than the full pipeline, and it composes reasonably well alongside DeepEval rather than competing with it. Braintrust supports both newer orchestration frameworks, like the Vercel AI SDK and the OpenAI Agents SDK, and established ones like LangChain and LlamaIndex, which makes it useful for teams with existing infrastructure who need evaluation added without rewiring the stack around it. Arize Phoenix and similar observability platforms round this out by handling production trace logging, closing the gap between what an offline eval catches and what actually shows up once the system is live and taking real traffic.

A sensible starting combination looks like this: a framework like DeepEval or RAGAS for retrieval and grounding metrics, an observability layer for freshness and latency signals once the system is in production, and LLM-as-judge used selectively for completeness and attribution checks at scale rather than everywhere at once. Resist the urge to over-invest in tooling before the dataset and metrics are stable. The framework serves the measurement design already worked out; deciding what to measure is the harder work.

Using public benchmarks to evaluate and compare search API providers

Internal metrics tell a team whether its own pipeline is regressing. They can't tell a team which provider to choose in the first place, because there's no shared dataset to compare against. That's what public benchmarks are for: standardized, reproducible comparison across providers on a fixed dataset, run the same way for everyone.

The Artificial Analysis Search Index, with data as of August 31, 2026, benchmarks 19 search API products across 9 providers using an equal-weighted composite of three benchmarks: DeepSearchQA F1, BrowseComp accuracy, and AA-Omniscience accuracy. It's a reasonable starting point for shortlisting providers before running any internal eval at all, and it matters more now than it might have a year earlier. The shutdown of the Microsoft Bing Search API in August 2025 pushed thousands of developers onto independent providers with little notice, and plenty of those teams found out the hard way that they had no systematic way to compare replacements.

That gap between adopting AI-powered search and actually measuring whether it works shows up in the broader industry numbers too. Reports from 2025 put the share of enterprise leaders who say AI-powered search improved marketing performance at 89%, yet a lot of those same organizations still lack a reliable way to measure that improvement directly. Instead, they rely on downstream business metrics that only loosely connect back to search quality, a weak substitute for actually testing the pipeline. That's the exact gap an eval suite like the one described here is built to close.

Public benchmarks are a starting point, not a substitute for domain testing. Treating a leaderboard win as the final word is a mistake worth naming plainly: a provider that leads DeepSearchQA won't necessarily lead on a legal corpus or a finance-specific query set. Developers building internal eval suites often start with public datasets covering factual and multi-hop query types, then layer domain-specific queries from production traffic on top, the same approach outlined earlier for building the test dataset itself. You.com publishes its own evaluation methodology and recommended datasets alongside its research API, which ranks first on DeepSearchQA for multi-hop reasoning; a published methodology like that gives teams something concrete to check their own eval design against, rather than building the whole thing from first principles alone.

None of this replaces judgment. A benchmark leaderboard and a well-built internal suite both narrow the search for the right answer, though neither hands it over outright. The real work sits in deciding what to measure, watching the metrics that don't move even when something's clearly broken, and treating a clean dashboard as a question worth asking again rather than a conclusion already settled.