How to Benchmark Search APIs for AI Applications

Search evaluation used to be built around a human sitting at a screen. Click-through rate, position on the results page, raw recall against a query; these metrics all assume someone is scanning a list, exercising judgment, and clicking the thing that looks right. None of that maps cleanly onto how an agent consumes a search result.

An agent doesn't browse. It ingests retrieved text as context and then acts on it, often without a human anywhere in the loop to catch a bad call. An irrelevant result isn't a minor annoyance the way it is for a person scrolling past it, then; it actively corrupts the reasoning that follows. Feed a model five snippets and one of them is off-topic or wrong, and there's a real chance that noise bleeds into the final answer.

Structured output quality is where this gets particularly sneaky, because classic recall metrics are blind to it entirely. A result can score as "relevant" under old-school evaluation and still be functionally useless to an LLM: JavaScript-gated content that never renders as text, a page truncated mid-paragraph, a snippet that captures the topic but not the actual fact needed. Recall says yes, but the agent gets garbage anyway.

Latency budgets are different too, and this one's easy to underestimate. A human tolerates a two-second wait for a page of results without noticing. A multi-step agent chaining five tool calls together cannot afford that same budget at every hop; delay compounds, and what felt instant at the single-query level turns into a multi-second stall by the time the chain resolves.

Then there's the benchmark contamination problem, which might be the least discussed and most consequential of the four. Many widely used datasets, SimpleQA and TriviaQA among them, are built from fixed question-answer pairs. A well-trained model can often answer these correctly straight from its training weights, no search required. Run that model through a search API and it "passes," but the API never actually contributed anything, so what's being measured, in that case, is certainly not retrieval.

Put those four failure modes together and a pattern emerges: accuracy, freshness, latency, and structured output quality aren't four separate boxes to check. They interact. A fast API that returns stale pages is fast at being wrong. A high-recall API that returns unstructured HTML is technically thorough and practically unusable. The rest of this piece treats them as one interconnected evaluation, not a checklist to tick through independently.

The public benchmark landscape and what each test actually measures

SimpleQA, built by OpenAI, runs thousands of questions across history, science, technology, and art, and it's designed to measure single-answer factual accuracy. It's a reasonable tool for measuring grounding lift: does adding a search API improve on what the model already knew? But it says nothing about freshness, and nothing about multi-hop reasoning, because its questions are single-step by design.

BrowseComp, released by OpenAI in April 2025, is a sharper instrument. It runs 1,266 hard questions that require actual multi-hop web navigation, and the results it surfaces are stark. Giving GPT-4o browsing tools moves its accuracy from 0.6% to just 1.9% (barely a nudge). Purpose-built agent architectures, by contrast, reach 51.5% on the same test. That gap is too wide to explain by tool access alone; it points to something structural in how retrieval and reasoning are architected together. BrowseComp, as a result, is probably the most predictive publicly available benchmark for complex agent tasks, because it's testing the system, not just the search call.

FreshQA takes a different angle: 600 time-sensitive questions that require current, real-world knowledge as of the moment they're asked. For any application where staleness itself is the failure mode (market data, regulatory changes, recent research findings), this is the closest public proxy available.

HLE, Humanity's Last Exam, gets cited a lot as a stand-in for agentic capability, but it primarily tests academic reasoning and domain knowledge. It's worth knowing about, though it isn't the same thing as testing retrieval.

Domain-specific tests are also starting to emerge, a Finance Benchmark of 120 queries covering SEC filings, earnings calls, and macroeconomic data being one example, which point toward where evaluation seems to be heading: narrower, more specific, closer to what a real production workload actually looks like.

Here's the shared limitation across every one of these: question sets are fixed at the moment of creation. A search API can win on the aggregate average of a benchmark and still fail badly on the specific domain, query pattern, or freshness window a real workload demands. Domain-specific leaderboards are useful data points precisely because they're narrower than general-purpose rankings, and domain-specific signal tends to carry more weight for teams building specialized products. Even that doesn't excuse a team from testing against its own queries, though, which is the subject of the next section.

Building an independent benchmarking methodology

Start with your own query corpus, not a benchmark's. A tool that wins on aggregate general questions can still lose on the specific slice of queries a production system actually runs, and production queries from your own domain are the only ground truth that settles the question for your use case.

Artificial Analysis, as of mid-2026, benchmarks 18 search API products across 8 providers, and that's a fine place to get a starting signal. It's worth treating as a first filter, not a final answer, because it's a leaderboard, and leaderboards average across question types that may have nothing to do with what a given team actually needs.

One credible independent method: run each API as a tool through its official SDK, evaluate the responses against a gold-standard answer set built in advance, and score each response as correct, partially correct, or incorrect. Use a panel of models rather than one, say Gemini 2.5 Pro, Claude Sonnet 4, and GPT-4.1 together, to cut down on any single model's judging bias. One evaluation using this approach ran 8 search APIs across 100 real-world AI and LLM queries, scoring 4,000 retrieved results total across relevance, quality, noise, and source type, and combined the findings into what it called an Agent Score: Mean Relevant multiplied by Quality.

That formula matters because it separates two distinct failure modes that get lumped together under "bad results" otherwise. An API can have low mean relevance, meaning a lot of queries come back with irrelevant hits, or it can have low quality on the hits that are relevant, meaning it finds the right page but returns it in a form that's hard to use. Both hurt an agent's performance, but they call for completely different fixes; one is an indexing or ranking problem, the other is an extraction problem.

Scale AI's tool-use benchmark for enterprise tasks found leading models topping out around 70% accuracy. That number is worth sitting with before evaluating any vendor, because it means "better than the field average" and "good enough for production" are not the same claim. Define an acceptable floor first, then measure candidates against it, not against each other.

What should get instrumented on every single API call? Four things, matching the four dimensions from the first section. Accuracy: does the retrieved content actually support the correct answer against a gold set built ahead of time? Freshness: what's the publication date on returned pages, and does the API even expose a way to filter by page age? Latency: measure the p99, not the mean, because tail latency is what breaks a multi-step chain. Structured output quality: is the API handing back full-page text or a thin snippet, can it handle JavaScript-rendered pages and PDFs, and are citations returned in a machine-readable form?

Test across more than one domain too. At minimum, general factual queries, time-sensitive queries, and whatever the primary domain is for the application being built. A provider that's strong on general knowledge questions can fall apart on specialized ones, and there's no way to know that without running the specific test.

Freshness and latency as first-class evaluation dimensions

Here's a failure mode worth sitting with: a search returns a cached page from six months ago, the model reads it, and the answer comes back sounding completely confident, because it's grounded in a real citation. The citation is just stale, and nothing about the output signals that to the person reading it.

What should get tested specifically? Whether the API exposes page-age filtering as a real, usable parameter. What the distribution of publication dates actually looks like across a representative query set. And whether "recency" ranking, when an API claims to offer it, actually surfaces content from the last few days rather than the last few months.

FreshQA's 600 time-sensitive questions are the closest public proxy for this kind of failure, but the real test is always the specific query type a workload runs. Financial data and regulatory filings have staleness windows measured in days. Product research or general background reading tolerates a much longer window. The benchmark can't know that distinction; only the team building the application can.

Latency deserves the same rigor, and the same caution against the easy metric. Mean latency flatters almost every API, because a handful of fast responses pull the average down and hide a longer tail of slow ones. Measuring p99 instead tells a different story, and it's the story that matters for agentic chains, where one slow hop early in the sequence delays every step that comes after it. As a reference point for what's achievable on infrastructure built specifically for this kind of workload, providers designed around AI retrieval from the start can reach meaningfully lower p99 figures than those adapted from consumer search products built for a different job.

Freshness controls and low-latency retrieval aren't premium features to negotiate up to. For anything running in production as part of an agentic system, they're the floor.

Structured output quality and citation faithfulness (the hidden failure mode)

Being cited and being correct are not the same thing, and this is probably the least understood failure mode in the entire retrieval stack. A grounded answer, one with a citation attached, can still point to the wrong page, quote information that's since changed, or overstate what the source actually says. The citation looks like proof, but it isn't always.

Research into RAG attribution has found that a substantial share of evaluated citations can show a pattern researchers call post-rationalization: the model reaches a conclusion first, then goes looking for a citation to justify it, rather than the citation actually driving the conclusion. That's a serious problem for anything claiming to be "grounded," because grounding implies the source came first.

What does structured output quality actually mean when evaluating an API? A few concrete things. Full-page content versus a snippet: a snippet forces the model to reason over a fragment, while full-page extraction lets it locate the relevant passage on its own, which tends to produce better answers. JavaScript-rendered page support: a meaningful chunk of the modern web doesn't respond to a simple HTTP GET request, and an API that can't render it will quietly return incomplete or broken content, often without any obvious signal that something went wrong. Document parsing: PDFs, SEC filings, research papers, this is where a lot of the highest-value information actually lives, and an API that can't extract it routes agents around the most authoritative sources available. Citation provenance, too: is the source URL, the publication date, and the author returned as structured, machine-readable fields, or does the agent have to guess at them from surrounding text?

Testing this directly isn't complicated. Build a set of queries where the correct answer sits inside a PDF, a second set where it's behind JavaScript rendering, and a third across a multi-page document. Then measure what percentage of the known gold-standard sources the API actually returns in usable form, since that percentage is often lower than teams expect.

Token efficiency rides along with this, and it's worth mentioning even though it's easy to overlook. A result that returns thousands of tokens of boilerplate wrapped around a single relevant sentence costs more to process and dilutes the signal the model has to work with inside its context window. Efficient extraction isn't just a cost line item; it's a quality issue too.

How major providers actually compare on these dimensions

A caveat before any numbers: these scores are useful for building a shortlist, not for making a final call. A provider that scores well on a general benchmark can still underperform on a specific domain, and the only way to know is to run the queries that matter to a given workload.

Parallel AI posted 58% accuracy on BrowseComp at $156 cost per million tokens (CPM), the strongest result-to-cost combination among the providers tested in that comparison. Perplexity came in at 22% ($256 CPM) on the same benchmark, and OpenAI's GPT-5 reached 53% ($253 CPM). Parallel raised a $100 million Series A in early 2025 on the argument that existing search infrastructure simply wasn't built with AI as its primary user, and results like this are the kind of evidence that thesis points to.

On the extraction side of the ledger, one provider, Firecrawl, has positioned itself around full-pipeline content handling, fetching over 8 billion pages over two years and building search, full-page retrieval, structured extraction, document parsing, and JavaScript rendering into a single path. For teams weighing structured output quality specifically, that's a relevant data point regardless of which provider they end up choosing.

Perplexity's Sonar, separately, reports around 22 million monthly active users and roughly 780 million queries in May 2025 alone, which speaks to real scale on the consumer side. Whether the enterprise SLAs and data retention terms match what a production deployment actually needs is worth checking, though, since a product built primarily for consumer use doesn't always carry the same guarantees a business deployment requires.

Some providers in that same independent 100-query, 4,000-result evaluation scored high quality on the results they returned as relevant but paired that with a lower mean relevance score, meaning a fair number of queries simply don't return the right hits at all. That profile makes such providers a stronger fit for teams that need structured access to specific search engines across different geographies, rather than teams optimizing purely for AI-native retrieval.

You.com's Research API holds the top spot on the DeepSearchQA benchmark, and its Finance Research API ranks first on FinSearchComp; combined with a reported 300ms p99 latency and zero data retention plus SOC 2 certification offered as a baseline rather than a paid add-on, it's a credible option for teams with enterprise compliance requirements sitting alongside their accuracy needs, and it serves enterprise clients including DuckDuckGo, Alibaba, and Amazon.

No single provider wins on every dimension measured here. That's not a hedge; it's the actual shape of the landscape. The evaluation matrix built in the sections above is what determines which tradeoffs are acceptable for a given workload, because the right answer depends entirely on what that workload actually needs.

Domain-specific evaluation and when general benchmarks mislead

A benchmark measures average performance against a fixed set of general questions someone else chose. A real workload is built from an organization's own domains, its own query patterns, its own freshness requirements, and its own definition of what counts as a correct answer. Those two things are related, but they are not the same thing, and conflating them is where a lot of evaluation mistakes start.

Finance is probably the clearest illustration available. The 120-query Finance Benchmark, covering SEC filings, earnings data, and macroeconomic indicators, produces a different provider ranking than general factual benchmarks do. A provider that performs well on SimpleQA can still have thin coverage of structured financial documents, because factual trivia and regulatory filing retrieval draw on completely different indexing and parsing capabilities.

Self-reported HLE scores make this same point from another angle. Parallel reports 47% on HLE, Perplexity 30%, with other providers in the low-to-mid twenties. These are interesting numbers, but HLE measures academic reasoning, not financial document retrieval and not regulatory freshness, so it's easy to over-read what these scores actually predict about a finance-specific workload.

Building a domain-specific gold set isn't complicated, but it takes real effort. Pull 50 to 100 representative queries from production logs or user research, not from a public dataset. Annotate the correct answer for each one along with the specific source page that supports it, not just any page that happens to contain the right words. Then test whether each candidate API actually returns that specific page, rather than crediting it for returning any page that happens to answer the question correctly. Source recall and answer accuracy are two different measurements, and they should be tracked separately: an API can land on the right answer through a secondary source while completely missing the authoritative one, and that distinction matters enormously in regulated domains.

Regulatory and compliance use cases add a third requirement that sits on top of accuracy and freshness: auditability. Can the citation chain be reconstructed and verified after the fact, months later, if a regulator or an internal audit asks for it? That's a different question from whether the answer was right at the moment it was generated, and it's one general benchmarks were never built to test.

Privacy, compliance, and data retention as non-negotiable evaluation criteria

Enterprise AI systems route sensitive query data through whichever search API sits underneath them: user questions, internal document lookups, financial research queries, all of it passing through a third party's infrastructure. That makes the provider's data retention and logging policy a legal exposure question, not a matter of preference or brand loyalty.

A few things worth verifying before any provider gets finalized. Zero data retention: are queries logged at all, and if so, for how long, and can that commitment be obtained in writing rather than taken on faith? SOC 2 Type II certification: is it current, and does it actually cover the specific API product being evaluated, rather than just the company's corporate infrastructure more broadly? Index independence: does the provider run its own index, or does it sit on top of a third party's (in which case that third party's data practices become part of the equation too, whether or not that's obvious from the outside). Geographic data residency, as well: where are queries actually processed, and where are results cached, since that answer can carry real regulatory weight depending on the industry and jurisdiction involved.

None of this is a box to check after the accuracy and latency work is done; it runs alongside it. A search API can score well on every dimension discussed above (accuracy, freshness, latency, structured output) and still be disqualified outright if its data handling doesn't meet what a specific regulated industry requires. Worth sitting with, that, because it's the one dimension where "good enough" isn't a sliding scale. It's a yes or no question, and it deserves to be asked early, not as an afterthought once a provider's already been selected on other merits.