Citation Quality as a Search API Evaluation Metric

Search APIs pass benchmarks and still hand developers citations that fall apart under a second look. The gap between an agent getting the answer right and its sources actually backing that answer up is what this piece works through, and the position it lands on is blunt: link validity and topical relevance are the wrong things to optimize for, because they're the two dimensions that already work fine.

Most search API evaluation treats citations as a downstream detail. If the model answers correctly, the thinking goes, the sources must be doing their job. That assumption is backwards, and the 2026 research paper "Cited but Not Verified" (Onweller et al.) shows why. Across 14 large language models, link validity stayed above 94% and topical relevance stayed above 80%. Factual accuracy, whether a cited source actually backs the specific claim attached to it, landed somewhere between 39% and 77%. Two numbers that ought to move together barely correlate at all.

That disconnect means an agent can serve up an answer backed by links that work and pages that are on-topic, and still misrepresent what those sources actually say. Trusting a citation by default, just because the link resolves and the topic matches, stops being reasonable once you look at that spread. Accuracy benchmarks alone can't tell a developer whether an agent's citations hold up. That takes a separate instrument built for the job, and the rest of this piece unpacks what that instrument measures, how it breaks down under real search conditions, and what it should change about how developers pick search infrastructure.

The three dimensions that actually define citation quality

The Onweller et al. (2026) framework out of PwC is the first pipeline to close the loop end to end: pull the citations a model actually made, retrieve the real content behind each source, then score the pairing on three separate dimensions instead of blending them into one number.

The first dimension, Link Works, asks whether the URL resolves and returns content a person could read. It's the easiest bar in the whole framework, and frontier models clear it reliably, above 94% in the PwC data. Clearing it proves almost nothing, though. A working link tells a developer the source exists. It says nothing about whether the source backs up the sentence sitting next to it.

The second dimension, Relevant Content, asks whether the cited page is actually about the same subject as the claim. Models do fairly well here too, above 80% across the tested systems, because retrieval systems are generally decent at matching subject matter. But passing this test doesn't mean much on its own. A page can be squarely on-topic, covering the right company, the right event, the right general subject, without the specific number or sentence the model relied on appearing anywhere in it.

The third dimension, Fact Check, is where the whole system comes apart. It asks the question that actually matters: does the source support the specific claim, not just the general topic? Across every model tested, factual accuracy fell between 39% and 77%, the weakest dimension by a wide margin and the one that most directly decides whether a citation means anything. The gap between dimension two and dimension three is the real story here. It's the reason a vendor quoting link validity and relevance numbers alone hasn't told a developer anything useful yet. Relevance and factual support are not the same property, and a search API can nail one while failing the other completely.

Earlier benchmarks named in the Onweller paper, AttributionBench, CiteME, CiteEval, mostly treat citation quality as a binary classification problem or a matching exercise. None of them combine citation extraction with a check across accessibility, relevance, and factual grounding in one pipeline. The PwC approach also uses an AST parser to pull citation-claim pairs out structurally, instead of relying on an LLM to do the parsing, which makes the evaluation reproducible rather than dependent on another model's judgment. Link, relevance, fact: hold onto these three, because the audit framework in the last section is built directly on top of them.

How search depth makes the factual consistency problem worse

More tool calls, more retrieved sources, more room to search. You'd expect that gives an agent's citations more to stand on. The Onweller et al. data says the opposite happens. As tool calls scaled from 2 up to 150 across two frontier models, Fact Check accuracy dropped by roughly 42% on average. Link validity and topical relevance barely moved.

That's a strange shape for a failure to take. Surface metrics hold steady while the dimension that actually matters collapses underneath them. Running more searches and pulling in more sources doesn't produce a better-grounded answer, it produces an answer that looks the same on the surface, working links, on-topic pages, while getting more confidently wrong underneath. This shows up hardest in deep-research configurations, exactly the setting where developers expect the most rigorous grounding, not the least.

Independent work backs this up. DeepTRACE (2025) found that deep-research configurations cut down on overconfidence and can rack up high citation thoroughness, meaning lots of citations attached to lots of statements. But a large fraction of those statements still go unsupported, with citation accuracy landing between 40% and 80% depending on the system. On average only about half, 51.5%, of the sentences generated by existing generative search engines were fully supported by the citations attached to them. Roughly half of what gets asserted has nothing solid underneath it, and that's before anyone asks whether the claim is even true.

DeepTRACE also flagged a related problem on debate-style queries, the kind where a topic genuinely has more than one side. One-sidedness showed up in 50% to 80% of responses across systems, and one major system produced one-sided answers in over 83% of debate queries, despite generating the longest responses of any system tested, averaging 18.8 statements per answer. Longer answers were not more balanced answers. Treating a long answer as a thorough one is exactly the mistake this data argues against, since volume of output tells a developer nothing about grounding.

So the practical takeaway is blunt: test depth settings and retrieval budgets against factual accuracy directly, never against how complete or how long the final answer looks on the page. Anyone tuning retrieval depth for "thoroughness" alone is optimizing the wrong variable, and the data above says so plainly.

What the DeepTRACE framework adds that citation-dimension scoring alone doesn't cover

DeepTRACE (2025) is a separate audit framework, built out of community-identified failure cases rather than a lab exercise, and it spans eight measurable dimensions across the answer text, the sources behind it, and the citations connecting the two.

The methodological choice that sets it apart is worth sitting with. DeepTRACE doesn't try to judge whether a statement is true in some absolute sense. It measures whether a system's own cited evidence actually grounds the statement it made. "Unsupported" and "factually incorrect" are treated as two different failures, not one, and that distinction matters more than it sounds like it should. A statement can be entirely true and still be unsupported by the citation sitting next to it. An unsupported statement that happens to be true is still a reliability failure, because the next one might not be true at all. Trusting a citation because the claim turned out correct means trusting the wrong signal entirely, and that's the mistake most manual QA processes make without realizing it.

To do this, DeepTRACE breaks answers down statement by statement, scores confidence at that level, and builds citation and factual-support matrices, audit work well beyond a simple pass or fail per citation.

A separate benchmark, Google's FACTS Grounding Leaderboard (2025), tests something adjacent but distinct: how well a model grounds long-form answers in document context it was actually given, up to 32,000 tokens, using a combined 1,719 public and private examples. That's a different axis from DeepTRACE, more about document fidelity than open web citation, but useful for anyone building document-heavy retrieval workflows.

Stacked together, these frameworks make the same point from three angles. Citation quality isn't a single test, it's a layered audit: citation-level accuracy from the Onweller et al. framework, system-level grounding and one-sidedness from DeepTRACE, document-context fidelity from FACTS Grounding. The broader implication is worth naming: citation quality and source grounding shape how useful a response actually is, independent of whether the answer was technically correct. Citation failures aren't just an accuracy problem sitting in an evaluation spreadsheet. They shape the user experience directly, whether anyone measures that or not.

Why citation failures are expensive in production, not just annoying in evaluation

None of this stays theoretical once an agent ships. Per Arthur AI, reported in Deepchecks' LLM Evaluation Benchmarks (2026), 34% of enterprises had a customer-facing incident tied to an LLM hallucination in the past year, with average remediation costs running past $50,000 per incident in regulated industries.

The failure rate isn't flat across task types either. Deepchecks (2026) put hallucination rates for extractive QA systems at 3% to 8% of responses, open-ended generation at 15% to 25%, and multi-step agent workflows at 20% to 40% of tool-call chains. Multi-step workflows are exactly where citation chains stretch longest and get hardest to check by hand, so the failure rate climbing there isn't a coincidence.

The damage gets sharper in high-stakes domains. Fabricated financial figures, invented legal precedent, technical documentation that doesn't exist: these aren't simple errors, they're the kind of mistake that costs real money or real legal exposure. And the underlying source problem is subtler than a broken link. The source the model cited might exist, might even be on-topic, and still not contain the specific number the model claimed it did.

You.com's Finance Research API documentation offers a concrete version of this: a query for a company's quarterly revenue can pull conflicting figures from different sources reporting different fiscal periods or definitions. A generic search API may return one of those numbers without flagging the disagreement, or return several without resolving which one actually answers the question asked. That's an evidence reconciliation failure, and it's a citation quality problem before it's anything else. The API needs to match the exact period and definition the query specified, throw out results that don't match, and point to the primary source. Leaving that reconciliation work for the model to botch downstream isn't a design choice worth defending. It's a gap that shows up later as an incident report, and by then it's a customer's problem, not an engineer's.

There's a runtime fix, and it isn't one single guardrail. Per a meta-analysis cited in Deepchecks' LLM Evaluation Benchmarks, layering guardrails together cut hallucination rates by 71% to 89% against unguarded baselines. Citation quality sits inside that grounding layer. It's the part current accuracy scores simply don't touch.

What current benchmark gaps mean for how developers should evaluate search APIs

Most benchmarking still leans on aggregate accuracy scores gathered under fairly artificial conditions: single-turn, API-based, no real agentic back-and-forth. Benchmark conditions matter more than they might appear to. A benchmark score collected without search enabled tells a developer very little about how that same system behaves once it's wired into a multi-turn agent loop.

AutoResearchBench (2026) shows how steep the drop-off gets once the task demands real research depth. Even the strongest LLMs tested managed only 9.39% exact-match accuracy on Deep Research tasks that require tracking down specific target papers across complex scientific literature. Plenty of otherwise strong baselines fell under 5%. General web-browsing agents can look sharp on open-ended tasks and then fall apart the moment the task demands a precise, multi-hop citation trail. Performance on easy benchmarks doesn't transfer to citation-heavy research work, and treating it as if it does is the single most common mistake in how teams pick search infrastructure.

The standard developer workflow, run a handful of test queries, eyeball whether the results look reasonable, ship it, breaks down on exactly the hard queries where citation failures cluster. A benchmark from unicodeveloper (Medium, 2026) tested time-sensitive queries drawn from FreshQA across four providers and found a wide spread between the best and worst performer, a gap casual spot-checking would never catch.

Part of why spot-checking fails is structural. Standard SERP-style APIs typically return only metadata, titles, URLs, and snippets of maybe 150 to 300 characters, which gives a model almost nothing to actually verify a claim against. Checking factual consistency requires the full page content, not a snippet. So the practical framework for developers is to test all three dimensions explicitly, link accessibility, topical relevance, factual consistency, in their own target domain, at the retrieval depth they actually plan to run in production, rather than leaning on a published benchmark run under conditions that may look nothing like theirs. Financial, legal, and medical queries are where a factual citation failure costs the most, so they deserve more testing attention than generic factual QA, not less.

What to look for in search API infrastructure when citation quality is a requirement

Part of why this evaluation question is live right now is structural. Microsoft retired its Bing Search APIs in August 2025, which pushed a wave of developers to re-evaluate their search infrastructure from scratch instead of treating it as a settled choice they could leave alone.

What an agent actually needs from a search layer isn't what a person typing into a search box needs. Grounded data, structured output, access to specialized datasources, a traceable citation trail, and some way to control retrieval budget: none of these come standard in a consumer-facing search API, and they're exactly the pieces that matter most once citation quality becomes the requirement rather than a nice-to-have.

The clearest dividing line is full content extraction versus snippet retrieval, and it isn't a matter of degree. An API either hands back the actual source content or it hands back a short snippet, and only the former gives a developer, or a judge model, enough to check a factual claim against. Specialized datasources matter for the same reason: regulatory filings, peer-reviewed papers, patents, clinical trial records, financial market data. These are exactly the domains where a citation failure costs the most, and an API that only indexes the crawlable public web can't reach most of this material at all.

Evidence reconciliation belongs on this list too, and it's the piece most vendors skip. An API that returns three conflicting financial figures without resolving them just hands the reconciliation problem to the model, which, is precisely where that problem tends to fail. Infrastructure that resolves the conflict at the data layer, before the model ever sees the result, reflects a different design decision about where the hard work happens. It is not a marginal improvement on the same idea.

You.com's Research API held the top spot on the DeepSearchQA benchmark at launch, and its Finance Research API ranks first on FinSearchComp's T2 historical-lookup sub-task, both public, reproducible results relevant to any developer who treats citation quality as a hard requirement. The Finance Research API in particular is built around the evidence reconciliation problem described above. DeepTRACE also included You.com in its comparison set alongside GPT-4.5/5, Perplexity, Copilot/Bing, and Gemini, making it one of the few independent evaluations to test citation support matrices across providers under comparable conditions.

Other infrastructure in the market approaches the same problem from different angles. Valyu indexes both the public web and specialized sources, regulatory filings, peer-reviewed research, patents, clinical trials, financial data, legal and life sciences material, and returns it parsed and structured rather than raw, a design that put it ahead in a five-domain benchmark run across more than 5,000 queries (unicodeveloper, Medium, 2026). Parallel, which raised a $100 million Series A led by Kleiner Perkins and Index Ventures, points to a 47% score on Humanity's Last Exam as evidence that every result comes with provenance attached. Firecrawl has crawled more than 8 billion pages over the past two years and passed a million developers on its platform, combining search and full content extraction into a single API call that returns clean markdown with the origin URL preserved on every record, which matters directly for audit trails.

Whatever provider a developer is weighing, the same handful of questions cut through the marketing fast. Does it return full content or just snippets? Does every result carry URL-level provenance? Does it resolve conflicting sources or just pass the conflict downstream? Does it publish benchmark results that include citation-support metrics rather than accuracy alone? In regulated industries, zero data retention and SOC2 certification belong on that list too, since the audit trail back to a raw source is a compliance question as much as a technical one.

A practical citation quality audit for your search API integration

Everything above points toward one conclusion: citation quality can't be verified by reading a vendor's accuracy number and calling it done. It has to be tested directly, against the same three dimensions the Onweller et al. framework isolated, run against the actual domains and retrieval depths a given agent will use in production. Skipping this and trusting the published benchmark number is the single most common and most avoidable mistake in the whole evaluation process.

That means checking, for a sample of real queries, whether links resolve, whether the cited pages are actually on-topic, and, critically, whether the specific claims attached to each citation trace back to text that actually appears in the source. That third check is the one manual spot-checking almost always skips, since it requires reading the full source content rather than just confirming a link works. It also means testing at more than one retrieval depth, since the Onweller data shows factual accuracy can fall as tool calls scale up even while link validity holds steady. And it means running domain-specific queries, financial, legal, medical, where the available benchmark data suggests failures cluster and where the cost of a bad citation runs highest.

To make that third dimension auditable rather than something a developer has to check by hand, the search API itself has to return full source content, not a URL and a snippet. That's the specific gap products like some search APIs are built to close: retrieving and structuring the actual page payload behind a citation so factual consistency can be checked programmatically, at scale, instead of one query at a time.

The pattern running through the PwC study, DeepTRACE, and FACTS Grounding stays consistent throughout. Link validity and topical relevance are the easy dimensions, the ones that already sit in the 80s and 90s across nearly every system tested, so testing them harder tells a developer almost nothing new. Factual consistency is the dimension that actually decides whether an answer can be trusted, and it's the one still sitting in the 40s, 50s, 60s depending on the system and the setting. Any audit of a search API's citation quality has to be built around that gap, not around the two dimensions that already look fine.