Open Benchmarks vs Proprietary Benchmarks for Search Infrastructure

Microsoft announced the retirement of every Bing Search API in May 2025, with a full cutoff by August 11. Ninety days to migrate infrastructure that a lot of production systems had quietly leaned on for years, and it forced a question teams had been able to dodge until then: when a search API vendor says its product beats a competitor's, how would anyone actually confirm that?

What the open/proprietary distinction actually means in search infrastructure benchmarking

Start with definitions, because the words get thrown around loosely. An open benchmark publishes its methodology, its dataset, its model harness, and its settings, so anyone outside the company can rerun the test and check the result. A proprietary benchmark is one where the vendor selling the product also designed the test, picked the dataset, ran the harness, and reported the score. Nobody else touches any part of it.

That difference isn't procedural trivia, and treating it as a minor footnote is where most buyers go wrong. A company that controls task selection can, consciously or not, choose tasks its product already handles well. A strong score on a benchmark like that tells you about performance on whatever got measured, chosen by the one party with every reason to choose well. That doesn't make the vendor a liar. It means the test was never built to be adversarial to its own interests, and that changes what the number is worth before anyone even reads it.

Open benchmarks fix this in one specific way: hold the model and the harness fixed, and change only the search API under test. That's the comparison a developer actually needs, because it isolates the single variable a purchasing decision hinges on.

Here's the distinction worth holding onto, because vendor marketing blurs it constantly: a company citing its rank on a third-party leaderboard is doing something different from a company citing a number from its own internal suite, even when both show up in the same paragraph of a pitch deck. The first is checkable. The second isn't, on its own, and no amount of confident phrasing changes that.

How the Artificial Analysis Search Index works and why its design choices matter

The clearest attempt to fix this in the search API market is the Artificial Analysis Search Index. As of September 2026 it covers 20 search API products across 10 providers, producing a composite score on a 0 to 100 scale from an equal-weighted mean of three benchmarks.

DeepSearchQA runs 900 research questions, each requiring multiple search queries to answer, scored on average F1. BrowseComp uses a hard 200-sample subset pulled from a larger pool of 1,266 questions. It's an open-source benchmark released by OpenAI in April 2025, built to test multi-hop web reasoning, the kind of task where an agent has to chain several searches together before it lands on an answer. The gap it exposes is stark: plain browsing tools alone score very low on BrowseComp. That gap says something concrete about how much persistence and multi-step reasoning matter in retrieval, well beyond single-shot lookup. AA-Omniscience uses 600 private held-out samples split evenly across six domains, 100 each, checking whether a model gives a correct answer when search context sits in front of it, rather than reciting what it memorized during training.

The methodology detail that carries the most weight: Artificial Analysis runs the same candidate model, on the same tasks, under the same settings, for every provider it tests. The search API is the only variable that moves. That's what makes the comparison mean something across providers, instead of just being internally consistent for one.

Sit with the baseline built into the index for a second. The same model, run with no search access at all, scores 33 on the composite. Turn search on, and scores across providers range from 65 to 75. So the jump from no search to some search is roughly the same size as the spread between the best and weakest search providers in the index. Provider choice sits at the center of the real decision, not off to the side. It's most of the decision, and anyone treating it as a secondary consideration has the math backwards.

Two extensions push the reproducibility question further. BrowseComp-Plus addresses a fair complaint about testing on the open web: the web changes underneath you, so a score from March doesn't necessarily mean the same thing in September. BrowseComp-ZH does something similar geographically, extending coverage to the Chinese web with questions designed around verifiable answers.

What the open leaderboard actually shows as of September 2026

As of that September 2026 snapshot, the top of the Artificial Analysis Search Index reads: Parallel at 75, then two other providers close behind at 74 and 73.

Worth flagging on its own, because it isn't obvious going in: better search quality tends to correlate with lower token cost downstream. A model that gets good results on the first pass doesn't burn tokens re-querying or reasoning around bad context. Parallel's own data shows its Advanced search mode cutting token use by more than 40% compared to its Basic tier, according to Artificial Analysis figures.

What the leaderboard leaves out matters as much as what it covers, and this is where a lot of procurement conversations quietly go wrong. It scores accuracy across three specific components. It says nothing about latency, nothing about how fresh the underlying data is, nothing about domain depth in financial filings or breaking news, and nothing about cost per query on its own terms. Treating this leaderboard as the whole picture means asking it questions it was never built to answer, and no amount of squinting at the composite score will produce a latency figure that isn't there.

Read these numbers as a starting point for evaluation, not the final word. The next section gets into why that's not just caution for caution's sake.

What proprietary benchmarks look like in practice, and what they obscure

Parallel's own self-benchmarking puts itself at 47% on HLE (Humanity's Last Exam), against 24% for one named competitor, 21% for another, and 30% for Perplexity, figures cited in Firecrawl's blog. Parallel's marketing cites its #1 ranking on the Artificial Analysis Search Index right alongside those self-reported HLE numbers, blending a third-party result with an internal one without drawing any line between them. That's the move worth watching for across this entire industry: stack a checkable number next to an uncheckable one, and let the reader's eye blur the difference. It works precisely because most readers won't stop to ask which number came from where.

Parallel also claims a 43% lower cost per task than its nearest competitor, an end-to-end figure covering model and search costs combined, reported by Parallel with no independent party checking the math. Separately, Parallel's own evaluation suite puts OpenAI's Web Search at 57.7% accuracy, a number that also hasn't been independently replicated.

Narrow the domain and the pattern holds. NewsCatcher ran a Q1 2026 benchmark across 32 event-detection queries and found its own CatchAll product hitting an F1 score of 0.705, more than double a named competitor's 0.317 on the same test. Cost per verified true positive came out to $0.185 for CatchAll, against $0.290 and $0.440 for two other providers in the same evaluation. That's a vendor-run test in a specific domain, news event detection, not a general-purpose leaderboard, and the distinction matters for how much weight the result should carry.

You.com's Finance Research API scores 87.29% on FinSearchComp, more than 14 points ahead of competitors at any price tier. That's domain-specific, and the methodology should be read as vendor-disclosed rather than independently reproduced. The caveat doesn't erase the result. It means the number reflects a real, measurable claim in a domain, financial research, where getting citations and source reconciliation right carries actual consequences.

Notice what every one of these figures has in common. Each vendor picked its own task, its own dataset, its own scoring method. Each number is internally consistent, nobody's cooking the arithmetic, but none of them sit on the same axis as anyone else's self-reported figure. Averaging NewsCatcher's F1 score with Parallel's HLE percentage produces nothing meaningful, because there's no shared denominator underneath either one.

None of this makes proprietary benchmarks worthless. A vendor benchmark can surface genuine depth in a narrow domain, financial filings, breaking news, that a general composite score was never built to capture. But the honest question, every time, is how much of a strong result reflects actual specialization versus a task the vendor chose because it already knew the outcome going in.

How to read a benchmark claim before trusting it in a production decision

Run a benchmark number through a short set of questions before it gets anywhere near a production decision.

Who ran the test? A party with money riding on the outcome, or a third party with nothing to gain either way? Was the model and harness held constant across every provider being compared, or did each vendor test on its own setup, under its own conditions? Is the dataset public and reproducible, or private, sitting behind a claim nobody outside the company can verify? Does the benchmark test the thing the use case actually cares about? A general-purpose composite score is not a substitute for a finance-specific or news-specific eval, and the reverse holds too. And when a vendor cites someone else's leaderboard, is it quoting the number as reported, or lifting the flattering part and leaving the rest on the floor?

One more check worth building into the habit: does the benchmark include a no-search baseline? The 33-point floor on the Artificial Analysis index does exactly this. It tells you how much search access helps before provider differences even enter the conversation, a different question entirely from which provider wins.

Open leaderboards and vendor benchmarks both earn a place in this process, but neither replaces testing on actual production data, and any team that stops at the leaderboard has stopped one step too early. Serious engineering teams are moving toward private, use-case-specific evaluation, built on their own traffic, measuring accuracy, latency, hallucination rate, and outcomes tied to the actual business problem. Startups including Braintrust, LangChain, Bigspin.ai, and Judgment Labs are building tooling for exactly this kind of internal eval infrastructure. None of this replaces open leaderboards. It's the layer that sits on top.

A reasonable cadence, given how fast this space moves, is to update internal benchmarks every quarter, or immediately after any major software update, infrastructure change, or reorg that touches the pipeline. Enterprise tool-use benchmarks currently put even leading models around 70% accuracy, a useful number to sit with before setting any threshold. Build a gold-standard dataset that looks like actual production traffic, set the accuracy bar against it, and raise that bar wherever an error would touch compliance or reach a customer directly.

The search APIs developers are actually choosing between in 2026

The field developers pick from in 2026 looks more crowded, and more differentiated, than it did before the Bing shutdown forced the issue. But crowded doesn't mean interchangeable, and a few of these products deserve more scrutiny than their marketing invites.

Parallel AI positions itself as AI-native and accuracy-first. Its Advanced mode holds the #1 spot on the Artificial Analysis Search Index at a score of 75 as of September 2026. Every result it returns comes with provenance and supporting evidence attached. Its product line runs from a standard Search API to a Task API built for deep research, up to a "Find All" tool meant for building datasets at scale. The open leaderboard rank is real and checkable. The HLE numbers sitting next to it in Parallel's own marketing are not, and treating them with equal weight is exactly the mistake this piece keeps flagging.

Firecrawl scored 73 on the same index. It keeps the origin URL attached to every extracted record, which gives compliance and risk teams something to actually audit, and it's built with LLM grounding workflows specifically in mind.

Tavily targets general agent web search, aimed at broad web coverage rather than structured or filing-grounded data. Some tools handle full SERP parsing across a wide catalog of search engines, suited to teams that need coverage breadth over depth in any single domain. Linkup focuses on fresh, low-latency results delivered as structured JSON, with GDPR compliance built in.

NewsCatcher's CatchAll product posted the strongest independently measured result specifically in event detection and news intelligence: an F1 of 0.705 against 0.317 for a named competitor in the vendor's own Q1 2026 test, described earlier.

You.com runs the broadest product line here, offering a Web Search API, an Answer API, a Contents API, a Research API, and a Finance Research API built specifically for financial workflows. The Research API holds a top benchmark position on DeepSearchQA, and the Finance Research API leads FinSearchComp at 87.29%, more than 14 points ahead of the next competitor at any price point. You.com lists clients including DuckDuckGo, Alibaba, and Amazon. The benchmark results are published openly, and the company's stated position is that a verifiable, cited result is the only honest basis for comparing infrastructure. That's a reasonable standard, and it's worth holding every vendor in this space to it, You.com included.

For regulated or brand-sensitive use cases, domain include and exclude controls matter more than they sound like they should. You.com and other providers let a team whitelist trusted sources outright, or block domains an agent should never cite, which turns into a real compliance tool rather than a nice-to-have.

Financial research applications usually need more than one type of data source: market data, fundamental filings, news and sentiment, alternative data. Most production systems in this space run at least two of those categories together, and source-reconciled, cited output is a core requirement built in from the start. It's the baseline requirement.

Why verifiability is the right selection criterion when accuracy and latency underpin production systems

The stakes aren't symmetric across use cases, and that asymmetry is the whole reason this distinction matters. A wrong result in consumer search costs someone a few extra seconds of scrolling. A wrong result inside a production agent handling financial research, a compliance query, or real-time monitoring propagates downstream before any human gets a chance to catch it.

Put a number on the risk and it turns concrete fast. A meaningful drop in recall could mean an agent misses a critical regulatory filing or a load-bearing research paper, and by the time anyone notices, the agent's downstream analysis, and any decision built on top of it, is already wrong. Recall carries direct, concrete stakes here. It's the difference between an agent that caught the filing and one that never knew to look for it.

Speed runs on a separate axis, but it isn't optional either. Responses returned quickly keep an agent in something like a working flow state; push past a certain threshold and task coherence starts falling off. Update frequency, whether a source refreshes in real time or once a day, is a different measurement from raw query latency, and how much it matters depends entirely on the use case sitting in front of it.

This is where the open benchmark earns its keep. By holding the model and the harness fixed and changing only the search API, an open leaderboard produces a result that transfers into a new team's context in a way a vendor's internal test suite generally can't, since that internal test was built around one vendor's own assumptions about what counts as a good task.

The architectural pattern underneath all of this is grounded AI: feeding a model live, cited web data instead of asking it to rely on what it memorized during training. The standard implementation is retrieval-augmented generation, RAG, where a web context API pulls current source documents, hands them to the model as context, and requires the model to cite the source URL in whatever it outputs. Verifiability at the infrastructure layer is what makes verifiability possible in the model's output. One doesn't happen without the other.

That points to the trust signal worth taking seriously when evaluating any vendor here, and it's the one most procurement checklists still underweight. A company that publishes its benchmark methodology and results on an independent leaderboard is making a claim that anyone with the time can check, challenge, and rerun. A company offering only self-reported figures is asking a team to take its word on faith, with no path to independent verification, and that gap should carry more weight in a vendor decision than a slide with a bigger percentage on it. You.com's Research API holds the top spot on DeepSearchQA, an open benchmark where the methodology and dataset are both public, meaning a developer can rerun the evaluation rather than take the ranking on trust. Parallel's self-reported HLE numbers cannot make that same claim, regardless of how confidently they're presented.

Use the open leaderboard as the entry point into evaluation, not the end of it. Treat domain-specific vendor benchmarks as supporting evidence for specialized cases, read with a clear eye on who ran the test and what they had riding on the outcome. And before committing to any provider at production scale, build or commission an internal eval on data that actually resembles the traffic the system will face. The open-versus-proprietary question was never about which number is bigger. It's about which number someone else can check, and that's the one that should decide the vote.