Handling Contradictory Web Sources in Agent Pipelines

Contradictory sources aren't an edge case in web retrieval. They're the default state, and any pipeline built as though clean, consistent data is the norm will fail in ways that are hard to catch and easy to trust anyway. This piece treats source conflict as a real architectural problem: sort it into types, catch it before it reaches the model, resolve it based on the kind of conflict it actually is, and keep the whole process auditable instead of a black box.

The common instinct is to treat contradictory sources as bugs, something you patch once a bad answer shows up in a demo or a customer complaint lands in the queue. That framing undersells how structural the problem is. The web guarantees conflict by design. Sources update at different speeds, and no crawl catches the internet at a single moment; a page indexed Tuesday may already be stale by Thursday, when an agent actually pulls it. Multiple authoritative voices exist on nearly any topic that's contested or still moving, from clinical guidelines to regulatory interpretations to earnings commentary. Structured data, the kind sitting in filings and databases, routinely disagrees with unstructured web text describing that same fact, too, because the two update on different schedules and pass through different hands before anyone publishes them.

Liu et al. (2025) found that roughly a quarter of open-domain queries retrieve contradictory evidence, and that held even for questions that weren't ambiguous or badly phrased to begin with. That figure holds even outside obviously contested topics, spread across ordinary queries rather than concentrated in a narrow band of disputed subjects. One in four queries running into disagreement across sources means conflict is baseline load, something you plan around from the start rather than patch after the fact. Most prior work on this problem focused on conflict between an LLM's parametric knowledge (the facts baked into its weights during training) and whatever gets retrieved at inference time. That's a real problem, and arguably the easier one. The harder case, and the less studied one, is conflict inside the retrieved corpus itself: three sources come back for one query, each asserting something different, none of them touching the model's prior knowledge at all.

The cost of getting this wrong shows up less as a crash and more as an answer that sounds too sure of itself. Research has shown that state-of-the-art LLMs often just ignore conflicting evidence and produce one fluent answer, with no sign it was pulled from a contested set of sources. It stays invisible until someone downstream acts on the wrong number.

A working taxonomy of the conflict types agents actually encounter

"Handle conflicts" isn't an engineering instruction until you can say what kind of conflict you mean. Skip the taxonomy and developers build one blunt mechanism, usually majority vote or recency preference, and slap it onto problems that need completely different treatment.

The DRAG taxonomy (arXiv:2506.08500, 2025) lays out five categories worth knowing cold. No conflict is the baseline: sources agree, nothing to resolve. Complementary information means sources cover different angles of the same topic, so the right move is synthesis, not arbitration. Conflicting opinions represent legitimate divergence: different experts or regions taking different stances on something genuinely unsettled, so the output should present multiple views instead of crowning a winner. Outdated information means one or more sources reflect a prior state of the world that's since changed, which makes recency the deciding signal. Misinformation means at least one source is just wrong, and that calls for credibility weighting or outright rejection.

Each of the five needs its own resolution behavior. Run one blanket strategy across all five and you'll get at least four of them wrong.

Developers most often collapse outdated information and misinformation into one bucket, and the distinction matters more than it looks at first glance. Outdated information was correct when it was written; the world just moved. Misinformation was never correct. Mix the two up and the resolution logic runs backwards: a pipeline that treats a stale-but-once-accurate figure as garbage, or one that treats a source that was wrong from day one as merely overdue for a refresh.

There's a sixth type too, one that doesn't show up in DRAG but shows up constantly in financial and technical pipelines: structured-versus-unstructured conflict. A balance sheet figure pulled from a filing API disagrees with a number quoted in a news article covering the same company's earnings. This kind of mismatch is usually systematic, not random; the article might be rounding, citing a different reporting period, or working off a secondary source that made an error upstream and never got caught. I kept turning this one over because it doesn't map cleanly onto any of the original five — it's not that either source is wrong exactly, it's that they're answering slightly different questions while appearing to answer the same one. Naming the type is the first gate. A pipeline that can classify what it's looking at routes to the right fix. One that can't is guessing, and guessing at scale is how confident wrong answers get produced by the thousand.

How LLMs behave badly when conflicts are left unmanaged

Left alone, an LLM doesn't handle conflicting evidence neutrally. It behaves in ways that are systematically biased, and the bias is predictable enough to name.

Research documented in "Whose Facts Win?" (arXiv:2601.03746, 2026) found that LLMs show consistent source preferences under conflict, favoring certain source types regardless of which one is actually correct. Two patterns stand out. The first is authority bias: models tend to overweight facts asserted directly by the user in a prompt, even when retrieved evidence flatly contradicts the user's claim. The second is loaded-question accommodation, where a model swallows a false premise buried in a query rather than correcting it. Ask a model something built on a wrong premise, and more often than not it just answers the question as posed instead of flagging that the premise is broken.

Overconfidence is the dominant failure mode across both patterns. The model produces something fluent and decisive, picks a side, gives no signal a conflict was ever present in what it retrieved. That missing signal is the real danger, more than the wrong answer itself. A wrong answer flagged as uncertain gets caught by someone; one delivered with total confidence usually doesn't get caught until something downstream breaks.

Source attribution alone doesn't fix this. Research out of the ArbGraph project (arXiv:2604.18362) found that attribution signals, essentially labeling where a claim came from, fail to capture how multiple pieces of evidence interact or contradict each other. Consistency needs those relationships modeled explicitly, before generation happens, rather than a citation stapled on afterward with the hope that a reader notices the gap.

This is where the line between arbitration and reconciliation starts to matter. Arbitration means picking one source and dropping the rest: the right move for misinformation, the wrong move for legitimate opinion divergence. Reconciliation means folding apparent contradictions into one coherent answer, which works for complementary information but fails badly when one source is simply false. You can't synthesize your way out of an error. The failure isn't picking the wrong strategy once. It's picking one and running it everywhere, on every conflict, regardless of type.

The problem compounds in long-horizon agents, the kind chaining multiple retrieval and reasoning steps together instead of answering in one shot. Long-horizon agent success rates tend to sit well below single-turn rates, and it isn't hard to see why: each retrieval hop that carries an unresolved conflict multiplies the error downstream instead of averaging it out. The model can't fix a problem it was never shown. Detection has to happen at the pipeline layer, before the prompt even gets assembled, because by the time a conflict reaches generation, the model has already committed to one version of reality as ground truth.

Detecting conflicts before they reach the LLM

The cheapest place to catch a conflict is before it becomes part of the prompt. Once contradictory claims are baked into context, the model has to reason its way out, and, as covered above, it usually doesn't manage it.

A handful of signal types make detection possible. Factual contradiction on the same named entity, date, or number across two or more sources is the obvious one. Timestamp divergence is another: sources describing the same "current" fact carry meaningfully different publication dates, and a claim from 2021 sitting next to a claim from 2024, both about the same supposedly present-tense fact, should trip a flag automatically. Source-type mismatch (structured data from a filing or database disagreeing with unstructured web text on the same figure) is its own signal worth tracking separately. Claim polarity, an affirmative statement next to a flat negation of the same proposition, is often the clearest signal of all, though also the easiest to miss if claims never get broken down to the atomic level first.

Experiments tied to the DRAG benchmark (arXiv:2506.08500, 2025) found that prompting a model to reason explicitly about conflicts in its retrieved documents meaningfully improves answer quality. But that only works if the conflict gets surfaced first. A buried conflict stays buried; the model can't reason about a disagreement it never sees framed as one.

In practice, detection runs through a few concrete steps. Cross-source claim extraction pulls atomic claims out of each retrieved document on its own, then checks them against each other on shared entities before anything merges into the final context window. Recency tagging attaches a retrieval timestamp and a publication date to every chunk of text, flagging cases where the freshness gap between sources crosses some threshold you set. Source-type labeling tags each chunk as structured or unstructured and surfaces mismatches explicitly, instead of letting them sit quietly side by side in the same context block. Semantic conflict scoring adds a subtler layer on top: an embedding-distance check where two claims about the same entity look topically similar but diverge sharply in what they actually assert.

What detection produces, when it's working, is a conflict manifest: a structured record of which sources disagree, on what claims specifically, and what category of conflict is at play. Resolution runs off that record.

Detection doesn't need to be perfect to earn its keep. A false positive, flagging something as a conflict when it isn't one, costs an extra reasoning step and maybe a slightly slower response. A false negative, missing a real conflict, costs a wrong answer delivered with total confidence. Those two error types don't cost the same, and any team tuning detection thresholds should weigh them accordingly rather than treating both as equally bad.

Resolution strategies matched to conflict type

No single strategy covers all five DRAG categories. Routing by conflict type isn't a nice extra; it's the requirement that makes the rest of the pipeline function at all.

Credibility-aware generation is the base layer for misinformation. This means scoring sources on metadata: domain authority, the type of publication, whether the claim gets corroborated across independent sources, how fresh the information is. Generation then gets steered toward the higher-credibility evidence instead of treating every source as equally trustworthy by default, which is the assumption that gets teams in trouble. The same logic carries over to structured-versus-unstructured conflicts; a primary-source filing should outrank a secondary news article's read on the same number, because the filing sits closer to the ground truth by construction.

For factual-level conflicts specifically, graph-based resolution beats heuristic picking. TruthfulRAG (Liu, Shang and Zhang) uses knowledge graphs to resolve factual conflicts in retrieval-augmented generation, representing claims as nodes and their relationships as edges. That structure lets arbitration follow how claims actually relate to each other, rather than falling back on a "trust the source with the higher domain score" shortcut that ignores the relationships between competing claims entirely.

Outdated information calls for recency routing. Retrieval infrastructure that surfaces publication date and crawl freshness lets the pipeline favor the most recent authoritative source when a claim is time-sensitive. There's a hard prerequisite here, and it's not optional: live web search. A static knowledge base can't resolve a temporal conflict if the fresher source doesn't exist inside it. No amount of clever resolution logic downstream fixes a corpus that stopped updating six months ago.

Opinion and complementary conflicts need something else entirely: synthesis routing. When divergence is legitimate (different experts holding genuinely different views, different regions under different regulatory rules), the right answer lays out the landscape, attributes each view to its source, and lets the reader see the actual shape of the disagreement instead of getting handed a winner. Pipelines should flag these cases for multi-perspective generation rather than forcing arbitration onto a question that doesn't have one correct answer.

Worth flagging separately: Change Data Capture, or CDC, works as a consistency tool for pipelines pulling from multiple live data systems. When those systems update asynchronously, which is the norm rather than the exception, agents can end up holding contradictory context purely from timing lag, not any real factual disagreement between sources. This one is easy to overlook precisely because it isn't a "conflict" in the DRAG sense at all — it's a plumbing problem wearing a conflict's clothes, and treating it with credibility scoring or synthesis routing would be solving the wrong layer entirely. CDC streams propagate updates across systems close to real time, which shrinks the whole class of conflicts caused by nothing more than one system running a few minutes or hours behind another.

Put together, the routing logic looks something like this. Misinformation gets credibility arbitration with the low-credibility source suppressed. Outdated information gets recency arbitration favoring the freshest authoritative source. Conflicting opinions get synthesized with attribution kept intact. Complementary information gets merged with no conflict flag needed downstream. Structured-versus-unstructured mismatches get resolved toward the structured primary source, with a note left behind documenting the discrepancy for anyone who checks later.

Making conflict resolution auditable and citation-grounded

Resolving a conflict silently leaves a gap. The agent needs to record which sources it weighed, which ones it discarded, and why, so a resolution can be traced back rather than taken on faith.

Auditability earns its keep well past compliance checkboxes. For debugging, a conflict manifest tells you whether a wrong answer came from a missed conflict, a correctly detected conflict resolved the wrong way, or something else entirely; without that record you're stuck guessing at where the pipeline actually broke. For trust, downstream systems and human reviewers can only verify what they can trace back to an actual source. For iteration, conflict logs become a dataset in their own right, one that recalibrates detection thresholds and resolution routing as the pipeline matures and the failure modes get clearer.

Citation grounding is what makes resolution checkable after the fact. Every generated claim should carry its source as a structural part of the output, not an optional footnote bolted on afterward. CiteLLM (Hong et al., 2026) shows what this looks like done well: context-aware query generation paired with full-text semantic retrieval, reaching 87.5% precision on citation accuracy. BibAgent (Li et al., 2026) takes a related approach, using retrieval-augmented evidence committees to check claims against source documents before anything gets output. The same principle holds whether it's happening inside one model or across an entire pipeline: verify before you generate, not after the fact.

A useful conflict resolution record holds four things: the classification of the conflict type; the sources involved, along with metadata such as URL, publication date, source type, and credibility score; the resolution strategy applied and the reasoning behind it; and the final claim or claims that made it into generation, each tied back to its source.

In financial pipelines this stops being optional. An answer that cites a balance sheet figure without tracing it back to a specific filing and date is unverifiable, full stop. Nobody downstream can check it against the primary record. Routing structured filing data through a domain-grounded finance API, rather than pulling it from general web search, is the architectural choice that makes that kind of traceability possible in the first place. That same audit record doubles as an input for measurement: faithfulness scores, citation grounding rates, and conflict detection recall can all get computed straight from it, closing the loop between detecting a conflict, resolving it, and checking later whether the resolution actually held up.

Measuring whether the conflict resolution layer is actually working

A conflict resolution layer nobody measures is an assumption dressed up as infrastructure.

Building a real evaluation set means starting with questions that have known ground truth but whose retrieval corpus contains genuine contradictory evidence, rather than the clean scenarios most eval sets default to because they're simply easier to build. Coverage should span all five DRAG conflict types. A system that scores well on misinformation detection but has never been tested against complementary-information synthesis has only been evaluated on part of the problem. The CONFLICTS benchmark, introduced alongside the DRAG paper (arXiv:2506.08500, 2025), is the first expert-annotated benchmark built specifically for this, and it's a reasonable place for teams to start when building their own eval sets.

A handful of metrics matter more than the rest. Conflict detection recall asks what fraction of real conflicts in retrieval actually got flagged. Resolution accuracy asks, among conflicts that got flagged, whether the pipeline applied the right strategy for that type. Faithfulness score checks whether the claims in the final output are actually backed by the context the pipeline let into the prompt. Citation grounding rate tracks what share of output claims carry a traceable source. Abstention rate on genuine misinformation checks whether the system holds back an answer or adds a caveat instead of confidently repeating something false.

Existing benchmarks like FACTS Grounding, from Google DeepMind, and LiveNewsBench (2026) are useful but incomplete here. Both evaluate grounding accuracy and factuality at the level of the final output. Neither isolates conflict detection and resolution as its own piece of the pipeline; they measure the answer that comes out the other end, not whether a conflict got noticed before generation even started. A 2026 arXiv paper makes the case for search-aware evaluation methodology, one that tracks how behavior shifts across different retrieval conditions rather than just scoring the final text. That direction seems right, even if it isn't standard practice yet.

Enterprise RAG deployments report hallucination rate reductions somewhere in the 70 to 90% range compared to standard LLM output. That's a real number, but it's aggregate, and aggregate numbers hide the detail that actually matters: which conflict types the pipeline handles well and which ones it's quietly failing on underneath the average. I'd treat that range with some suspicion until it's broken out by conflict category — a pipeline could be excellent at misinformation suppression and mediocre at everything else and still land inside that band. A team that instruments its evaluation by conflict category, instead of settling for one blended hallucination-reduction figure, can actually find where the system breaks instead of just knowing that it sometimes does. Production conflict logs feed back into the eval set, which recalibrates detection thresholds and resolution routing, which then gets re-measured. A pipeline that never closes that loop drifts slowly out of step with the sources it's supposed to be reconciling.

Infrastructure choices that determine how well the architecture can execute

Every resolution strategy above rests on one thing: retrieval infrastructure that delivers the right signals (freshness, source type, provenance metadata) at the moment of retrieval, rather than bolted on afterward as a post-processing step. Get this wrong and the most carefully designed resolution logic in the world has nothing solid to work with.

The clearest example is the gap between live web search and a static knowledge base. Temporal conflicts, the outdated-information category from the DRAG taxonomy, simply can't get resolved if the fresher source never enters the pipeline in the first place. Real-time web retrieval is what makes recency routing possible at all. A static corpus, however large, has a ceiling baked in from the day it was indexed, and no clever arbitration logic downstream can manufacture a source that was never captured.

The same logic applies to source-type metadata. A pipeline that strips out structured filing data or database records somewhere upstream, flattening everything to plain text by the time it reaches the resolution layer, has thrown away exactly the signal that structured-versus-unstructured detection depends on. Provenance metadata (publication timestamps, domain information, source type) needs to travel with the content through every stage of the pipeline, not get stripped for convenience somewhere in the middle because it's inconvenient to carry along.

This calls for discipline more than exotic engineering: deciding early that conflict is the normal condition of working with the open web, and building retrieval, detection, and resolution as one continuous system rather than three afterthoughts bolted together after something breaks in production. The pipelines that hold up under real use are the ones where somebody asked these questions before the first user query ever came in, not after.