Tool Selection and Routing in Multi-Tool Agents

Before any routing logic executes, the model has already been shaped by the language it was given. That framing should be uncomfortable, because it means the primary failure point in many systems isn't the routing algorithm or the retrieval strategy. It's a documentation problem that nobody on the team is responsible for fixing.

The most common failure: tool descriptions are written for human developers, not for LLM pattern-matching. "Handles user account management operations" is meaningful to an engineer who has read the codebase. It is nearly useless for a model deciding whether to invoke it given a query about resetting a password on behalf of a service account with elevated permissions. The description simply doesn't answer that question. No routing logic, however sophisticated, can recover from that upstream ambiguity.

The second failure is what practitioners call skill collision. Two tools whose descriptions overlap enough that the model cannot reliably distinguish them produce both false positives (invoking a tool that shouldn't apply) and false negatives (skipping one that should). These surface differently in production, which makes attribution genuinely difficult. I've spent more than one debugging session convinced the model had made a reasoning error before realizing the actual problem was that two tool descriptions were functionally synonymous at the embedding level.

Under-specified argument schemas create a third, subtler problem. Even when the model selects the right tool, a vague schema causes it to pass malformed inputs. The selection was correct; the invocation wasn't. Without deliberate logging at that boundary, the two failure modes look identical in the output.

The frontier response is to treat tool documentation as a maintained artifact and use LLMs and execution traces to rewrite it systematically: collect false positive and false negative cases from production, diagnose which description ambiguities caused them, revise accordingly. Adobe's REGAL architecture (2026) takes this seriously at scale, incorporating a registry-driven compilation layer that enforces alignment between tool specification and execution, explicitly treating "tool drift" as an engineering concern rather than a documentation afterthought.

At scale, beyond roughly fifty tools, embedding similarity largely takes over from prompt-level reasoning as the selection mechanism. This makes description quality more consequential, not less. A vague description produces a vague embedding, which retrieves poorly. The upstream problem becomes the downstream problem, compounded.

The retrieval layer: how agents find the right tool when the registry is too large to fit in context

Beyond roughly fifty tools, it is no longer practical or token-efficient to enumerate all tool descriptions in the prompt. A retrieval step must occur before selection even begins. The model may not see most of the registry. It sees only what retrieval surfaces. That constraint shapes everything that follows.

Early retrieval approaches relied on lexical methods: TF-IDF, BM25, sparse matching. Fast and interpretable, but brittle when user query phrasing diverges from tool description phrasing. A user asking "what did my pipeline spend last quarter" and a tool described as "retrieves historical compute cost aggregations by billing cycle" may be a perfect semantic match; lexical retrieval may not surface it at all.

Dense retrieval, embedding tool descriptions and ranking them against an embedded query, handles semantic mismatch better and scales to large registries. It is now the mainstream approach. But it retrieves individual tools in isolation, without regard for whether those tools belong to a bundle designed to handle multi-step tasks coherently together. Retrieving the right tools independently does not guarantee the agent has the right combination for the task at hand. That gap is harder to close than it sounds.

Query rewriting addresses the intent-to-spec mismatch: when a user's natural language is unlikely to match tool description language, rewrite the query before retrieval. Iterative and progressive selection goes further, decomposing complex tasks into sub-queries and running retrieval for each. This improves recall on multi-step tasks where a single query would retrieve an incomplete tool set, though it adds latency that matters in user-facing contexts.

The underlying tension (retrieval optimized at the individual tool level losing the coherence of agent bundles; retrieval optimized at the agent level obscuring individual tool capability) is not a retrieval problem. It is an architectural problem that retrieval alone cannot solve. Understanding that distinction is what clarifies why the routing architectures covered in the next section are structured the way they are.

Routing architectures that go beyond single-step, single-model decisions

Most non-trivial agent tasks require a sequence of tool calls. The model best suited to invoking a particular tool is not necessarily the model best suited to synthesizing its result or planning the next step. Single-round, one-to-one routing assumes a flatness that real workflows don't have, and that assumption is where a lot of performance gets left on the table.

MasRouter (arxiv:2502.11133, ACL 2025) illustrates what layered routing looks like in practice. It uses a three-layer cascaded controller: first, determining collaboration mode, whether a task warrants a solo agent or multiple agents; second, allocating roles across agents; third, routing each role to the LLM best suited for it. On MBPP, it improved over prior state-of-the-art by 1.8 to 8.2 percent. On HumanEval, it reduced overhead by 52 percent. The numbers matter less than what they demonstrate structurally: routing is not one decision but a stack of decisions, each narrowing scope for the next.

Router-R1 formulates multi-LLM routing as a sequential decision process using reinforcement learning, interleaving reasoning with dynamic model invocation. It advances toward multi-step routing, though it still optimizes at the query level rather than over full pipeline trajectories. RouteLLM learns routing from human preference data and generalizes across strong-weak model pairs. MixLLM frames routing as a contextual bandit problem, enabling adaptation as query distributions shift. Both move routing from hand-coded heuristics to data-driven policies; the catch is that generating meaningful training signal requires production traffic volume that many teams don't yet have. For those teams, the heuristics remain the practical starting point, not a failure of ambition.

The pattern taking shape around Model Context Protocol, which gained significant traction through 2025 and into 2026, introduces a further routing dimension: which model handles which tool categories best? Tool-heavy subtasks are being routed to models with strong function-calling benchmarks; reasoning-heavy subtasks to chain-of-thought-strong models; final synthesis to response-quality-tuned models. MCP Gateways are emerging as unified control planes that route both tool requests and model selection simultaneously, collapsing two separate routing problems into one proxy layer. Whether that simplification holds under real production complexity is still being tested. The architecture is elegant enough that I'm cautiously optimistic, but elegant architectures have a way of revealing their rough edges in production that they don't reveal in design.

Handling ambiguity: what routing systems do when the right tool isn't obvious

Routing ambiguity in production takes four forms that are worth distinguishing precisely because they call for different responses. Query-spec mismatch: user language doesn't map cleanly onto any tool description. Skill collision: two or more tools plausibly match, but only one is correct for this specific context. Underspecified intent: the query is too vague to uniquely determine a tool without additional information. Multi-tool queries: the request spans multiple domains and requires parallel or sequential invocation of several tools.

Rule-based fallbacks (keyword matching, pattern recognition) still function as first-pass filters in many production systems before more expensive LLM-based routing. They are fast and deterministic, which matters for latency budgets. They fail on inputs they weren't designed to handle, which in a sufficiently diverse production environment is a regular occurrence, not an edge case. Their value is as a coarse filter.

The ReAct loop (Think, Act, Observe) is the most widely deployed mechanism for handling ambiguity without requiring upfront resolution. Rather than committing to a complete plan before any tool is invoked, the agent acts, observes the result, and uses that partial information to inform the next routing decision. A wrong initial selection, or an underspecified query, can be corrected mid-sequence because each tool result is fed back before the next step. Sequential disambiguation has real practical value. It does not eliminate bad routing decisions; it reduces their cost. That's a distinction worth holding clearly.

Hierarchical and self-reflective agent patterns, where agents evaluate whether to invoke a tool before committing, add another layer of deliberation. They typically operate at a single granularity per step, though, either agent-first or tool-only. The hybrid problem (selecting the right agent bundle and the right individual tool within it in a single routing decision) remains an active research frontier. Most production systems I've encountered resolve this by choosing one granularity and accepting the costs that come with it. That's not an architectural statement; it's a practical concession.

Why most enterprise deployments aren't running true multi-tool agents yet — and what that reveals

The numbers that have circulated suggest only around 16 percent of enterprise deployments and 27 percent of startup deployments qualify as true agents: systems where an LLM plans, executes, observes feedback, and adapts. The majority are fixed-sequence or routing-based workflows wrapped around a single model call. Prompt design remains the dominant technique, followed by retrieval-augmented generation. Multi-tool orchestration, despite the volume of research attention it attracts, is not the typical production configuration.

This should calibrate how practitioners read the routing literature. MasRouter, Router-R1, MCP gateways represent where the field is heading. They do not describe where most production systems currently sit. That gap is real, and it mostly isn't a gap in ambition.

Beneath the routing gap is an infrastructure gap. The figure that should give practitioners pause is that the overwhelming majority of enterprise generative AI pilots in 2025 delivered no measurable P&L impact. Not because the models were insufficient, but because the data infrastructure feeding them was failing to keep up. An agent with sophisticated routing logic but poor-quality tool outputs will still fail. The routing layer is only as good as what the tools return, and this seems to be the lesson that needs to be learned through painful experience rather than anticipation.

The organizational pattern is consistent. Teams invest in the LLM and the orchestration framework before hardening the data pipelines and tool contracts that routing depends on. The routing architecture looks sound; the tool reliability doesn't match it. Morgan Stanley's DevGen.AI, announced in January 2025, reviewed over nine million lines of code and saved approximately 280,000 developer hours. What made it work wasn't only the model or the routing logic. The underlying data pipeline (legacy code as structured, queryable input) was a prerequisite the team built before the agent could function at all. The tool contract came first. That ordering matters, and it's the part most teams get backwards.

How grounding and real-time data access change the routing problem

Routing is not just about picking a tool. It is about picking a tool that will return accurate, current data. A correctly selected tool returning stale or low-quality data degrades agent reasoning as effectively as a wrong selection. This means routing decisions must encode not just capability but data quality and freshness expectations. Most routing architectures today don't do this explicitly, which is a gap that tends to stay invisible until it produces a consequential error.

The hallucination problem in agents is largely a grounding problem. Agents reasoning in isolation from real-world data will confabulate. The ReAct Think-Act-Observe loop is specifically designed to prevent this by anchoring each reasoning step in a tool result rather than in the model's parametric knowledge. Routing that surfaces the wrong tool, or a tool with a stale data source, undermines that grounding at precisely the moment it matters most.

Enterprise grounding requirements are more complex than web search. Agents connecting to CRMs, ERPs, policy repositories, and operational systems are dealing with tools that carry their own reliability, latency, and freshness profiles. A routing decision that ignores these profiles is implicitly assuming all tools are equivalent along those dimensions. They aren't. A compliance tool pointing to a policy repository updated quarterly behaves very differently (from a routing perspective) than a market data tool updated in real time. The routing system that treats them identically is making a category error.

Real-time web data introduces freshness as an explicit routing criterion. A query about current events should route to real-time search, not a static knowledge base; the routing system must encode this distinction. Speed and accuracy interact: a search tool that returns faster but with lower precision imposes a cost on the agent's downstream reasoning. The routing decision implicitly trades these off, often without any instrumentation to make the tradeoff visible, which makes it difficult to improve deliberately.

For high-stakes domains (finance, legal, compliance) the cost of a routing error is domain-dependent and potentially severe. Missing a critical filing because a poorly selected search tool had lower recall is not an abstract quality metric; it is a material error. Routing systems operating in these domains need to encode that asymmetry explicitly, and currently most don't.

REGAL (Adobe, 2026) addresses the data-tool coupling directly. Its Medallion ELT pipeline produces replayable, semantically compressed Gold artifacts; its registry-driven compilation synthesizes MCP tools from declarative metric definitions. The tool contract and the data contract are unified rather than managed separately. That unification is not a convenience feature: it is what allows the routing layer to make informed decisions about data quality and freshness without requiring routing logic to know the internals of each underlying data system.

Measuring routing quality: the benchmarks that matter and what they miss

End-to-end task success is an insufficient routing metric. It conflates routing quality with tool output quality and synthesis quality. A correct routing decision can produce a bad outcome if the tool returns poor data; an incorrect routing decision can produce a passable outcome if the wrong tool happens to return something plausible. Neither case teaches you anything useful about the routing layer in isolation. Treating task success as a routing signal is the kind of measurement error that feels reasonable until you spend time debugging systems built on it.

The metrics that isolate routing are drawn from retrieval evaluation. Precision: what proportion of tool invocations were appropriate for the query? Recall: what proportion of relevant tools were actually invoked, which matters for multi-tool queries where completeness is required? False positive and false negative rates: the diagnostic tools for identifying skill collisions and informing description rewrites. These metrics require logging at the routing step, which many production systems don't yet instrument explicitly. Adding that instrumentation after the fact is possible but costs more than doing it from the start.

Speed matters as a routing-layer metric independently of accuracy. A routing decision that is correct but adds 400 milliseconds to a user-facing response is a different engineering problem than one that is fast but imprecise. The tradeoff between routing latency and routing quality is real and context-dependent. For an async background pipeline, a more deliberate multi-step routing process may be entirely acceptable. For a real-time user interaction, it may not be. Collapsing these into a single performance budget obscures what's actually being traded off.

The deepest gap in current benchmarks is their failure to capture how routing errors compound in multi-step pipelines. A single-step routing evaluation (did the model pick the right tool for this query) does not capture how a wrong selection at step two propagates through steps three, four, and five. Evaluating routing quality at the pipeline level, tracking how early errors amplify downstream, is methodologically harder but closer to the problem that actually matters in production. Most teams are still relying on end-to-end task success and inferring routing quality from it. That inference is unreliable. The gap it creates is where the next generation of routing instrumentation will need to work, and given how much effort is currently going into routing architectures, it's worth asking why the evaluation side has received so much less attention.