Home/Blog/Per-step model selection is replacing monolithic inference in distributed handoffs
Per-step model selection is replacing monolithic inference in distributed handoffs
October 12, 2026

Per-step model selection is replacing monolithic inference in distributed handoffs because complex AI workflows no longer behave like one prompt, one model call, and one response. When an agent must plan, retrieve data, call tools, delegate work, validate an output, and respond in real time, forcing every stage through the same model and serving path creates avoidable latency, cost, and operational risk.
For platform teams, the practical shift is not simply to add more models to a picker. It is to make each handoff intentional: select an appropriate model, expert path, device, tool, and context package for the current step, then preserve enough state for the next service or agent to continue reliably. This turns inference into a distributed systems discipline with routing policies, observability, fallbacks, and explicit trade-offs.
What per-step model selection means in distributed handoffs
Direct answer: Per-step model selection is an orchestration approach in which a system chooses the model or execution path for each task stage, rather than sending an entire workflow through one fixed model. In distributed handoffs, the system also transfers the required context, tool state, and control signals between specialist services or agents.
A “step” can be a visible stage, such as drafting a plan or reviewing a document, but it can also be an internal action. Examples include classifying intent, deciding whether to retrieve information, selecting a coding subagent, checking a tool result, or producing a short user-facing response. The selection may happen at the workflow level, the agent level, the request level, or, in mixture-of-experts systems, at the token level.
Monolithic inference assumes that one model invocation should carry most of the cognitive and operational burden. That assumption can be convenient for a simple task. It becomes weaker when the workflow contains very different kinds of work: low-latency interaction, structured extraction, long-context reasoning, tool invocation, verification, and multimodal generation.
The unit of optimization is increasingly the end-to-end execution path, not an isolated model call.
OpenAI’s work on the GPT-5.6 efficiency stack makes this system view explicit. Its stated efficiency mechanisms include load balancing, speculative decoding, caching, and kernel optimization. OpenAI also notes that multi-step tasks can require many model requests, meaning small delays at an individual step compound across the complete interaction.
That observation matters for an orchestration workspace. A workflow may be logically correct while still being operationally poor if every handoff repeats context assembly, invokes a high-cost reasoning model for routine routing, or waits on a congested shared execution path. Per-step selection gives teams a way to optimize the actual workflow rather than the benchmark performance of a single model in isolation.
Why monolithic inference breaks down in multi-agent workflows
Monolithic inference is not obsolete for every use case. A single capable model can be the right choice for a bounded task with one input, one output, and no meaningful tool or agent delegation. The architecture becomes strained when one request expands into a coordinated sequence of heterogeneous operations.
Different steps have different requirements
Consider an enterprise operations agent handling a service incident. It may first classify the request, then retrieve runbook material, query observability systems, ask a specialist agent to interpret a deployment change, propose a remediation, and require an approval-aware action path. These steps differ in the capabilities they need and in the consequences of failure.
Intent classification may favor fast, inexpensive execution.
Retrieval planning may need reliable tool selection and a constrained schema.
Incident analysis may need more reasoning effort and longer context.
A tool call needs validated arguments, authorization checks, timeouts, and retry behavior.
A final response needs grounded synthesis, concise explanation, and a record of what was actually done.
Using one model configuration for each stage can over-provision routine work and under-specify critical controls. It also hides the reason for a poor outcome. If a workflow fails, operators need to know whether the issue came from model reasoning, stale context, a routing decision, a tool contract, service saturation, or an unsafe action policy.
Latency accumulates across handoffs
Multi-step work magnifies tail latency. A workflow is only as responsive as its slow dependencies and its coordination over permit. OpenAI’s description of GPT-5.6 emphasizes that small per-request delays compound when a task makes many model requests; in practice, this means a locally acceptable model call can still produce an unacceptable end-to-end experience.
Per-step model selection does not eliminate this problem automatically. It gives the platform more levers: select a faster path for routine decisions, reuse a cached intermediate result, run independent checks in parallel, or reserve deeper reasoning for a step where it changes the outcome. The goal is not to minimize every individual call. It is to meet a workflow-level latency and quality objective.
The serving path is part of the product
OpenAI’s low-latency voice architecture provides a concrete example of distributed handoff design. An edge “transceiver” converts client traffic into simpler internal protocols that support model inference, transcription, speech generation, tool use, and orchestration. Rather than treating voice interaction as one opaque inference endpoint, the architecture separates concerns and creates internal handoff boundaries.
This pattern applies beyond voice. A control plane can accept an incoming request, normalize it into a durable workflow envelope, route it to specialist MCP-connected agents, and return an outcome while retaining traceability. The important design choice is that transport, model execution, tool use, and orchestration can evolve independently without forcing a single monolithic path to own all of them.
How routing moves from model choice to an inference policy
Per-step model selection should be designed as a policy, not as a collection of ad hoc conditional statements. A useful policy combines task signals, operational constraints, and governance requirements. It should be explicit enough to inspect and change, while leaving room for adaptive decisions when workload conditions shift.
At the model layer, product interfaces are already moving toward more granular choices. OpenAI’s ChatGPT Enterprise and Edu release notes describe a simplified model picker intended to help users choose a speed-versus-reasoning balance without changing the underlying models. This is a product-level signal that selection can be framed around the work required rather than a permanent attachment to one model identity.
For orchestrated systems, the same principle becomes more detailed. The router needs to decide not just “which model?” but “what execution route is justified for this step?”
Classify the step.
Identify whether it is a routing decision, extraction task, planning task, tool invocation, verification pass, synthesis stage, or user interaction.
Establish constraints.
Evaluate latency budget, allowed data scope, authorization requirements, expected output format, tool availability, and failure tolerance.
Select an execution path.
Choose a model or specialist agent, determine the reasoning effort where available, and decide whether retrieval, a tool call, or human approval is needed.
Package the handoff.
Send only the necessary task state, evidence, instructions, identifiers, and output contract to the next worker.
Verify and record.
Check the returned artifact against its contract, capture route and timing data, and retain state needed for retries or later review.
A policy should also define defaults and exceptions. A system that always escalates to its most capable path may be expensive and slow. A system that always stays on a fast path may be unable to complete difficult or high-consequence work. The operational value comes from making escalation and de-escalation deliberate.
Signals worth using carefully
Useful routing signals can include task type, requested response time, context size, tool risk, prior validation failures, queue state, and whether a step is user-visible. Teams can also use confidence-like signals where their systems produce them, but should avoid treating an uncalibrated score as a guarantee of correctness.
Policy design must keep sensitive attributes and access controls separate from convenience routing. A model or agent should not receive data merely because it is available or because a route appears efficient. In enterprise environments, the handoff contract needs to enforce tenant boundaries, permissions, retention rules, and tool-specific authorization.
Token routing and expert routing make selection more granular
The shift toward distributed inference is not limited to application-level agent handoffs. Modern mixture-of-experts, or MoE, research treats routing, topology, load balancing, and execution structure as central system concerns. A 2026 arXiv survey describes systems involving token dispatch, device placement, all-to-all communication, and overlap between communication and computation.
In this framing, “single model inference” is often too coarse an abstraction. A request can traverse different experts, devices, or execution paths as tokens are processed. Application orchestration and MoE routing operate at different layers, but they share a core idea: work should go where it can be handled appropriately under current capability and system constraints.
Adaptive routing changes the fixed-path assumption
ProbMoE, released in June 2026, explicitly supports dynamic-k routing. Its routing cardinality can be constrained within a predefined range for training and inference. That is a formal example of adaptive selection: not every token must activate the same fixed number of experts.
This does not mean an agent platform should attempt to reproduce token-level routing logic in its workflow engine. It does mean platform engineers should expect the underlying inference stack to be dynamic. Workflow-level assumptions such as fixed cost, fixed latency, or a uniform execution path may not remain valid when the serving layer itself varies expert activation and device dispatch.
Routing has a real systems cost
More granular choice introduces over. MoRE, published in September 2026, argues that a standard linear router can become a bottleneck at large expert counts. It proposes hardware-aware low-rank routing and reports a fused inference kernel intended to avoid expensive high-bandwidth-memory operations. The implication is important: routing is not free metadata processing around inference. It can become a first-class performance concern.
An empirical 2026 study of MoE inference reinforces the end-to-end view. It found that routing can be a small fraction of decode cost on edge backends, while total parameter footprint, expert dispatch, and KV-cache pressure still dominate. For distributed handoffs, teams should therefore resist over-optimizing the decision function while ignoring payload size, context movement, cache strategy, and dispatch behavior.
Optimize the route decision enough that it is predictable and observable.
Measure context transfer and expert or agent dispatch costs across the full workflow.
Keep handoff payloads bounded and reference
durable state
instead of duplicating large artifacts.
Test congestion and failure behavior, not only median response time.
Design context handoffs that preserve state without copying everything
A distributed handoff is successful only when the receiving agent or service gets the information necessary to act correctly. Passing the full conversation, raw tool outputs, and every prior scratch artifact at each step is rarely the best answer. It increases token use, can expose irrelevant or sensitive data, and makes it harder to identify the evidence behind a decision.
Instead, treat context as structured state with explicit ownership. The handoff should identify the workflow, current objective, permitted actions, relevant evidence, required output schema, and any assumptions that the next step must verify. Large artifacts can be stored separately and passed by reference where the receiving service is authorized to access them.
A practical handoff envelope
A useful internal envelope is not a hidden chain-of-thought transcript. It is an operational contract. It can contain a task summary, a correlation identifier, data classifications, tool permissions, source references, expected completion condition, deadline, retry policy, and a compact record of prior verified outputs.
This structure makes specialist agents easier to compose. A database investigation agent can return a structured finding with evidence references; a remediation agent can consume that finding without inheriting unrestricted access to the original conversation or unrelated systems. The control plane retains the ability to audit who acted, which route was selected, and which tools were invoked.
Persistent intermediate results can reduce recomputation
Recent OpenAI community feedback on persistent memory frames stored intermediate results as an inference-efficiency mechanism. The basic architectural point is sound: if a result has already been created and validated, a later step may be able to reuse it rather than recomputing it.
Reuse needs discipline. Cached or persisted state must be scoped to the correct user, tenant, workflow, and freshness requirement. A stale retrieval result or a prior tool observation can create a confident but incorrect downstream answer. Teams should define invalidation conditions, provenance requirements, and whether a result is reusable as evidence, as a suggestion, or only as a performance hint.
Balance speed, reasoning effort, and monitorability
Per-step selection is a trade-off space, not a universal mandate to use the smallest model first. A 2026 OpenAI paper on monitorability found that, at a fixed capability level, smaller models with higher reasoning effort can improve monitorability, but they consume more inference compute. This creates a design tension between inspectability, responsiveness, cost, and task quality.
For a platform team, that tension should be visible in policy. The system might choose a fast route for a reversible classification step, a more deliberate route for a high-impact recommendation, and an independent verification path before an external side effect. The right sequence depends on what can go wrong and how quickly the workflow must respond.
Use risk tiers, not a single quality setting
A useful operating model is to define service tiers around the consequences of the step. Low-risk steps can prioritize speed and bounded cost. Medium-risk steps can require structured outputs and automated checks. High-risk steps can require additional reasoning, evidence gathering, approval gates, or independent review.
This is especially relevant for tool-backed agents. A response-generation mistake may be recoverable in a chat interface. A mistaken action against production infrastructure, a financial system, or an enterprise record may not be. Per-step model selection allows the system to reserve deeper scrutiny for the moments that deserve it without imposing that over everywhere.
OpenAI’s GPT-5.5 announcement describes use on messy, multi-part tasks where the system plans, uses tools, verifies work, and continues. It also says that serving GPT-5.5 at GPT-5.4 latency required rethinking inference as an integrated system. Together, those points show why model capability and systems design cannot be evaluated separately in long-running workflows.
Build observability around the route, not just the final answer
Once a workflow contains distributed handoffs, final-output evaluation alone is insufficient. A seemingly correct answer may have taken an unnecessarily expensive route, relied on a stale artifact, or nearly failed because a tool retry happened to succeed. Conversely, a poor final answer may originate in a handoff contract or an unavailable dependency rather than the selected model.
OpenAI’s model-inference hiring description references architecting, building, observing, and debugging production distributed systems for performant and efficient model inference. That emphasis is appropriate for agent platforms: model execution belongs inside normal production engineering practices.
Trace the decisions that shape execution
At minimum, capture a trace that lets an operator reconstruct the workflow path without exposing data that should not be retained. A trace should connect the incoming request, route decisions, model or agent invocations, tool calls, validation results, retries, and terminal outcome.
Workflow and handoff identifiers that correlate events across services.
Selected route and the policy reason or rule category that produced it.
Latency for queueing, execution, tool waits, state retrieval, and validation.
Input and output schema validation outcomes, with safely redacted error details.
Cache or persistent-state hits, misses, freshness checks, and invalidations.
Fallback, escalation, cancellation, timeout, and human-approval events.
These records support engineering decisions that aggregate model metrics cannot answer. For example, if tool wait time dominates a workflow, changing the model may not help. If context preparation dominates, payload reduction or state references may offer more value. If a fast route repeatedly escalates, a different initial selection policy may reduce total work.
Evaluate paths, not only models
Evaluation should include representative workflow traces and adversarial operational conditions. Test partial tool outages, delayed dependencies, invalid structured outputs, oversized context, concurrency spikes, and conflicting agent results. The aim is to see whether the system degrades safely and whether operators can understand the failure.
Codex discussions that mention manual model selection, Goal mode, and subagents offer a practical illustration of coordinated work being made more explicit. The lesson for enterprise builders is not that every workflow needs manual intervention. It is that longer tasks benefit when selection, delegation, and goals are visible system concepts rather than hidden behavior inside one opaque call.
Where per-step model selection should not be overused
More routing layers can create their own failure modes. Every decision boundary adds policy complexity, trace volume, possible context loss, and another component that must be tested. A fragmented workflow can be slower and less reliable than a direct call if the task is simple or if handoffs are poorly designed.
Do not split a task merely because specialist labels sound appealing. Split it when the step has a distinct capability need, tool boundary, latency profile, risk profile, or reusable output. Keep a simple path available for simple requests, and use a fallback route when the router lacks enough information to make a confident specialized choice.
There are also limits to what routing can solve. Per-step selection cannot make an unavailable tool reliable, correct an ambiguous business policy, or guarantee factual accuracy from incomplete evidence. It should be paired with tool contracts, access controls, validation, reliable state management, and clear ownership of workflow outcomes.
The best architecture is usually selective rather than maximal. Start with the handoffs that cause measurable latency, cost, reliability, or governance pain. Add specialization where it improves the end-to-end path, then confirm that the added coordination cost is justified.
Adopt per-step model selection through an incremental operating model
Teams do not need to replace all existing inference calls at once. An incremental approach reduces risk and creates evidence for which routes actually improve outcomes. Begin with a workflow that already has recognizable stages, such as intake, retrieval, analysis, action proposal, and final response.
Map the current path.
Identify every model call, tool call, context transfer, retry, and manual checkpoint. Measure end-to-end timing rather than only model latency.
Choose one high-value decision boundary.
For example, separate inexpensive request classification from expensive analysis, or introduce a verifier before an external action.
Define a handoff contract.
Specify required inputs, allowed data, output schema, evidence references, deadline, ownership, and failure behavior.
Implement deterministic guardrails first.
Apply authorization, schema checks, tool allowlists, and retry limits before relying on adaptive selection.
Add routing policy and fallbacks.
Make the default route simple, then define escalation when complexity, risk, or validation failure justifies it.
Instrument and review traces.
Compare whole-path latency, quality checks, tool success, and operator burden across routes.
Expand only after validation.
Turn successful patterns into reusable control-plane primitives for other agents and workflows.
This approach aligns with the broader direction of inference engineering. Load balancing, caching, speculative decoding, kernel optimization, edge protocol adaptation, expert dispatch, and agent delegation all affect the user-visible result. A control plane for multi-agent work should expose these execution choices at the right abstraction level rather than pretending that all intelligence resides in one model endpoint.
Per-step model selection is replacing monolithic inference where workflows are distributed, tool-backed, and multi-stage because it better matches how work is actually completed. The durable advantage is not simply lower cost or faster responses; it is the ability to route work deliberately, retain verified state, govern access at each handoff, and diagnose the complete execution path when something goes wrong.
For platform engineers and operations teams, the next practical step is to identify one workflow where a single inference path is carrying incompatible responsibilities. Make the handoff explicit, select a route based on the step’s requirements, and instrument the result. That creates a foundation for specialist agents that can coordinate effectively without becoming an unobservable collection of model calls.