Reworking workflow layers now that context is sent with every request

AI workflow architecture is moving beyond the assumption that a model call is a self-contained event. When every request can carry a deliberately assembled package of instructions, user intent, retrieved records, tool results, memory, and workflow state, the central engineering question changes. The task is no longer simply to write a stronger prompt; it is to decide what the model should know for this step, what should remain outside the request, and which system layer is accountable for making that decision.

For platform engineers and teams operating specialist agents, this shift has practical consequences. Routing, state management, retrieval, validation, audit trails, and persistent transport can no longer be treated as incidental plumbing around a prompt. They become explicit workflow layers with contracts, ownership, and measurable behavior. Context-first workflow design is replacing monolithic prompt stuffing because reliable agentic systems need each request to be complete enough to act correctly, but focused enough to act on the task at hand.

Why per-request context changes the workflow boundary

In an earlier chat-centric design, an application could treat conversation history as the primary state mechanism. A user message arrived, the application appended it to prior messages, and the model received the expanding transcript. That pattern remains useful for simple interactions, but it is weak as the foundation for tool-backed, multi-agent work.

A production request commonly needs inputs that do not belong to the visible chat at all. Current architecture references describe a request blob assembled from system prompts, user input, retrieved data, tool outputs, memory, and prior turns. In an enterprise workflow, it may also need tenant policy, identity and authorization scope, task status, agent handoff notes, schema constraints, and the minimum evidence needed to support an answer.

OpenAI’s recent engineering guidance states that errors often come from “insufficient context gathering or analysis.”

That observation matters because it identifies a failure mode that prompt wording alone cannot solve. An agent can follow a well-written instruction and still produce a poor result if it was not given the relevant account record, current tool state, governing policy, or source material. Conversely, providing every possible artifact can bury the active task in stale or irrelevant material.

From a prompt boundary to a request boundary

The useful unit of design is therefore the request boundary: the point at which an orchestration system commits a selected set of information to a model invocation. At that boundary, the workflow should be able to answer clear operational questions:

• What user objective is this request advancing?

• Which specialist agent is responsible for the next action?

• Which facts, documents, prior outputs, and tool results are relevant now?

• What authority does the agent have to read data, call tools, or propose an action?

• What output format, validation rules, and evidence requirements apply?

• Which state should be retained after the step finishes?

These questions turn context into an engineered artifact rather than an accidental by-product of a conversation transcript. They also create a better interface between agents. Instead of handing a downstream agent an opaque wall of upstream reasoning, an orchestrator can pass a compact task brief, verified facts, references, constraints, and a structured handoff state.

This does not mean every request must be large. It means every request should be intentional. The design tradeoff is context quality versus context volume: enough material to make a grounded decision, without forcing the model to reason through work that is no longer relevant.

Replace monolithic prompt stuffing with explicit workflow layers

Recent 2026 architecture guidance commonly separates an LLM application into layers for ingress or API handling, an AI gateway, context assembly, model execution, output validation, observability, and a feedback store. The exact number of services is less important than the separation of responsibilities. A team should be able to change routing logic without rewriting retrieval, revise validation without changing the model transport, and inspect a failed request without reconstructing it from scattered logs.

A practical seven-layer operating model

1. Ingress and API layer.

   Authenticate the caller, establish tenant and user scope, normalize the incoming event, apply rate or policy controls, and create a traceable request identity. This layer should avoid deciding what the model ought to know beyond the information necessary to establish a safe request envelope.

2. AI gateway and routing layer.

   Select a model, provider path, specialist agent, and execution mode. Routing can account for task type, latency requirements, tool eligibility, cost controls, or organizational policy. It should remain separate from model reasoning so that routing choices are inspectable and testable.

3. Context assembly layer.

   Gather and rank the inputs needed for the current step. This includes stable instructions, scoped retrieval, approved memory, relevant prior turns, tool outputs, and structured handoff artifacts. It is the layer that enforces the context budget and provenance rules.

4. Model execution layer.

   Invoke the selected model using the assembled request. This layer manages the model-facing protocol, structured output expectations, retries where appropriate, and transport-specific capabilities such as persistent connections or state references.

5. Output validation layer.

   Check that the output meets syntax, schema, policy, and task requirements before it becomes input to another agent or an external action. Validation may route work for repair, request additional evidence, or prevent an unsafe tool call.

6. Observability layer.

   Record the model request, response, metadata, timing, tool calls, routing decisions, and validation outcomes in a governed form. LLM observability needs more than ordinary HTTP-level application performance monitoring because the semantic contents of each call affect quality and risk.

7. Feedback and state layer.

   Retain approved summaries, task outcomes, evaluation signals, and workflow state. This layer should distinguish durable operational records from temporary model context and from content that should not be retained.

MLflow’s 2026 architecture guidance similarly identifies a workflow layer responsible for request routing, sequencing, tool calling, and state management across multi-step interactions. That framing is valuable for enterprise teams because it prevents the model call from becoming the hidden coordinator of the entire system. The orchestrator owns the workflow; the model contributes bounded reasoning and generation within it.

In practice, these layers can start as modules in one service. A mature platform does not need seven separate deployments to gain the benefits. What matters is that interfaces are explicit: context assembly returns a governed package, execution returns a typed result, validation emits a decision, and observability links all of it to the same workflow trace.

Build context assembly as a product capability, not a helper function

Context assembly increasingly deserves dedicated ownership, tooling, and tests. If it is hidden inside agent prompts or scattered across tool adapters, it becomes difficult to explain why an agent saw a particular fact, why it missed a relevant policy, or why its request size grew over time. A first-class context layer makes those decisions visible.

Classify context by purpose and lifetime

A useful starting point is to separate context into categories with different acquisition and retention rules. Stable system instructions are not the same as task-specific evidence. A tool result from the current step is not the same as a long-lived user preference. Treating all of them as interchangeable chat messages leads to unnecessary payloads and ambiguous authority.

• Invariant instructions:

  role boundaries, output contracts, safety rules, and organization-wide operating guidance.

• Request identity and policy:

  tenant, user role, data entitlements, workflow objective, and applicable approval requirements.

• Task evidence:

  retrieved documents, records, citations, and facts selected for the present decision.

• Live execution state:

  current plan stage, tool-call results, errors, pending approvals, and idempotency details.

• Memory:

  compact, approved information that remains useful beyond a single step, such as durable preferences or a validated project summary.

• Handoff state:

  structured outputs from one specialist to another, including what was completed, what remains unresolved, and what evidence supports the handoff.

This classification helps answer a critical question: should this item be sent to the model, stored for later retrieval, attached only to a tool call, or kept outside the model path entirely? For example, an access-control decision should be enforced by the platform rather than presented as optional prose for a model to interpret.

Use a context manifest

For high-value workflows, create a machine-readable manifest alongside the final model request. The manifest does not need to expose sensitive content in every operational view. It should identify the source class, retrieval or selection method, timestamps where relevant, authorization scope, truncation or summarization actions, and the reason each artifact was included.

A manifest supports incident review and evaluation. When an answer is wrong, a team can distinguish among several causes: the required source was unavailable, retrieval ranked it too low, summarization omitted a condition, the routing layer chose the wrong specialist, the model failed to use supplied evidence, or validation allowed an unsupported output through. Without this separation, every failure gets mislabeled as a generic model-quality issue.

OpenAI’s guidance to use more opinionated instructions and citations for factual claims from retrieved context also fits naturally here. The assembler can attach source material and the execution contract can require claims to be grounded in that material. The validator can then check whether required citations or references are present before a response is released.

Choose selective retention over full-history replay

Sending full conversation history on every turn is easy to implement, but ease is not an architecture strategy. It couples cost, latency, and relevance to the total length of a session, even when the next task depends on only a small portion of what happened earlier. It can also cause a model to focus on previous agent work rather than the current objective.

The stronger pattern is selective retention: preserve the state needed for continuity, then assemble only the relevant subset for a specific request. This may combine recent tool-call pairs, a compact task summary, durable memory, and freshly retrieved evidence. The approach is especially important in multi-agent systems, where each handoff should reduce ambiguity rather than replicate the entire chain of work.

What the benchmark evidence supports,and what it does not

A June 2026 arXiv paper evaluated GPT-5 configurations on a 50-task hotel-expense benchmark. For that workflow class, pruning to recent tool-call pairs plus compact summarization improved reliability and efficiency compared with full-history retention. This is useful evidence for selective retention, but it is not a universal rule that every workflow should summarize aggressively.

Context policies should depend on the task. A compliance review may require precise historical excerpts. A support workflow may need the latest account status and recent troubleshooting actions. A research agent may need a broader evidence set but not every intermediate draft. The point is to test a policy against the actual workflow rather than assume that longer history equals better continuity.

A retention policy for specialist agents

1. Keep raw records in their system of record.

   Do not use a model transcript as the sole durable store for critical business state.

2. Persist structured workflow state separately.

   Store task status, approvals, tool results, and identifiers in forms that downstream services can verify without model interpretation.

3. Summarize only after validation.

   A summary that contains an unsupported claim can spread that error to every later request. Generate or approve summaries under a clear evidence and validation policy.

4. Retrieve by current task, not by chronological convenience.

   Rank information according to the next decision, required entities, applicable policy, and recency where relevant.

5. Set explicit expiration and refresh rules.

   Live data, permissions, and tool state can change. A retained summary should not silently substitute for a fresh source when current status matters.

This approach makes agent handoffs cleaner. A billing specialist does not need a complete support transcript; it needs verified account identifiers, the customer’s request, relevant entitlement facts, completed diagnostic steps, and any required policy citations. A downstream agent can receive that package with a clear confidence and provenance boundary.

Use persistent session state without giving up request discipline

Per-request context does not require resending identical content on every request. OpenAI’s recent Responses API WebSocket work highlights the cost of treating each request as independent and reprocessing conversation state and reusable context on every follow-up. Its redesign moves away from new HTTP connections plus full history each turn toward persistent WebSocket transport and cached state.

OpenAI reported close to a 45% improvement in time to first token from this workflow redesign, while noting the result was still not fast enough for the newest Codex variant it was targeting. The precise performance outcome belongs to that implementation, not to every agent system. The architectural lesson is broader: repeated prompt transmission and repeated state processing can be a workflow-layer bottleneck, and persistent state can reduce that over.

Separate transport efficiency from context selection

It is tempting to treat persistent sessions as a reason to stop designing requests carefully. That would recreate the full-history problem in a different protocol. A cached session may avoid transmitting a large payload again, but the orchestration layer still needs to control which state is active, whether cached context remains authorized and current, and how it affects the next decision.

A disciplined design can use persistent connections for responsive interaction while retaining a logical context manifest for every model step. The physical request may reference cached state; the workflow record should still state what that state represents, which additions were made for this turn, and what instructions or evidence governed the result.

• Use persistent transport when interactive latency and repeated session setup are meaningful constraints.

• Cache reusable, policy-approved material rather than indiscriminately caching all prior content.

• Invalidate or refresh cached state when permissions, data freshness, workflow stage, or user intent changes.

• Keep tool actions and durable workflow state independently traceable, even if the model session remains open.

• Test reconnect, failover, and session-expiry paths so that continuity does not depend on one transient connection.

For a control plane that routes work across MCP-connected specialists, this distinction is operationally important. A persistent session can improve responsiveness inside a specialist interaction, while the control plane still owns cross-agent state, authority boundaries, and handoff contracts. Session convenience should not become hidden orchestration.

Make routing, sequencing, and tool use explicit orchestration concerns

As enterprise AI adoption broadens beyond technology-led product embedding into operational and workflow deployments, the value of an agent system increasingly comes from coordinated execution. A useful workflow may classify a request, retrieve records, call approved tools, hand work to a specialist, validate a structured result, and seek approval before making a consequential change. No single prompt should be expected to govern all of those responsibilities.

Routing and sequencing deserve their own policies because they determine which context is needed and when. A retrieval specialist needs a search objective and access scope. A policy specialist needs the relevant policy corpus and case facts. An execution agent needs a validated action plan, tool permissions, and idempotency controls. Passing the same giant prompt to each agent is both inefficient and difficult to audit.

Design handoffs as typed contracts

A handoff should contain more than natural-language prose. Define a contract that includes the task objective, input identifiers, evidence references, completed steps, unresolved questions, output schema, confidence or review status, and the next agent’s permitted actions. The receiving specialist can then focus its context budget on its own work rather than reverse-engineering upstream intent.

Typed handoffs also simplify recovery. If a tool call fails, the workflow engine can retry or route to remediation with an explicit state record. If validation rejects a result, the engine can invoke a repair step with the failed schema or missing evidence details. This is more reliable than asking a general agent to infer what failed from an unstructured transcript.

Keep deterministic controls outside model judgment

Models can help interpret intent, draft plans, and produce structured candidates. They should not become the only place where deterministic business controls live. Authorization, tenant isolation, rate limits, approval gates, tool allowlists, data retention rules, and idempotency are platform responsibilities. The workflow layer should enforce them before and after a model call.

This division supports both safety and throughput. It narrows the reasoning task given to the model, makes critical controls repeatable, and gives operators concrete places to inspect policy enforcement. It also limits the blast radius of a weak or unexpected model output: a generated instruction does not become an action until it satisfies the tool and workflow contracts around it.

Validate and observe every request as a complete workflow event

When context is assembled per request, the request becomes one of the most valuable diagnostic objects in the system. Braintrust’s 2026 observability framing describes LLM call observability as capturing the full request, response, and metadata for every model API call, extending beyond conventional HTTP-level APM. That distinction is essential: transport success does not show whether the model saw the right evidence, followed a schema, or made a claim that the supplied sources support.

Validation should match the kind of output

Not every response requires the same guardrails. A draft for internal review may need formatting and provenance checks. A workflow that proposes a database update may require a strict schema, referential validation, a policy check, and human approval. A multi-agent handoff may require verified completion status and explicit unresolved items.

• Structural validation:

  parse structured output, enforce schemas, and reject malformed tool parameters.

• Grounding validation:

  require citations or references when a task depends on retrieved facts, and check that claims are traceable to the provided context where the workflow supports that check.

• Policy validation:

  apply data handling, authorization, and business rules independently of model prose.

• Execution validation:

  confirm tool preconditions, approval status, idempotency keys, and target scope before an external action.

• Quality validation:

  use task-specific checks, evaluations, or review queues for outputs that cannot be fully verified with deterministic rules.

Validation is not merely a final gate. Its results should feed the next workflow decision. A missing citation can trigger evidence retrieval. A schema failure can trigger a constrained repair request. A policy conflict can route the task to an approval queue. By making these outcomes first-class state transitions, teams avoid treating failures as silent model quirks.

Trace the context decision, not only the output

A reliable trace connects ingress identity, routing choice, context manifest, model configuration, request and response references, tool calls, validation outcomes, latency, and final workflow status. Sensitive data should be governed appropriately, with access controls and redaction strategies suited to the deployment. Observability does not justify indiscriminate logging; it requires enough governed evidence to investigate behavior responsibly.

Review traces at the workflow level. A low-quality final answer may originate in a retrieval miss, an outdated cached state, an incorrect specialist route, a failed tool call, or a validator that accepted a weak handoff. Measuring only model latency or token usage cannot reveal these distinctions. Per-request context makes the necessary causal chain available, provided the platform records it.

Adopt context-first workflow design incrementally

Reworking workflow layers does not require a wholesale rewrite. The pragmatic path is to identify where context is currently assembled implicitly, make that process observable, and then move high-risk or high-volume flows toward explicit contracts. Start with workflows where wrong context has clear operational cost: tool execution, customer operations, compliance-sensitive tasks, or multi-agent handoffs.

A staged implementation plan

1. Inventory the current request payload.

   Identify everything that reaches the model today: system instructions, chat history, retrieved content, tool outputs, hidden application state, and repeated boilerplate. Measure the shape of the request before trying to optimize it.

2. Define the workflow objective and state model.

   Describe the stages, responsible agents, allowed tools, approval points, and durable records. This makes it possible to separate genuine workflow state from transcript residue.

3. Introduce a context manifest.

   Record why each source was selected, how it was transformed, and which policy scope applies. Begin with one workflow and use incident reviews to improve the manifest fields.

4. Extract routing and validation from prompts.

   Move deterministic selection, authorization, schemas, and approval rules into the

   orchestration layer

   . Leave the model with a focused reasoning or generation task.

5. Test selective retention policies.

   Compare full-history replay with task-relevant retrieval, compact summaries, and recent tool state on representative cases. Evaluate reliability, latency, and operational usability together.

6. Use persistent state where it solves a measured bottleneck.

   Add persistent transport or cached session state for interactive workflows that repeatedly transmit reusable context, while preserving explicit invalidation and traceability.

7. Close the feedback loop.

   Feed validation failures, operator corrections, and workflow outcomes into evaluations and context-selection improvements rather than making ad hoc prompt edits after every incident.

Ownership is as important as implementation. Platform engineering may own gateway policy, tracing, and execution reliability. Product or operations teams may own task definitions and acceptance criteria. Domain teams may own the source corpus and policy interpretation. A shared context contract lets these groups contribute without each embedding inconsistent assumptions in separate agent prompts.

The goal is not to eliminate prompts or to make every workflow mechanically complex. Prompts remain useful for behavior, tone, and local reasoning guidance. The change is to stop asking prompts to carry the burden of identity, state, retrieval, policy, transport, validation, and orchestration all at once. Those concerns are better handled by the layers designed for them.

Reworking workflow layers around per-request context gives agent platforms a more defensible operating model. Each specialist receives a bounded, relevant, and attributable package of information; each tool action passes through explicit controls; and each model output can be evaluated in the context that produced it. That foundation supports faster iteration because teams can improve a retrieval policy, handoff contract, or validator without turning every change into a prompt rewrite.

The durable principle is simple: send the right context for the current request, not all available context by default. Combine selective retrieval and retention with persistent state where it reduces repeated work, then make routing, validation, and observability visible parts of the workflow. For enterprise multi-agent systems, that is the path from conversational prototypes to controlled, tool-backed operations.