The KV cache as an agent runtime

Thoughts and our recent work on how sharing and scheduling KV-cache state lets pretrained LLMs observe, reason, and act concurrently without additional training

How KV-cache manipulations can make LLMs more interactive The link has been copied to clipboard

One of the next frontiers in AI systems is interactivity.

An interactive model must be able to receive new information while it is already computing, revise its trajectory without restarting from scratch, emit useful partial actions, and coordinate processes that progress at different rates: perception, reasoning, communication, acting, and tool use.

This requirement applies to language assistants, but it becomes unavoidable for multimodal systems embedded in games, robots, live video, operating systems, and other continuously evolving environments. A model that stops observing whenever it reasons cannot be interactive by design.

A sequential model processes a fixed input while decoding. A shared-state runtime lets observation, reasoning, and output progress concurrently as new information arrives.

We have systems today that are interactive such as the Wan-Streamer models

[1] or Thinking Machines Lab's Interaction Models

[2]. Wan-Streamer handles perception, response timing, speech, and visual generation, Interaction Models replace turn-based exchange with time-aligned micro-turns and combine a real-time interaction model with asynchronous background reasoning. Most existing solutions agree that the common paradigm of sequential tool calls, thinking, input and output streams is not enough for interactive systems. They also illustrate one path toward solving the problem – changing the model through pretraining, post-training, new data formats, block-causal attention, streaming encoders, or additional policy heads.

We explore how much of this behavior can be achieved at inference time, without changing model weights.

Our position

Our position is that a significant part of interactivity is also an inference-runtime problem. Existing pretrained models already expose reusable execution state. By changing how this state is partitioned, ordered, exposed, and scheduled, an inference engine can implement interaction protocols that were absent from the original training procedure.

For Transformers, that execution state is primarily the KV cache.

From parallel prompting to concurrent execution The link has been copied to clipboard

Parallel LLM inference is not new. Methods such as self-consistency

[3], multi-agent debate

[4], and Skeleton-of-Thought

[5] exploit multiple independent or partially structured generations, while approaches such as PASTA

[6] and Parallel-R1

[7] explicitly target parallel reasoning within a single model.

Related work also introduces asynchrony at the systems level: AsyncLM overlaps generation with tool execution

[8], while LLMCompiler parallelizes independent tool calls

[9]. Speculative Programmatic Tool Calling introduces asynchronous execution at the harness level

[10].

These methods place parallelism at different layers – external orchestration, learned generation structure, or tool execution. Hogwild! Inference

[11] and AsyncReasoning

[12] explore a different point in this design space: changing the execution protocol and shared model state while keeping the model weights fixed.

KV cache could serve more than just inference optimization The link has been copied to clipboard

The KV cache is usually presented as a decoding optimization. During autoregressive inference, the keys and values computed for previous tokens remain unchanged, so the model stores them rather than recomputing the full prefix at every step.

Under this description, the cache is passive and frozen: it makes the same sequential computation cheaper.

We argue that this view is limiting. The KV cache is the model’s active execution state. It determines which previous computations are visible to each new query, their causal ordering, and which partial results can affect the next token. Changing the cache therefore changes the computation exposed to the model, even when the parameters and token representations remain unchanged.

Instead of representing inference as one prompt followed by one output sequence, we can represent it as a collection of evolving cache blocks. A block may correspond to a user-input stream, a reasoning process, a public response, another model worker, a tool result, or a visual observation. Different consumers can receive different causal views over the same physical blocks.

For RoPE-based models, these views do not require repeatedly re-encoding or physically copying the cache. Let $\rho(x,i)$ denote applying the rotary transformation for position $i$. Because RoPE attention depends on relative position, an attention product can be rewritten as

$$\rho(q,i_q)\rho(k,i_k)^\top = \rho(q,i_q-i_k)k^\top$$

A cache block can therefore be stored once in block-local coordinates. At query time, the inference kernel applies a block-specific rotation to the current query, making that same block appear at a different logical offset in each stream’s view.

This converts the cache from a flat append-only tensor into a reusable, multi-view memory.

A cache block is stored once in block-local coordinates and can appear at different logical offsets in different attention views.

Hogwild! Inference: collaboration through shared state The link has been copied to clipboard

We first explored this abstraction in Hogwild! Inference [11].

Traditional multi-agent systems generally impose a collaboration strategy outside the model: vote over independent samples, debate for a fixed number of rounds, assign specialized roles, decompose the task into predefined branches, or execute an externally generated task graph.

These structures can be effective, but no single structure is appropriate for every problem. Fixed decomposition may fail when the initial plan is wrong. Parallel branches may duplicate work. A system may wait for a straggling subtask after the rest of the plan has become irrelevant. Replanning typically requires another orchestration layer.

Hogwild! Inference instead runs several instances of the same pretrained model concurrently and allows them to write into a shared attention memory. Each worker sees the other workers’ partial generations immediately, rather than after a complete message or reasoning trajectory has been produced.

The workers can then decide at generation time whether to divide the task, verify another solution, pursue an alternative derivation, detect redundant work, or continue a promising argument started elsewhere. The runtime provides concurrency and visibility, while the pretrained model supplies much of the collaboration policy.

Technically, each worker (let’s call them Alice and Bob) needs a different logical ordering of the cache. Alice should see the common prompt, Bob’s current work, and then Alice’s own tokens as the immediate continuation. Bob should see the common prompt, Alice’s work, and then Bob’s tokens. Re-encoding every block for every worker would erase much of the efficiency gain.

Hogwild! stores worker histories as reusable blocks and uses per-block query rotations to construct these views inside the attention kernel. This is similar in spirit to paged attention, except that the same physical page may occupy a different logical position for each worker.

Hogwild! workers share physical cache blocks while attending to different logical orderings of the prompt and worker histories.

In the end modern reasoning models can perform nontrivial shared-state coordination without having been trained as multi-agent models.

AsyncReasoning: thinking and communication should not block each other The link has been copied to clipboard

AsyncReasoning applies the same runtime principle to a different bottleneck [12].

Reasoning models are commonly deployed through a strict read–think–answer loop: encode the complete input, generate a private reasoning trace, and only then produce visible output. This is reasonable for offline evaluation. It is a poor protocol for interactive systems.

A voice assistant should not remain silent for minutes before acknowledging a request. A game agent should not stop observing while planning. A safety monitor should not freeze perception while composing an explanation. An agent that receives a correction halfway through a long reasoning trace should not have to discard all previous computation.

AsyncReasoning divides inference into three logical streams: incoming user information, private reasoning, and public output. The thinker continues generating internal reasoning while the writer produces visible text using the thinker’s partial progress. The thinker also observes the current response and can pause the writer when the available reasoning is insufficient.

Both streams reuse the same encoded tokens, but receive different logical contexts. In the writer view, partial thoughts precede the public response. In the thinker view, the current public response appears as an earlier interaction, allowing the model to reason about what it has already communicated.

AsyncReasoning lets private reasoning and public output progress concurrently through separate views over shared KV-cache state.

This positioning is distinct from merely reducing the amount of reasoning. Other methods like DEER

[13] or ThinkSwitcher

[14] reduce or route inference-time compute; they do not make communication concurrent with that compute.

It is also distinct from sequential interleaving. Plantain trains models to alternate between private reasoning and useful intermediate responses, with an explicit plan surfaced early

[15]. StreamingThinker overlaps reading and thinking

[16]. LLM-based simultaneous-generation policies decide when to read more input and when to write

[17]. Mind-Paced Speaking uses a specialized dual-brain architecture to overlap formulation and speech

[18].

AsyncReasoning targets a separate axis: overlapping thinking and writing, while also allowing additional inputs to enter an active inference process. The mechanisms are complementary. A future system could think while reading, think while speaking, execute tools asynchronously, and use learned policies to control all of these transitions.

In the experiments reported in the paper, this runtime transformation reduces time to the first non-thinking token by up to $80\times$ and total user-perceived delay by up to $12\times$, while preserving much of the accuracy benefit of reasoning [12].

The conceptual shift is that reasoning and communication become independently scheduled processes. “Thinking longer” no longer has to imply “responding later.”

Multimodal agents are asynchronous I/O systems The link has been copied to clipboard

This cache-based view is not limited to text. Modern multimodal models map images, video, and audio into token-like representations that are consumed in the same model context as text. Once a modality is represented as a sequence of states, the same questions arise: which process owns those states, when are they appended, and which other streams may attend to them?

An interactive agent can be represented as a set of asynchronous input and output channels:

• Text input
• Vision
• Audio
• Reasoning
• Speech output
• Actions
• Tool calls
• Tool results;

These streams do not naturally begin or finish at the same time. A camera continues producing observations while the model reasons. The user may interrupt while the agent is speaking. A tool may return after the agent has already started another task. An action may need to be emitted before the full long-horizon plan is complete.

Forcing these events into a synchronous request–response queue creates idle time and stale state. A multi-stream runtime instead treats them as asynchronous I/O operations. New observations can be encoded while reasoning continues. Tool results can become visible immediately after completion. Actions can be generated without waiting for unrelated internal work. The scheduler controls stream visibility and synchronization without restarting the model from the original prompt.

This is closely related to the Asynchronous I/O abstraction proposed for Speculative Interaction Agents

[19], where the main reasoning-and-action process is decoupled from waiting for user or environment events. Our goal is to generalize this idea from agent-level control flow to the model-serving layer: streams should be first-class inference objects rather than messages passed between blocking model calls.

A training-free Doom agent The link has been copied to clipboard

As a concrete demonstration, we are developing a system in which a multimodal Qwen3.5 model plays Doom while its visual context is updated continuously.

The environment produces a stream of frames. New visual tokens are inserted into the active context as the game evolves. In parallel, the model maintains a reasoning stream and emits keyboard or controller actions. Observation, reasoning, and acting are therefore becoming concurrent views over a persistent inference state.

Crucially, this setup requires no additional model training. The model is not fine-tuned on Doom trajectories or retrained with a specialized streaming objective. The interaction protocol is implemented at inference time by reorganizing how visual observations, reasoning tokens, and action outputs update and access shared state.

In Doom, the environment keeps changing while the model thinks: an action chosen from one frame may already be outdated when it is executed. The agent needs to incorporate new frames and revise its actions while reasoning is still in progress. We are using this setup as a testing ground for the runtime for models that interact with continuously changing environments, and will describe the framework and implementation in an upcoming paper.

A training-free Doom agent: visual observations and generated actions progress concurrently.

Building the serving stack The link has been copied to clipboard

These methods should not remain isolated research kernels wrapped in Python control loops. We are working on an SGLang implementation of the broader asynchronous I/O framework, together with custom kernels for efficient GPU inference of multi-stream and multi-view setups

[20].

The serving problem involves more than splitting a cache into tensors. A practical runtime must support named streams, block-local positions, per-consumer visibility rules, asynchronous append operations, interruptions, mode switches, tool completions, continuous batching, and reclamation of long-running state.

At the kernel level, attention should process differently transformed queries against multiple cache blocks without materializing separate cache copies. Recurrent layers should compose affine block summaries efficiently. At the scheduler level, independent streams should be batched whenever possible without forcing all requests into the same synchronization point.

The target abstraction resembles an operating-system runtime more than a conventional text-generation endpoint. Streams are processes. Cache blocks are shared memory. Attention views are access mappings. Mode-switching decisions are scheduling events. Tool and environment updates are interrupts.

This analogy should not be taken too literally, but it points toward the systems interfaces that interactive models require.

Toward interactivity as inference primitive The link has been copied to clipboard

One way to view our preliminary results is that some aspects of interactivity may be better handled by the inference runtime rather than by the model architecture alone.

In this view, the cache is not only a tensor attached to a single request, but a structured model state that can potentially support multiple blocks, views, and streams. Similarly, observations, reasoning, tool results, and outputs do not necessarily need to progress through one strictly blocking sequence.

This also suggests that some agent roles could be implemented as different views over shared model state rather than as completely separate model invocations. Which parts of this should live in the runtime, and which should be controlled by prompts, learned policies, or external orchestration, remains an open design choice.

We see training-free runtime methods and interactive training as complementary. Runtime mechanisms provide a convenient way to explore new interaction protocols without modifying model weights, while training can later improve behaviors such as stream coordination, response timing, and robustness.

Finally, for interactive systems, conventional throughput metrics such as tokens per second may not capture the full picture. Measures such as time to first useful action, reaction latency to new observations, and the amount of computation that can be overlapped with perception or tool execution may be equally important.

Limitations The link has been copied to clipboard

Training-free interactivity comes with clear limitations. Models trained for sequential interaction may handle partial or stale information poorly, duplicate work across streams, or make the wrong decision about when to wait and when to act. On the systems side, cache manipulation complicates batching, memory management, speculative decoding, distributed inference, and support for hybrid architectures. Multimodal settings add another layer of difficulty because encoders and action interfaces also need to operate incrementally.

But these limitations are exactly why we think this direction is worth studying.

A large part of the current agent frontier is framed around better models, better post-training, and more capable harnesses. Runtime-level interactivity points to another axis: changing how an existing model is executed, how its state is exposed, and how concurrent processes interact with that state.

This does not replace interactive training or specialized architectures. It gives us a different way to ask where agent capabilities should come from: model weights, orchestration logic, or the inference runtime itself.

Hogwild! Inference and AsyncReasoning suggest that this third component may have more room than the standard request–response interface makes apparent.

Acknowledgment The link has been copied to clipboard

We thank Denis Mazur, Vyacheslav Zhdanovskiy, Vladimir Bartenev, Vadim Pastushenko, Timofey Byzov and Vladimir Kaurkin for their contributions to our ongoing work on asynchronous agents and interactive LLM inference. Their ideas, discussions, experiments, and engineering efforts are helping shape the systems described here.

Correspondence email: yakushev-ga@yandex-team.ru