Session-Aware Agentic Inference with NVIDIA Dynamo

NVIDIA Dynamo introduces a unified session-level identifier that turns request-level inference serving infrastructure into program-aware infrastructure for agentic workloads. The system recognizes session headers from coding agents like Claude Code, Codex, and OpenCode out of the box, supports plugins for other harnesses, and offers a custom header for any harness. This session ID powers session-linked tracing and replay (via AISimulate and AIPerf), session-aware scheduling that pauses and resumes programs based on KV pool utilization thresholds to reduce cache thrashing, a shared KV cache indexer that extends routing visibility into external stores like Mooncake, and a proposed KvHint interface (Share, Prefetch, Demote, Pin, Retain) letting the router express cache movement policy while engines execute it. Benchmarks show the ThunderAgent-based scheduler improving throughput roughly 12-16% over KV-aware routing alone on SWE-bench, and 11-14.6% higher model-token throughput on agentic RL rollouts at medium concurrency while maintaining over 94.5% prefix-cache hit rate.

Source: https://pytorch.org/blog/session-aware-agentic-inference-with-nvidia-dynamo. 8 Sync News only summarizes and links out; content copyright belongs to the authors and original sources.