Why Adding More AI Agents Made Our System Slower

When scaling a multi-agent LLM system at Planck, adding more agents caused latency to increase unexpectedly. The root cause wasn't the LLM providers but Python's event loop being blocked by accumulated CPU-bound work — JSON serialization, token counting, and response validation — that piled up as hundreds of concurrent agent calls completed simultaneously. The GIL meant only one thread could handle this CPU work at a time, creating a bottleneck invisible in small-scale tests. The fix wasn't micro-optimizing individual operations (switching to orjson helped marginally) but redesigning the architecture: introducing a router that fans out requests to multiple workers, each with its own event loop and connection pool, distributing CPU work horizontally across cores and machines. The key insight is that async I/O removes waiting but doesn't eliminate computation — at scale, tiny CPU tasks after each I/O response become the real bottleneck.

Source: https://towardsdatascience.com/why-adding-more-ai-agents-made-our-system-slower. 8sync News only summarizes and links out; content copyright belongs to the authors and original sources.