The open AI-native stack on Kubernetes: From open models to distributed inference and agents

Google Cloud outlines an open-source, Kubernetes-native stack for running AI workloads at scale, covering RL post-training, distributed inference, and agent execution. Key components include OpenRL for RL fine-tuning, llm-d 1.0 for distributed inference, KubeRay for distributed Ray workloads, and Agent Substrate plus gVisor for sandboxing untrusted agent code. The Endpoint Picker in llm-d doubled KV cache hit rates from 35% to 70% and cut median time-to-first-token by 70-85% on multi-turn agent workloads. Testing Gemma 4 with OpenRL's RL loops on Harvey.ai's Legal Agent Benchmark lifted accuracy from 50% (SFT) to 70%. The projects are presented ahead of KubeCon + CloudNativeCon North America 2026.

Source: https://cloud.google.com/blog/topics/developers-practitioners/the-open-ai-native-stack-on-kubernetes-from-open-models-to-distributed-inference-and-agents. 8 Sync News only summarizes and links out; content copyright belongs to the authors and original sources.

Recommended for you

Daily Dose of Data Science | Avi Chawla | Substack51 Hot8 min2 d agoTop

7 MoE routing strategies in production, clearly explained

+ a practical learning checklist to learn AI engineering.

#ai-inference#deepseek#mixture-of-expertsSource

Flavio Copes133 Hot6 min9 d ago

How to run an LLM locally

Run an LLM locally with Ollama. See how much memory you need, where LM Studio and llama.cpp fit, and how to call the model from code.

#self-hosting#ai-inference#ollamaSource

Daily Dose of Data Science | Avi Chawla | Substack151 Hot7 min10 d ago

RAG vs. Vectorless RAG, clearly explained!

...explained visually (with tradeoffs in production).

#llm#rag#vector-searchSource

Latent Space1 Hot4 min7 h ago

What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team

Handshake went from launch to $1B in one year pushing the envelope of specialized data for frontier labs. Our first deep dive into a data lab with specific examples across Finance, Bio, and Alignment

#machine-learning#ai-agent#reinforcement-learningSource

Machine Learning Pills1 Hot9 min8 h ago

Issue #147 - Reinforcement Learning Explained with a Practical Python Example

How agents learn from rewards, why exploration matters, and how to build your first Q-learning agent.

#python#reinforcement-learning#numpySource

InfoQ1 Hot34 min13 h ago

Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM

Edoardo Vacchi covers the evolving landscape of WebAssembly on the backend, running Java workloads on GPUs using TornadoVM, and optimizing distributed AI inference using llm-d and vLLM on Kubernetes.

#kubernetes#webassembly#vllmSource

ITNEXT26 Hot56 min6 d ago

Kubernetes Multi-Tenancy, Layer by Layer: Isolation Proved by Being Refused

“” is published by Filipe Motta in ITNEXT.

#kubernetes#multi-tenancy#ciliumSource

Isovalent1 Hot6 min15 h ago

A refreshed Network Policy Editor website and two new VS Code extensions

Meet the refreshed Network Policy Editor and two new VS Code plugins.

#kubernetes#vscode#ciliumSource