Optimizing distributed AI inference: Advanced deployment patterns

How to use traces to avoid breaking changes

Việc sử dụng tracing giúp phát hiện sớm các vấn đề tiềm ẩn khi thay đổi hệ thống bằng cách theo dõi luồng dữ liệu và sự kiện trong môi trường phân tán. Các thư viện phổ biến như OpenTracing, OpenTelemetry, Zipkin và Jaeger hỗ trợ giám sát, trong khi Digma cung cấp phản hồi tức thì trong quá trình phát triển.

Lập trình viên nên đọc bài này để hiểu cách sử dụng tracing để phát hiện và tránh các break changes trong hệ thống phân tán, từ đó giảm thiểu rủi ro khi cập nhật hoặc mở rộng ứng dụng.

The Hot Path Belongs to GBDTs, Agents Own the Cold Path: A Payment-Fraud Benchmark

A reproducible benchmark comparing gradient-boosted decision trees (GBDTs) vs. LLM-based scoring for payment fraud detection across three dimensions: latency, cost, and determinism. On a single CPU core, GBDTs hit p99 latency of 0.15ms vs. ~1,200ms for LLMs — well outside the 100ms ISO 8583 authorization budget. Cost-wise, GBDTs run ~$54/hour at 50K TPS vs. $16,200–$351,000 for LLM tiers. Determinism is the most critical issue for regulated environments: GBDTs return identical scores on identical inputs while LLMs produce hundreds of distinct outputs even at temperature=0. The recommended architecture keeps deterministic tree ensembles on the synchronous hot path and deploys LLM agents on the asynchronous cold path for SAR drafting, evidence gathering, and agent-as-a-judge validation before human review. All benchmark code is open-source and reproducible on a laptop.

How we built saga rollbacks for Cloudflare Workflows

Cloudflare Workflows now supports saga-style rollbacks, letting developers attach compensation logic directly to each step.do() call. When a multi-step workflow fails, registered rollback handlers execute in reverse step-start order, each running through Workflows' durable step machinery with retries, timeouts, and lifecycle events. The post explains the API design decisions (fluent vs. builder vs. options object), how rollback handlers are stored as callable stubs, how replay rebuilds handlers after engine restarts, and the key behavioral rules around ordering and eligibility for failed steps.

3 Agents. 3 LLMs. 1 Aging GPU: Engineering Parallel Inference on Bare Metal

Running three different LLMs simultaneously on a single 8GB GPU fails because llama.cpp pre-allocates the full KV cache upfront, causing OOM errors for the second and third processes. The solution is a C++ daemon called lmxd that implements Connection Admission Control (borrowed from 5G/telecom) as a VRAM ledger: it tracks allocated bytes, enforces a 90% cap, and refuses new agent registrations before any GPU allocation is attempted. The daemon also handles KV-cache swapping to host RAM between agent switches, enabling multiple agents to share one GPU context slot. Additionally, a layer streaming technique using two CUDA streams overlaps compute and weight transfer, achieving ~22-32% wall-clock savings on a GTX 1080. The repo ships the admission control daemon and the streaming primitive as separate, composable components.

Databricks’ former AI chief thinks he can cut AI’s power bill by 1,000x

Unconventional AI, led by former Databricks AI chief Naveen Rao, has released Un0, an image-generation model built on a software simulation of a novel oscillator-based computing architecture. The company claims this architecture could reduce AI inference power consumption by up to 1,000x compared to conventional chips. Un0 performs comparably to state-of-the-art diffusion models like Stable Diffusion, serving as a proof-of-concept for the new architecture. The company plans to release actual chip schematics soon and eventually build a full inference stack, positioning itself as a compute provider running at a fraction of current energy costs.

TokenSpeed-Kernel: Portable APIs and High-Performance Kernels for Multi-Silicon LLM Inference – PyTorch

TokenSpeed-kernel is an open-source, standalone subsystem that provides a clean layered API and registry system for LLM inference kernels across multiple hardware backends. It decouples the high-level runtime from hardware-specific kernel implementations using a decorator-based registration system where kernels declare their platform capabilities, tensor format signatures, and priorities. The selector then dispatches to the best available implementation at runtime. Using GPT-OSS 120B on AMD MI355X (CDNA4) as a validation target, the post demonstrates how Gluon-backed attention and MoE kernels achieve 1.6–3.6x end-to-end throughput improvements over portable Triton baselines, while NVIDIA paths (via FlashInfer/TensorRT-LLM wrappers) use the same public APIs. The AMD-specific kernels are published as a standalone pip package (tokenspeed-kernel-amd) reusable by other inference engines like vLLM.

Apache Flink 2.3.0 Release Announcement

Apache Flink 2.3.0 is now available, implementing 15 FLIPs with major improvements across SQL, connectors, and runtime. New SQL operators FROM_CHANGELOG and TO_CHANGELOG bridge append-only and dynamic changelog tables. Materialized tables gain DDL parity with regular tables and fine-grained refresh control via a new START_MODE clause. The SinkUpsertMaterializer is reworked with an explicit ON CONFLICT clause and watermark-based compaction to reduce state size. A new native S3 filesystem plugin built on AWS SDK v2 replaces Hadoop/Presto-based connectors with non-blocking I/O and zero Hadoop dependencies. Runtime improvements include adaptive partition selection for backpressure handling, watermark alignment redesign for faster backlog processing, checkpointing during recovery from unaligned checkpoints, and application-level lifecycle management with a new Web UI Applications tab.