Deploying distributed AI inference: Blueprints & troubleshooting

How to use traces to avoid breaking changes

Theo dõi (tracing) giúp phát hiện sớm hành vi và hiệu suất hệ thống, từ đó ngăn ngừa thay đổi gây lỗi bằng cách ghi lại luồng dữ liệu trong hệ thống phân tán. Các công cụ như OpenTracing, OpenTelemetry, Zipkin, Jaeger hỗ trợ theo dõi, trong khi Digma cung cấp khả năng quan sát và phản hồi trong quá trình phát triển.

Lập trình viên nên đọc bài này để hiểu cách sử dụng tracing để phát hiện và tránh giải phóng không mong muốn trong các ứng dụng phân tán, từ đó bảo vệ codebase và cải thiện chất lượng hệ thống.

TokenSpeed-Kernel: Portable APIs and High-Performance Kernels for Multi-Silicon LLM Inference – PyTorch

TokenSpeed-kernel is an open-source, standalone subsystem that provides a clean layered API and registry system for LLM inference kernels across multiple hardware backends. It decouples the high-level runtime from hardware-specific kernel implementations using a decorator-based registration system where kernels declare their platform capabilities, tensor format signatures, and priorities. The selector then dispatches to the best available implementation at runtime. Using GPT-OSS 120B on AMD MI355X (CDNA4) as a validation target, the post demonstrates how Gluon-backed attention and MoE kernels achieve 1.6–3.6x end-to-end throughput improvements over portable Triton baselines, while NVIDIA paths (via FlashInfer/TensorRT-LLM wrappers) use the same public APIs. The AMD-specific kernels are published as a standalone pip package (tokenspeed-kernel-amd) reusable by other inference engines like vLLM.

Apache Flink 2.3.0 Release Announcement

Apache Flink 2.3.0 is now available, implementing 15 FLIPs with major improvements across SQL, connectors, and runtime. New SQL operators FROM_CHANGELOG and TO_CHANGELOG bridge append-only and dynamic changelog tables. Materialized tables gain DDL parity with regular tables and fine-grained refresh control via a new START_MODE clause. The SinkUpsertMaterializer is reworked with an explicit ON CONFLICT clause and watermark-based compaction to reduce state size. A new native S3 filesystem plugin built on AWS SDK v2 replaces Hadoop/Presto-based connectors with non-blocking I/O and zero Hadoop dependencies. Runtime improvements include adaptive partition selection for backpressure handling, watermark alignment redesign for faster backlog processing, checkpointing during recovery from unaligned checkpoints, and application-level lifecycle management with a new Web UI Applications tab.

The Hot Path Belongs to GBDTs, Agents Own the Cold Path: A Payment-Fraud Benchmark

A reproducible benchmark comparing gradient-boosted decision trees (GBDTs) vs. LLM-based scoring for payment fraud detection across three dimensions: latency, cost, and determinism. On a single CPU core, GBDTs hit p99 latency of 0.15ms vs. ~1,200ms for LLMs — well outside the 100ms ISO 8583 authorization budget. Cost-wise, GBDTs run ~$54/hour at 50K TPS vs. $16,200–$351,000 for LLM tiers. Determinism is the most critical issue for regulated environments: GBDTs return identical scores on identical inputs while LLMs produce hundreds of distinct outputs even at temperature=0. The recommended architecture keeps deterministic tree ensembles on the synchronous hot path and deploys LLM agents on the asynchronous cold path for SAR drafting, evidence gathering, and agent-as-a-judge validation before human review. All benchmark code is open-source and reproducible on a laptop.

3 Agents. 3 LLMs. 1 Aging GPU: Engineering Parallel Inference on Bare Metal

Running three different LLMs simultaneously on a single 8GB GPU fails because llama.cpp pre-allocates the full KV cache upfront, causing OOM errors for the second and third processes. The solution is a C++ daemon called lmxd that implements Connection Admission Control (borrowed from 5G/telecom) as a VRAM ledger: it tracks allocated bytes, enforces a 90% cap, and refuses new agent registrations before any GPU allocation is attempted. The daemon also handles KV-cache swapping to host RAM between agent switches, enabling multiple agents to share one GPU context slot. Additionally, a layer streaming technique using two CUDA streams overlaps compute and weight transfer, achieving ~22-32% wall-clock savings on a GTX 1080. The repo ships the admission control daemon and the streaming primitive as separate, composable components.

Coinspaid Dev emerges as a dedicated blockchain infrastructure engineering brand

Coinspaid Dev, the engineering team behind Coinspaid Solutions, has launched as an independent brand after over 11 years of building blockchain infrastructure. With 120+ engineers, the team has experience across 20+ blockchain networks, covering distributed systems, backend architecture, cloud infrastructure, cybersecurity, and reliability engineering. The new brand aims to become a center of competence for blockchain infrastructure engineering and contribute practical expertise to the broader digital asset ecosystem.

How we built saga rollbacks for Cloudflare Workflows

Cloudflare Workflows now supports saga-style rollbacks, letting developers attach compensation logic directly to each step.do() call. When a multi-step workflow fails, registered rollback handlers execute in reverse step-start order, each running through Workflows' durable step machinery with retries, timeouts, and lifecycle events. The post explains the API design decisions (fluent vs. builder vs. options object), how rollback handlers are stored as callable stubs, how replay rebuilds handlers after engine restarts, and the key behavioral rules around ordering and eligibility for failed steps.