Apache Flink 2.3.0 Release Announcement

How to use traces to avoid breaking changes

Việc sử dụng tracing giúp phát hiện sớm các vấn đề tiềm ẩn khi thay đổi hệ thống bằng cách theo dõi luồng dữ liệu và sự kiện trong môi trường phân tán. Các thư viện phổ biến như OpenTracing, OpenTelemetry, Zipkin và Jaeger hỗ trợ giám sát, trong khi Digma cung cấp phản hồi tức thì trong quá trình phát triển.

Lập trình viên nên đọc bài này để hiểu cách sử dụng tracing để phát hiện và tránh các break changes trong hệ thống phân tán, từ đó giảm thiểu rủi ro khi cập nhật hoặc mở rộng ứng dụng.

Client-Side Load Balancing at a Million Requests Per Second

Zalando's engineering team built an in-process client-side load balancer (CSLB) to handle over a million requests per second of internal fan-out traffic for their Product Read API, replacing shared Skipper ingress hops. The implementation replicates Skipper's xxHash64 consistent-hash ring for cache locality, uses a Kubernetes watch-based informer for pod discovery, and adds N-ring fade-in to prevent cold-cache spikes on scale-up. A key innovation is occupancy-based bounded load using Little's Law (seconds of work per second) rather than in-flight counts or throughput, combined with a latency multiplier borrowed from Finagle. Results include eliminating Skipper's fleet from 50+ pods to 8, reducing their own pod fleet by 25%, and saving over $1,000/day. AZ-aware routing was prototyped but paused due to edge cases around bounded-load threshold miscalculation during dual fade-in. The post also covers pipeline improvements, retry hardening, FIFO buffering, and how detailed logging revealed mysterious node-level network freezes that had previously been invisible.

Event Sourcing: Aggregates, Dynamic Consistency Boundaries, or what?

Part eleven of an event sourcing series explores how to handle consistency boundaries without relying on DDD aggregates or Dynamic Consistency Boundaries (DCBs). The author argues that the best approach depends on the actual problems at hand. Two alternatives are discussed: replacing concurrent designs with non-concurrent ones (e.g., a draft-registration phase processed by a single-threaded algorithm), and using Azure Service Bus sessions to serialize workday validation, eliminating race conditions within a consistency boundary. The post emphasizes solving real problems holistically rather than applying patterns preemptively, and shows how task-based UIs and small data models reduce the likelihood of concurrency conflicts in the first place.

Confluent: Data in motion

Batch customer data platforms can't capture user intent as it forms — by the time a nightly sync completes, the intent moment is gone. A streaming-native architecture built on Apache Kafka and Apache Flink handles the full spectrum of personalization latency windows, from sub-100ms real-time bidding to multi-day email campaigns, using the same four-job pipeline: connect, stream, process, and govern. An AI-native layer (Confluent Intelligence) sits on top, enabling streaming agents with MCP tool-calling, a real-time context engine for LLMs, and built-in ML functions (ML_PREDICT, AI_COMPLETE) for embedding, ranking, and generative copy — all running as Flink jobs with exactly-once semantics and full lineage. The guide covers three production patterns (retail product recommendations, media feed personalization, cross-channel cart abandonment orchestration), a five-capability vendor evaluation framework, and a three-phase rollout roadmap from streaming backbone to autonomous agentic personalization.

How to Eliminate Training-Serving Skew in MLOps (2026)

Training-serving skew — the divergence between features used during model training and those seen at inference time — silently degrades ML accuracy and doubles infrastructure costs. The solution is a unified kappa architecture: compute features once in Apache Flink, dual-write to an offline store (Apache Iceberg or Delta Lake) for training and an online key-value cache for serving. DoorDash measured a 35.7% feature-value mismatch in their dual-pipeline setup; Netflix replaced a $93M/year dual-pipeline backfill with a $2M/year kappa replay. The reference architecture covers Kafka ingestion via Confluent's Kora engine, serverless Flink with event-time watermarks and exactly-once semantics, Tableflow for automated Iceberg/Delta materialization, and Stream Governance for schema enforcement and lineage. A tooling comparison covers Databricks, SageMaker+Kinesis, Tecton, Feast, and Confluent, with a decision framework based on latency requirements, existing stack investment, and pipeline fragmentation. The post is authored by a Confluent employee and promotes the Confluent Data Streaming Platform throughout.

Optimizing distributed AI inference: Advanced deployment patterns

Bài viết hướng dẫn kỹ thuật sâu về ba phương pháp tối ưu hóa inference AI phân tán ở quy mô lớn: tách rời prefill/decode (P/D), chiến lược KV cache, và giải mã dự đoán (speculative decoding). P/D disaggregation đề xuất tỷ lệ worker 1:3 đến 1:5, sử dụng KV-transfer connector (NixlConnector, LMCacheConnector, MooncakeConnector) và routing thông minh (llm-d) giúp cải thiện TTFT lên tới 57 lần. KV cache được phân cấp (HBM/DRAM/NVMe), tối ưu chia sẻ tiền tố (prefix sharing) và tái sử dụng (reuse), cân nhắc lượng tử hóa FP8/FP4, cùng so sánh kiến trúc PagedAttention và RadixAttention. Phần speculative decoding so sánh EAGLE 3.1, self-speculative, Medusa heads, MTP, đồng thời cảnh báo rằng chế độ giải mã hạn chế (JSON mode, tool calls) có thể làm giảm tỷ lệ chấp nhận.

Lập trình viên chuyên phát triển hệ thống AI quy mô lớn cần đọc để tối ưu hóa hiệu suất và chi phí của các ứng dụng phân tán, từ cách phân tán tiền xử lý/giải mã đến lựa chọn cache KV hiệu quả và chiến lược dự đoán để giảm thời gian phản hồi mà không ảnh hưởng đến độ chính xác.

Reliability fail: No automated zone failover for Coinbase’s global trading service

On May 7, 2026, Coinbase suffered a nearly 10-hour global trading outage triggered by a regional AWS disruption. The root cause was that Coinbase's matching engine ran in a single AWS Cluster Placement Group (a single availability zone) with no automated cross-zone failover. Recovery required an emergency code change and manual quorum restoration. The author criticizes this as amateurish for a $40B company processing $5.2 trillion annually, drawing an unfavorable comparison to Uber's multi-region failover drills from a decade earlier. The piece also notes that Coinbase had previously suffered a similar AWS-related outage in October 2025 and pledged to review its regional deployment strategy — a review that apparently failed to address this single-zone dependency.