Kafka's log compaction corrupts data. Here's how we fixed it

How to use traces to avoid breaking changes

Việc sử dụng tracing giúp phát hiện sớm các vấn đề tiềm ẩn khi thay đổi hệ thống bằng cách theo dõi luồng dữ liệu và sự kiện trong môi trường phân tán. Các thư viện phổ biến như OpenTracing, OpenTelemetry, Zipkin và Jaeger hỗ trợ giám sát, trong khi Digma cung cấp phản hồi tức thì trong quá trình phát triển.

Lập trình viên nên đọc bài này để hiểu cách sử dụng tracing để phát hiện và tránh các break changes trong hệ thống phân tán, từ đó giảm thiểu rủi ro khi cập nhật hoặc mở rộng ứng dụng.

How we built saga rollbacks for Cloudflare Workflows

Cloudflare Workflows now supports saga-style rollbacks, letting developers attach compensation logic directly to each step.do() call. When a multi-step workflow fails, registered rollback handlers execute in reverse step-start order, each running through Workflows' durable step machinery with retries, timeouts, and lifecycle events. The post explains the API design decisions (fluent vs. builder vs. options object), how rollback handlers are stored as callable stubs, how replay rebuilds handlers after engine restarts, and the key behavioral rules around ordering and eligibility for failed steps.

Client-Side Load Balancing at a Million Requests Per Second

Zalando's engineering team built an in-process client-side load balancer (CSLB) to handle over a million requests per second of internal fan-out traffic for their Product Read API, replacing shared Skipper ingress hops. The implementation replicates Skipper's xxHash64 consistent-hash ring for cache locality, uses a Kubernetes watch-based informer for pod discovery, and adds N-ring fade-in to prevent cold-cache spikes on scale-up. A key innovation is occupancy-based bounded load using Little's Law (seconds of work per second) rather than in-flight counts or throughput, combined with a latency multiplier borrowed from Finagle. Results include eliminating Skipper's fleet from 50+ pods to 8, reducing their own pod fleet by 25%, and saving over $1,000/day. AZ-aware routing was prototyped but paused due to edge cases around bounded-load threshold miscalculation during dual fade-in. The post also covers pipeline improvements, retry hardening, FIFO buffering, and how detailed logging revealed mysterious node-level network freezes that had previously been invisible.

Event Sourcing: Aggregates, Dynamic Consistency Boundaries, or what?

Part eleven of an event sourcing series explores how to handle consistency boundaries without relying on DDD aggregates or Dynamic Consistency Boundaries (DCBs). The author argues that the best approach depends on the actual problems at hand. Two alternatives are discussed: replacing concurrent designs with non-concurrent ones (e.g., a draft-registration phase processed by a single-threaded algorithm), and using Azure Service Bus sessions to serialize workday validation, eliminating race conditions within a consistency boundary. The post emphasizes solving real problems holistically rather than applying patterns preemptively, and shows how task-based UIs and small data models reduce the likelihood of concurrency conflicts in the first place.

Confluent: Data in motion

Batch customer data platforms can't capture user intent as it forms — by the time a nightly sync completes, the intent moment is gone. A streaming-native architecture built on Apache Kafka and Apache Flink handles the full spectrum of personalization latency windows, from sub-100ms real-time bidding to multi-day email campaigns, using the same four-job pipeline: connect, stream, process, and govern. An AI-native layer (Confluent Intelligence) sits on top, enabling streaming agents with MCP tool-calling, a real-time context engine for LLMs, and built-in ML functions (ML_PREDICT, AI_COMPLETE) for embedding, ranking, and generative copy — all running as Flink jobs with exactly-once semantics and full lineage. The guide covers three production patterns (retail product recommendations, media feed personalization, cross-channel cart abandonment orchestration), a five-capability vendor evaluation framework, and a three-phase rollout roadmap from streaming backbone to autonomous agentic personalization.

How to Eliminate Training-Serving Skew in MLOps (2026)

Training-serving skew — the divergence between features used during model training and those seen at inference time — silently degrades ML accuracy and doubles infrastructure costs. The solution is a unified kappa architecture: compute features once in Apache Flink, dual-write to an offline store (Apache Iceberg or Delta Lake) for training and an online key-value cache for serving. DoorDash measured a 35.7% feature-value mismatch in their dual-pipeline setup; Netflix replaced a $93M/year dual-pipeline backfill with a $2M/year kappa replay. The reference architecture covers Kafka ingestion via Confluent's Kora engine, serverless Flink with event-time watermarks and exactly-once semantics, Tableflow for automated Iceberg/Delta materialization, and Stream Governance for schema enforcement and lineage. A tooling comparison covers Databricks, SageMaker+Kinesis, Tecton, Feast, and Confluent, with a decision framework based on latency requirements, existing stack investment, and pipeline fragmentation. The post is authored by a Confluent employee and promotes the Confluent Data Streaming Platform throughout.

What is the dltHub Context Layer?

dltHub introduces a 'context layer' that stores and carries pipeline metadata — schemas, connectors, deployment configs, logs — across the entire data stack so AI agents can build, deploy, and maintain pipelines with minimal human intervention. A single command scaffolds a workspace and runs an example pipeline end to end. The system organizes work into phases (extract, model, deploy, run, maintain) with guided skill sequences and guardrails. When a source breaks months later, the agent can diagnose and fix it in minutes because all context is already available. Users stay at a high-level intent layer and only intervene for judgment calls, not errand-running.