Introducing Meerkat: an experiment in global consensus

Kafka's log compaction corrupts data. Here's how we fixed it

Apache Kafka có lỗ hổng trong cơ chế log compaction khiến dữ liệu bị hỏng do xung đột giữa compaction và replication, gây ra bốn vấn đề: dữ liệu đã xóa tái xuất hiện, giao dịch bị hủy hiện dưới dạng đã commit, dữ liệu đã commit bị ẩn, và consumers read_committed bị đóng băng partition. Redpanda Streaming khắc phục bằng giao thức compaction phối hợp, sử dụng các cặp offset (MCCO/MTRO, MXFO/MXRO) để đảm bảo tombstones và transaction markers không bị xóa trước khi tất cả replicas xử lý xong. Lỗi này có thể tái hiện trên Kafka phiên bản 3.9 đến 4.2 bằng Docker Compose.

Lập trình viên cần đọc bài này để hiểu cách giải quyết vấn đề lỗi race condition trong log compaction của Kafka, giúp tránh mất dữ liệu và bảo đảm tính nhất quán khi xử lý các trường hợp đồng bộ hóa dữ liệu trên nhiều broker.

How to use traces to avoid breaking changes

Tracing giúp theo dõi luồng dữ liệu trong hệ thống phân tán thông qua các thành phần như tracer, span, hay context propagation, từ đó ngăn ngừa những thay đổi gây lỗi. Các thư viện phổ biến như OpenTracing, OpenTelemetry, Zipkin, Jaeger hỗ trợ triển khai tracing, trong khi Digma cung cấp công cụ quan sát để phản hồi kịp thời trong quá trình phát triển.

Lập trình viên nên đọc bài này để hiểu cách sử dụng tracing để phát hiện và tránh giải phóng (breaking changes) khi cập nhật hệ thống phân tán một cách an toàn.

Production-grade ingestion with orchestration

Part 2 of a series on building Atlassian Teamwork Graph Connectors, focusing on orchestration — the layer that makes connectors production-ready. Covers declaring a task runner in the Forge manifest, scheduling recurring root tasks idempotently, implementing a task runner that dispatches by task type, fanning out hierarchical data sources into child tasks, classifying failure reasons (UNAUTHORIZED, RATE_LIMITED, RETRYABLE_ERROR, etc.) so the platform can respond appropriately, and managing cross-execution state with persistentTaskMetadata and KVS storage. Includes a full end-to-end lifecycle walkthrough and best practices for scale.

BSidesSF 2026 – Orchestrating Resilience: Composing A New Score For Netflix Service Reliability

A BSidesSF 2026 conference talk by Sandhya Narayan and Prachi Jain covering Netflix's approach to service reliability and resilience orchestration. The content is a video presentation published on the BSidesSF YouTube channel, with minimal descriptive text available beyond the title and presenter names.

Deploying distributed AI inference: Blueprints & troubleshooting

A practical guide to deploying distributed AI inference using vLLM and llm-d across six traffic-shaped blueprints: high-concurrency chat, long-context RAG, high-throughput batch, distributed AI-grid (Model-as-a-Service), hybrid sovereign-to-cloud-burst, and edge inference on workstation GPUs. Each blueprint covers workload signature, topology, key mechanisms (prefill/decode disaggregation, KV-cache tiering, speculative decoding, model cascading), and cost shape. The post also provides inference troubleshooting recipes for TTFT/TPOT regressions using vLLM Prometheus metrics and NVIDIA Nsight tools, and closes with a four-step scaling roadmap from a single vLLM instance to a full distributed AI grid on Red Hat OpenShift AI.

Netflix Cuts Cassandra Read Latency from Seconds to Milliseconds with Dynamic Partition Splitting

Netflix engineers built a dynamic partition-splitting mechanism for Apache Cassandra to tackle wide partitions in time-series workloads. The system automatically detects oversized partitions exceeding predefined thresholds and asynchronously splits them into smaller child partitions, maintaining a metadata layer to track parent-child relationships and route reads transparently. Results from multiple child partitions are merged before returning to clients, requiring no application changes or downtime. After deployment on Netflix's TimeSeries Abstraction platform, average read latency dropped from seconds to low double-digit milliseconds, tail latencies fell to under 200ms, and CPU utilization and read timeouts decreased significantly. The initial implementation focused on immutable partitions for safety, with phased rollouts and validation pipelines comparing old and new read paths. Future work includes support for mutable partitions and reprocessing of failed splits.

How to Implement the Outbox Pattern in Distributed Systems

The Outbox Pattern solves a common distributed systems problem: ensuring database updates and event publishing remain consistent without distributed transactions. Instead of publishing events directly to a message broker (which can fail independently), the pattern stores events in a dedicated Outbox table within the same local database transaction as the business data. A separate background publisher then reads and forwards pending events to the message broker. This guarantees at-least-once delivery, and consumers should be designed to be idempotent to handle occasional duplicates. The post covers the full workflow, failure handling, real-world use cases (e-commerce, payments, inventory), benefits, trade-offs, and best practices like retry logic, correlation IDs, and Outbox table cleanup.