Scaling StreamHub: Transitioning from Kinesis to Kafka for 145 Billion Daily Events

Atlassian's engineering team shares how they scaled StreamHub from 22 billion to 150 billion daily events by migrating from Amazon Kinesis to Apache Kafka via AWS MSK. The post details why Kinesis hit its limits (shard costs, retention constraints, consumer throughput), how Kafka Tiered Storage with S3 cut infrastructure costs, and the six critical operational challenges encountered at scale — including broker network saturation, S3 request rate storms during retention changes, managed control plane unavailability during AZ failures, and slow scaling operations during incidents. Solutions include intentional over-provisioning, ingress rate limiting, Kafka client quotas, cluster sharding with failover runbooks, and companion regions for compliant regional failover.

Source: https://www.atlassian.com/blog/how-we-build/scaling-streamhub-transitioning-from-kinesis-to-kafka-for-145-billion-daily-events. 8sync News only summarizes and links out; content copyright belongs to the authors and original sources.