How Intuit and AWS systematically improved resiliency on ElastiCache using AWS Fault Injection Service

Intuit and AWS partnered to systematically test and improve resiliency of Amazon ElastiCache clusters underpinning services like TurboTax and QuickBooks. Investigation found client-side misconfigurations (outdated Lettuce versions lacking topology refresh, insufficient retry tuning, and limited observability) rather than ElastiCache itself caused repeated disruptions. They built a Resiliency Checklist covering client configuration, cluster architecture, observability, and operational readiness, then validated it with a production-scale GameDay using AWS FIS to simulate an Availability Zone power interruption. Results: mean time to recovery dropped from tens of minutes to under 2 minutes, customer impact fell to effectively zero, time to detect improved by roughly 75 percent, and repeat incidents from the same root cause dropped by 90 percent or more. Key fixes included enabling Lettuce topology refresh (version 6.2.2+), exponential backoff with jitter, static stability capacity planning, and CloudWatch alarms on bandwidth, CPU, and replication lag. The checklist is now a company-wide standard referenced in runbooks and architecture templates.

Source: https://aws.amazon.com/blogs/database/how-intuit-and-aws-systematically-improved-resiliency-on-elasticache-using-aws-fault-injection-service. 8 Sync News only summarizes and links out; content copyright belongs to the authors and original sources.