If You Think You Can Do Real-World Text-to-SQL – Communications of the ACM

Current text-to-SQL benchmarks like Spider and Bird-SQL show 80-90%+ accuracy, but they fail to reflect real-world data warehouse conditions. Real warehouses suffer from schema rot (non-intuitive table/column names, overlapping semantics), idiosyncratic domain-specific data, complex multi-join queries, and private data LLMs have never seen. The authors introduce the Beaver benchmark, built from real MIT and other enterprise data warehouses, which exposes these gaps: pure LLM approaches score zero, and even with RAG, prompt engineering, and agentic AI, accuracy only reaches 10-30%. This is 50%+ lower than public benchmarks, representing the gap between 'promising technology' and 'technology that doesn't work.' The authors make Beaver publicly available and point to their Rubicon system as a direction for improvement.

Source: https://cacm.acm.org/blogcacm/if-you-think-you-can-do-real-world-text-to-sql. 8sync News only summarizes and links out; content copyright belongs to the authors and original sources.