Loading feed…
AWS Database Blog0 Hot0 comments18 min read
Scale pgvector with binary quantization on Amazon Aurora PostgreSQL
Binary quantization with reranking (HNSW+BQ) lets pgvector scale to hundreds of millions or billions of vectors on Amazon Aurora PostgreSQL by compressing float32 vectors to single bits, shrinking a 100M-vector index from 367 GB to ~38 GB so it fits in buffer cache. Benchmarks across LAION 100M, OpenAI 5M, and Cohere 10M datasets show HNSW+BQ often matches or beats disk-based ANN with much faster index builds (1.1 hrs vs 31.5 hrs at 100M scale), though effectiveness is distribution-dependent and fails badly on clustered embeddings like Cohere's. Sizing tables, recall validation steps, and NVMe-tiered scaling guidance (up to ~20B vectors theoretically) are provided, along with guidance on when to prefer full-precision HNSW or halfvec instead.
Source: https://aws.amazon.com/blogs/database/scale-pgvector-with-binary-quantization-on-amazon-aurora-postgresql. 8 Sync News only summarizes and links out; content copyright belongs to the authors and original sources.
