Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40%

The team optimized RAG at scale by replacing standard semantic search with a measured, tunable retrieval pipeline, achieving 95% recall@10 and cutting latency by 40%. They addressed production failures where fixed 512-token chunks disrupted legal contracts and API docs. By rebuilding the retrieval layer from first principles, they overcame latency issues caused by embedding, vector search, and LLM processing, moving beyond basic text-embedding-3-small implementations to handle complex conversational contexts and diverse document types.