Speculative RAG

Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting is a method described by Wang et al. (arXiv:2407.08223) that splits the work between a small specialist model and a larger generalist one. Retrieval runs first; the retrieved documents are then split into subsets, and the smaller drafter model generates several answer drafts in parallel, one per subset, each with its own rationale. Because each draft sees only one subset of the documents, the drafter works with a short input, several perspectives on the evidence are covered at once, and position bias over a long context is reduced. The larger generalist model then acts as the verifier: in a single pass it scores each draft together with its rationale and picks the highest-scoring one, without going back to the retriever. The latency gain comes from the long context being read by the small model while the large model only reads short drafts; on the PubHealth benchmark, with a Mistral-7B drafter and a Mixtral-8x7B verifier, the authors report roughly 51% lower latency together with higher accuracy than their baseline. Engineers tune draft length, confidence thresholds, and verifier strictness to balance speed and accuracy. The method shines in real-time chatbots, voice agents, and mobile apps where every 100 ms matters, yet factual grounding cannot be sacrificed. Speculative RAG borrows its name and its draft-then-verify structure from speculative decoding, applied to whole candidate answers rather than to individual tokens.

Related terms

• AI what is RAG
• C-RAG: Certified Generation Risks for Retrieval-Augmented Language Models
• Gen AI RAG meaning
• Graph RAG
• Hire Retrieval-Augmented Generation (RAG) developers

Related services: RAG development service, Agentic AI consulting.

← Back to the AI Glossary