Enhancing Retrieval-Augmented Generation (RAG) Systems for the CRAG Meta KDD Competition

Optimized RAG Pipeline: Developed an enhanced Retrieval-Augmented Generation (RAG) system for the CRAG Meta KDD competition, focusing on retrieval precision, prompt tuning, and reducing hallucinations.

Advanced Retrieval & Reranking: Used recursive chunking, dense retrieval (multi-qa-distilbert-cos-v1), and reranking (BAAI/bge-reranker-v2-m3) for better document selection.

Time-Aware Query Processing: Classified queries as static, slow-changing, fast-changing, and real-time, enabling contextualized retrieval for different domains.

Improved Generative Model: Integrated GPT-4o-mini with Chain-of-Thought prompting, improving accuracy and contextual alignment while mitigating errors.

Key Results

• Retrieval Efficiency: Recursive chunking + reranking improved accuracy by ~7%, but increased hallucinations slightly.
• Domain-Specific Performance: Best accuracy in Open-ended (42.16%) but higher hallucinations in Finance (21.47%) and Sports (24.90%).
• Query Type Performance: Static queries (29.59%) outperformed fast-changing (13.56%) and real-time (9.17%) due to retrieval challenges.
• Model Limitations: Numerical approximation errors, false premise detection issues, and multi-step reasoning gaps remain key challenges.

Follow "this repository" for more details.

Comparison of Improvements RAG with Vanilla baseline and RAG baseline on LLama3