Advanced Methods for Improved Retrieval-Augmented Generation (RAG) System Performance in E-Learning

This study evaluates two search methodologies—Hybrid Search and Semantic Search—withinaRetrieval-AugmentedGeneration(RAG)frameworkforanE-Learning use case. The goal was to enhance the accuracy and efficiency of Large Language Models (LLMs), such as GPT-4, through advanced Prompt Engineering Techniques and optimized retrieval processes. Thus, efficient search and chunking methods are essential for improving thequality ofsystem-generatedanswers. UsingtheEvaluation frameworkforRetrieval-AugmentedGeneration(RAG)pipelines, theRagasframework as our testing framework, we measured five key metrics: answer correctness, context recall, context precision, faithfulness, and answer relevancy. The dataset utilized in this study comprises question-answer pairs, with the answers established as ground truth, derived from educational sources such as textbooks, research papers, and lectures with over 215 pages of highly complex theoretical and practical learning material. In order to evaluate the chunking and search methodologies the Ragas testing framework dataset covers 57 questions out of the used educational material related to generative AI concepts and prompt engineering techniques. These source documents were pre-processed into smaller, manageable chunks and indexed using both vector embeddings and keyword-based indexing, aimed at facilitating efficient retrieval and improving response accuracy. The ground truth constituted the benchmark for assessing the performance of the Ragas testing framework. The AI model used for embeddings, OpenAI’s text-embedding-ada-002, generated highdimensional representations to capture deep semantic meanings. The study tested three chunking strategies (Token-Based, Recursive, and BERT-based) and compared the search methods using statistical analyses like ANOVA and paired t-tests. The results show that Hybrid Search consistently outperformed Semantic Search across all metrics. However, the effect size (Cohen’s d 0.11) indicated that the practical difference was negligible. Token-Based Chunking underperformed in Context Recall compared to BERT-based and Recursive Chunking. These findings offer valuable insights for optimizing RAG systems in E-Learning, with future directions focusing on continuously improving chunking techniques and integrating long-context LLMs for enhanced scalability and accuracy.