Development of an Academic Services Chatbot Based on Retrieval-Augmented Generation (RAG)

Abstrak

Higher education institutions struggle to provide accurate and accessible academic information. Traditional chatbots are often limited in capability, while standard Large Language Models (LLMs) pose a significant risk of factual "hallucinations," rendering them unsuitable for official university use where trustworthiness is paramount. This study aims to increase the accessibility and effectiveness of academic services by developing a trustworthy chatbot. The primary objective is to implement the Retrieval-Augmented Generation framework to create a reliable AI assistant that is factually grounded in a verified, domain-specific knowledge base. A knowledge base was constructed from official FPMIPA UPI documents and structured using hierarchical chunking. The system employs a multi-stage RAG pipeline featuring query contextualization and reranking before generation with Gemini 2.5 Pro. Performance was evaluated using metrics from the RAGAS framework on a 100-question dataset categorized into factual, reasoning, and out-of-context queries. The evaluation revealed strong performance on factual queries, achieving a Faithfulness score of 0.9100. A significant performance decrease was observed for reasoning tasks, with Context Recall dropping to 0.5926. Crucially, the system successfully handled 81.25% of out-of-context questions by correctly refusing to answer, thereby effectively preventing hallucination. The RAG framework provides a viable and empirically-validated blueprint for creating a trustworthy conversational AI for higher education. The model proves to be an effective tool for factual information delivery and has strong potential to modernize how student support and academic services are delivered.