Implementation of RAG-Based LLM Chatbot for Hotel Services: A Case Study

Abstract

The need for up-to-date and dynamic information services in hotel operations encourages the use of automated interactive systems to support guest service activities. One widely used approach to provide such services is the implementation of chatbots, which can be developed using either rule-based models or generative models. Generative chatbot models are capable of producing more flexible and natural responses. However, their development requires the use of Large Language Models (LLMs) to enable deeper contextual understanding. Despite their advantages, standard LLM-based chatbots still face significant limitations, particularly in generating specific and reliable responses, and may produce incorrect or hallucinated information when relevant context is unavailable. To address this technical challenge, this study proposes the implementation of the Retrieval Augmented Generation (RAG) approach in the development of an LLM-based hotel chatbot, with a case study conducted at Hotel Forriz Yogyakarta. The proposed system adopts a RAG architecture consisting of three main stages: indexing, retrieval, and generation, allowing the chatbot to retrieve relevant external hotel documents as contextual input before producing responses. System performance is evaluated using the RAGAS framework on 15 test cases comprising user queries, generated answers, ground truth references, and retrieved contexts. The evaluation results show that the RAG-based chatbot achieves strong performance, with a Context Precision score of 0.889, Context Recall of 1.0, Faithfulness of 0.882, and Answer Relevancy of 0.851. These findings indicate that the RAG approach effectively improves response relevance and reduces hallucination in hotel information services.