|
EFFICACY ASSESSMENT OF A RAG-BASED CHATBOT FOR EDUCATIONAL COMPUTER PROGRAMMING |
|---|---|
| รหัสดีโอไอ | |
| Creator | Waris Rattananimit |
| Title | EFFICACY ASSESSMENT OF A RAG-BASED CHATBOT FOR EDUCATIONAL COMPUTER PROGRAMMING |
| Contributor | Jatuporn Sakchaikul |
| Publisher | มหาวิทยาลัยเซาธ์อีสท์บางกอก |
| Publication Year | 2569 |
| Journal Title | วารสารวิทยาศาสตร์และเทคโนโลยี มหาวิทยาลัยเซาธ์อีสท์บางกอก |
| Journal Vol. | 6 |
| Journal No. | 1 |
| Page no. | e265155 |
| Keyword | Computer Programming, Artificial Intelligent, Large Language Model, Retrieval-Augmented Generation, Educational Chatbot |
| URL Website | https://ph02.tci-thaijo.org/index.php/JSCI/article/view/265155 |
| Website title | วารสารวิทยาศาสตร์และเทคโนโลยี มหาวิทยาลัยเซาธ์อีสท์บางกอก |
| ISSN | 2773-9120 |
| Abstract | The purposes of the research were to 1) develop an educational Python chatbot utilizing the open-source Large Language Model, Gemma2 9B, combined with Retrieval-Augmented Generation (RAG) to provide personalized, non-intimidating support for novice programmers facing complex abstract concepts, 2) implement a secure, locally-hosted AI system using Ollama and Docker, managed via OpenWebUI, to ensure a context-aware knowledge base specifically tailored for Python programming instruction, and 3) evaluate the performance and efficacy of the chatbot through a multi-dimensional assessment focusing on semantic consistency, response accuracy via BERT Score, and linguistic fluency via BLEU Score.The research findings showed that the RAG-based chatbot significantly outperformed the baseline (No-RAG) model across all evaluation dimensions. It achieved a high Consistency score of 0.96 compared to the baseline's 0.82, indicating highly stable and predictable responses. Furthermore, the system demonstrated robust semantic understanding with an impressive BERT Score (Precision: 0.90, Recall: 0.88, F1-score: 0.89), successfully capturing the relevance of educational content compared to the baseline's F1-score of 0.76. However, while the BLEU Score of 0.29 was an improvement over the baseline's 0.21, it highlighted limitations in generating nuanced, human-like linguistic responses. Ultimately, integrating external knowledge via RAG provides a reliable, accurate, and powerful solution for decentralized educational tools, though further optimization in prompt engineering is recommended to enhance natural language flow. The evaluation was conducted using a benchmark dataset consisting of 50 Python programming questions; therefore, the findings should be interpreted within the scope of this dataset. |
