Evaluating the Usability of AI-Powered Chatbots in Informatics Learning at a Vocational High School Using the USE Framework

Abstract

In the Society 5.0 era, vocational education demands continuous, personalized technical support to facilitate independent and self-regulated learning. However, high student-to-instructor ratios often impede the delivery of immediate scaffolding during practical sessions. To address this instructional gap, this study aimed to develop and systematically evaluate "Bot-Kom", an Artificial Intelligence-powered educational chatbot specifically designed for the 10th-grade "Computer Systems" module. Employing a Research and Development (R&D) methodology based on the ADDIE (Analysis, Design, Development, Implementation, and Evaluation) framework, the system was engineered using a modern web stack (React and Node.js) and a hybrid AI architecture. This architecture integrated Google’s Dialogflow for precise intent classification and OpenAI GPT for dynamic, natural language generation. Following expert validation, a field evaluation was conducted involving 59 vocational students majoring in Computer and Network Engineering. A mixed-methods data collection approach was utilized, employing the USE (Usefulness, Satisfaction, and Ease of use) Questionnaire for quantitative metrics and open-ended questions for qualitative diagnostic insights. The results indicate that the deployed application effectively minimizes cognitive load through a highly user-centric interface and accurate Natural Language Understanding (NLU). The usability evaluation confirms that the chatbot meets high standards across all four critical dimensions: Usefulness, Ease of Use, Ease of Learning, and Satisfaction. Ultimately, this study demonstrates that intelligently architected conversational agents can serve as highly accessible, engaging, and pedagogically robust tools, successfully bridging the gap between traditional classroom instruction and autonomous vocational practice.