Evaluation and comparison of customer service performance generated by generative AI

Abstract

Generative AI has the ability to interact in a conversational manner and understand natural language, where there are a variety of programs available. If Generative AI is high-performance with the ability to answer questions similar to humans, it will impress users and be useful for customer service applications. Therefore, the objective of this research was to evaluate and compare the effectiveness of customer questioning-answering services generated by Generative AI models, namely ChatGPT, Claude, and Gemini. The research procedure involved preparing a dataset of questions and accurate answers, generating responses using Generative AI models (ChatGPT, Claude, and Gemini), and assessing these responses using the ROUGE-L algorithm and the evaluation of experts to compare the efficiency of each model's answers. ChatGPT, Claude, and Gemini were used to answer questions in two formats: the standard prompt format and a roleplay prompt format, with each question answered three times per format. The results of this study were as follows: 1) According to the standard format evaluation using the ROUGE-L algorithm, the answers generated by ChatGPT were most similar to the correct answers, with an average score of 0.26547. In the roleplay format, ChatGPT also produced answers most similar to the correct ones, with an average score of 0.29307; 2) Regarding the expert evaluations of the standard format responses, ChatGPT's answers were rated the closest to the correct ones, with an average score of 4.1556. In the roleplay prompt format, ChatGPT again scored the highest similarity, with an average of 4.2867.