Evaluating model performance for sentiment analysis on game reviews

Abstract

This research aimed to examine game reviews and compare the characteristics of classification models using Thai language data, particularly in contexts with limited resources. The experiment evaluates the performance of machine learning models, including Naïve Bayes, support vector machines, and 1D-CNN, and compares them with transformer-based models, namely BERT Multilingual and WangchanBERTa, in analyzing game-related reviews using a dataset primarily consisting of Thai-language comments. This research employs Bag of words, TF-IDF, and word2vec techniques to generate text transformations for machine learning models and CNN model. In contrast, transformer models employ pre-trained embeddings. The experimental results indicate that WangchanBERTa achieves the highest overall performance, with an accuracy of 82.16%, a precision of 87.06%, a recall of 86.18%, and an F1-score of 86.62%. Meanwhile, the support vector machine method employing the Bag of Words technique demonstrates the best performance among the machine learning models, with an accuracy of 81.26%. The experimental findings within the text transformation methods indicate that feature extraction is a critical factor in optimizing the performance of machine learning models. The study reveals that feature extraction methods focusing exclusively on individual words, such as Bag-of-Words and TF-IDF, demonstrate greater effectiveness in feature extraction compared to context-aware approaches such as word2vec. An analysis of the experimental results reveals that the accuracy of the best-performing models in both categories differs by approximately 1%. The machine learning models achieve accuracy levels comparable to those of the transformer models while requiring fewer resources, making them well-suited for resource-constrained environments.