Automatically Correcting Noisy Labels for Improving Quality of Training Set in Domain-specific Sentiment Classification
Article Sidebar
Main Article Content
Abstract
Classification model performance can be degraded by label noise in the training set. The sentiment classification domain also struggles with this issue, whereby customer reviews can be mislabeled. Some customers give a rating score for a product or service that is inconsistent with the review content. If business owners are only interested in the overall rating picture that includes mislabeling, this can lead to erroneous business decisions. Therefore, this issue became the main challenge of this study. If we assume that customer reviews with noisy labels in the training data are validated and corrected before the learning process, then the training set can generate a predictive model that returns a better result for the sentiment analysis or classification process. Therefore, we proposed a mechanism, called polarity label analyzer, to improve the quality of a training set with noisy labels before the learning process. The proposed polarity label analyzer was used to assign the polarity class of each sentence in a customer review, and then polarity class of that customer review was concluded by voting. In our experiment, datasets were downloaded from TripAdvisor and two linguistic experts helped to assign the correct labels of customer reviews as the ground truth. Sentiment classifiers were developed using the k-NN, Logistic Regression, XGBoost, Linear SVM and CNN algorithms. After comparing the results of the sentiment classifiers without training set improvement and the results with training set improvement, our proposed method improved the average scores of F1 and accuracy by 20.59%.
Keywords: label noise; sentiment classification; polarity label analyzer; k-NN; logistic regression; XGBoost; linear SVM; CNN
*Corresponding author: Tel.: (+66) 43654359 ext. 5365, 5003
E-mail: jantima.p@msu.ac.th
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Copyright Transfer Statement
The copyright of this article is transferred to Current Applied Science and Technology journal with effect if and when the article is accepted for publication. The copyright transfer covers the exclusive right to reproduce and distribute the article, including reprints, translations, photographic reproductions, electronic form (offline, online) or any other reproductions of similar nature.
The author warrants that this contribution is original and that he/she has full power to make this grant. The author signs for and accepts responsibility for releasing this material on behalf of any and all co-authors.
Here is the link for download: Copyright transfer form.pdf
References
Kaushik, R., 2012. Impact of social media on marketing. International Journal of Computational Engineering and Management, 15(2), 91-95.
Appel, G., Grewal, L., Hadi, R. and Stephen, A.T., 2020. The future of social media in marketing. Journal of the Academy of Marketing Science, 48, 79-95.
Chong, A.Y.L., Lacka, E., Li, B. and Chan, H.K., 2018. The role of social media in enhancing guanxi and perceived effectiveness of E-commerce institutional mechanisms in online marketplace. Journal of Information and Management, 55(5), 621-632.
He, W., Wang, F.-K. and Akula, V., 2017. Managing extracted knowledge from big social media data for business decision making. Journal of Knowledge Management, 21(2), 275-294.
Karakaya, F. and Barnes, N.G., 2017. Impact of online reviews of customer care experience on brand or company selection. Journal of Consumer Marketing, 27(5), 447-457.
Medhat, W., Hassan, A. and Korashy, H., 2017. Sentiment analysis algorithms and applications: A survey. Ain Shams Engineering Journal, 5(4), 1093-1113.
Feldman, R., 2013. Techniques and applications for sentiment analysis. Communications of the ACM, 56(4), 82-89.
Mohammad, S.M., 2021. Sentiment Analysis: Automatically Detecting Valence, Emotions, and Other Affectual States from Text. [online] Available at: https://arxiv.org/pdf/2005.11882.pdf.
Mohammad, S.M., 2017. Challenges in sentiment analysis. In: E. Cambria, D. Das, S. Bandyopadhyay and A. Feraco, eds. A Practical Guide to Sentiment Analysis. Cham: Springer, pp. 61-83.
Pang, B., Lee, L. and Vaithyanathan, S., 2002. Thumbs up?: sentiment classification using machine learning techniques. Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing, Philadelphia, USA, July 6-7, 2002, pp. 79-86.
Ye, X., Dai, H., Dong, L.-A. and Wang, X., 2021. Multi-view ensemble learning method for microblog sentiment classification. Expert Systems with Applications, 166, DOI : 10.1016/j. eswa.2020.113987.
Alamoudi, E.S. and Alghamdi, N.S., 2021. Sentiment classification and aspect-based sentiment analysis on yelp reviews using deep learning and word embeddings. Journal of Decision Systems, 30(2-3), 259-281.
Brodley, C.E. and Friedl, M.A., 1999. Identifying mislabeled training data. Journal of Artificial Intelligence Research, 11, 131-167.
Wang, H., Liu, B., Li, C., Yang, Y. and Li, T., 2019. Learning with Noisy Labels for Sentence-Level Sentiment Classification. [online] Available at: https://arxiv.org/abs/1909.00124.
Wang, L., Xu, X., Guo, K. and Cai, B., 2018. Visual sentiment analysis with noisy labels by reweighting loss. 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan, October 7-10, 2018, pp. 1873-1878.
Wilson, D., 1972. Asymptotic properties of nearest neighbor rules using edited data. IEEE Transactions on Systems, Man, and Cybernetics, SMC-2(3), 408-421.
Tomek, I., 1976. An experiment with edited nearest-neighbor rule. IEEE Transactions on Systems, Man and Cybernetics, 6(6), 448-452.
Guyon, I., Matic, N. and Vapnik, V., 1996. Discovering informative patterns and data cleaning. In: U.M. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurasamy, eds. Advances in Knowledge Discovery and Data Mining. Cambridge: AAAI/MIT Press, pp. 181-203.
Oka, N. and Yoshida, K., 1996. A noise-tolerant hybrid model of a global and a local learning model. Proceedings of the AAAI-96 Workshop: Integrating Multiple Learned Models for Improving and Scaling Machine Learning Algorithms, Oregon, USA, August 2-8, 1966, pp. 95-100.
Natarajan, N., Dhillon, I.S., Ravikumar, P.K., and Tewari, A., 2013. Learning with noisy labels. In: C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, eds. Advances in Neural Information Processing Systems. New York: Curran Associates, Inc., pp. 1196-1204.
Liu, T. and Tao, D., 2016. Classification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(3), 447-461.
Reed, S.E., Lee, H., Anguelov, D., Szegedy, C., Erhan, D., and Rabinovich, A., 2015. Training Deep Neural Networks on Noisy Labels with Bootstrapping. [online] Available at: https://arxiv.org/abs/1412.6596.
Han, J., Luo, P. and Wang, X., 2019. Deep Self-learning from Noisy Labels. [online] Available at: https://arxiv.org/abs/1908.02160.
Sanchez, G., Guis, V., Marxer, R. and Bouchara, F., 2020. Deep Learning Classification with Noisy Labels. [online] Available at: https://arxiv.org/abs/2004.11116.
Cordeiro, F.R. and Carneiro, G., 2020. A Survey on Deep Learning with Noisy Labels: How to Train Your Model When You Cannot Trust on the Annotations? [online] Available at: https://arxiv.org/abs/2012.03061.
Song, H., Kim, M., Park, D., Shin, Y. and Lee, J.-G., 2021. Learning from Noisy Labels with Deep Neural Networks: A Survey. [online] Available at: https://arxiv.org/abs/2007.08199.
Patrini, G., Rozza, A., Menon, A., Nock, R. and Qu, L., 2021. Learning from Noisy Labels with Deep Neural Networks: A Survey. [online] Available at: https://arxiv.org/abs/1609.03683.
Garg, S., Ramakrishnan, G. and Thumbe, V., 2021. Towards Robustness to Label Noise in Text Classification via Noise Modeling. [online] Available at: https://arxiv.org/abs/2101.11214.
Wang, H., Lin, B., Li, C., Yang, Y. and Li, T., 2011. Learning with Noisy Labels for Sentence-Level Sentiment Classification. [online] Available at: https://arxiv.org/abs/1909.00124.
Malik, H.H. and Bhardwaj, V.S., 2011. Automatic training data cleaning for text classification. 2011 IEEE 11th International Conference on Data Mining Workshops, Vancouver, Canada, December 11, 2011, pp. 442-449.
Eamwiwat, C., Thanasutives, P., Saetia, C. and Chalothorn, T., 2019. Using label noise filtering and ensemble method for sentiment analysis on Thai social data. The 14th International Joint Symposium on Artificial Intelligence and Natural Language Processing, Chiang Mai, Thailand, October 30-November 1, 2019, pp. 251-256.
Yu, H. and Hatzivassiloglou, V., 2003. Towards answering opinion questions: separating facts from opinions and identifying the polarity of opinion sentences. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Sapporo, Japan, July 11-12, 2003, pp. 129-136.
Liu, Y., Yu, X., Liu, B. and Chen, Z., 2014. Sentence-level sentiment analysis in the presence of modalities. 15th International Conference on Intelligent Text Processing and Computational Linguistics, Kathmandu, Nepal, April 6-12, 2014, pp. 1-16.
Prasetiyowati, M.I., Maulidevi, N.U. and Surendro, K., 2021. Determining threshold value on information gain feature selection to increase speed and prediction accuracy of random forest. Journal of Big Data, 84, 2-22.
Polpinij, J. and Luaphol, B., 2021. Comparing of multi-class text classification methods for automatic ratings of consumer reviews. International Conference on Multi-disciplinary Trends in Artificial Intelligence, Kuala Lumpur, Malaysia, November 17-19, 2021, pp. 164-175.
Huq, M.R., Ali, A., and Rahman, A., 2017. Sentiment analysis on Twitter data using KNN and SVM. International Journal of Advanced Computer Science and Applications, 8(6), 9-25.
Shakhovska, K., Shakhovska, N., and Veselý, P., 2020. The sentiment analysis model of services providers’ feedback. Electronics, 9(11), 2-15.
Polpinij, J., and Ghose, A.K., 2008. An ontology-based sentiment classification methodology for online consumer reviews. 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, Washington DC, USA, December 9-12, 2008, pp. 518-524.
Luaphol, B., Polpinij, J., and Kaenampornpun, M., 2021. Mining bug report repositories to identify significant information for software bug fixing. Applied Science and Engineering Progress, 15(3), DOI: 10.14416/j.asep.2021.03.005.
Pennington, J., Socher, R., and Manning, C., 2014. Glove : Global vectors for word representation. The Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, October 25-29, 2014, pp. 1532-1543.