maisМоделирование и анализ информационных системModeling and Analysis of Information Systems1818-10152313-5417Yaroslavl State University10.18255/1818-1015-2023-1-86-100mais-1768Research ArticleTheory of DataРазметка корпусов текстов по тональности и наличию иронии в рамках проекта гражданской наукиAnnotation of Text Corpora by Sentiment and Presence of Irony within a Project of Citizen Sciencehttps://orcid.org/0000-0003-3984-8423ПарамоновИлья ВячеславовичParamonovIlya Vyacheslavovichilya.paramonov@fruct.orghttps://orcid.org/0000-0003-0116-4739ПолетаевАнатолий ЮрьевичPoletaevAnatoliy Yurievichanatoliy-poletaev@mail.ruЯрославский государственный университет им. П. Г. ДемидоваРоссияP. G. Demidov Yaroslavl State UniversityRussian Federation20232804202330186100Copyright © Парамонов И.В., Полетаев А.Ю., 20232023Парамонов И.В., Полетаев А.Ю.Paramonov I.V., Poletaev A.Y.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://www.mais-journal.ru/jour/article/view/1768

Статья посвящена построению корпуса предложений, размеченных по общей тональности на 4 класса (положительный, отрицательный, нейтральный, смешанный), корпуса фразеологизмов, размеченных по тональности на 3 класса (положительный, отрицательный, нейтральный), и корпуса предложений, размеченных по наличию или отсутствию иронии. Разметку проводили волонтёры в рамках проекта «Готовим тексты алгоритмам» на портале «Люди науки». На основе имеющихся знаний о предметной области для каждой из задач были составлены инструкции для разметчиков. Также была выработана методика статистической обработки результатов разметки, основанная на анализе распределений и показателей согласия оценок, выставленных разными разметчиками. Для разметки предложений по наличию иронии и фразеологизмов по тональности показатели согласия оказались достаточно высокими (доля полного совпадения 0.60--0.99), при разметке предложений по общей тональности согласие оказалось слабым (доля полного совпадения 0.40), по-видимому, из-за более высокой сложности задачи. Также было показано, что результаты работы автоматических алгоритмов анализа тональности предложений улучшаются на 12--13 % при использовании корпуса, относительно предложений которого сошлись мнения всех разметчиков (3--5 человек), по сравнению с корпусом с разметкой только одним волонтёром.

The paper is devoted to construction of a sentence corpus annotated by the general sentiment into 4 classes (positive, negative, neutral, and mixed), a corpus of phrasemes annotated by the sentiment into 3 classes (positive, negative, and neutral), and a corpus of sentences annotated by the presence or absence of irony. The annotation was done by volunteers within the project “Prepare texts for algorithms” on the portal “People of science”. The existing knowledge on the domain regarding each task was the basis to develop guidelines for annotators. A technique of statistical analysis of the annotation result based on the distributions and agreement measures of the annotations performed by various annotators was also developed. For the annotation of sentences by irony and phrasemes by the sentiment the agreement measures were rather high (the full agreement rate of 0.60--0.99), whereas for the annotation of sentences by the general sentiment the agreement was low (the full agreement rate of 0.40), presumably, due to the higher complexity of the task. It was also shown that the results of automatic algorithms of detecting the sentiment of sentences improved by 12–13 % when using a corpus for which all the annotators (from 3 till 5) had the agreement, in comparison with a corpus annotated by only one volunteer.

анализ тональноститекстовый корпусстатистический анализпоказатели согласиягражданская наукаsentiment analysistext corpusstatistical analysisagreement measurescitizen scienceИсследование выполнено в рамках проекта гражданской науки ЯрГУ No CS-02/2022.The research was performed within the YarSU citizen science project No. CS-02/2022.ReferencesV. Masoumi, M. Salehi, H. Veisi, G. Haddadian, V. Ranjbar, and M. Sahebdel, “TeleCrowd: A Crowdsourcing Approach to Create Informal to Formal Text Corpora.” 2020.V. Masoumi, M. Salehi, H. Veisi, G. Haddadian, V. Ranjbar, and M. Sahebdel, “TeleCrowd: A Crowdsourcing Approach to Create Informal to Formal Text Corpora.” 2020.E. Mitiagina, M. Borodataya, E. Volchenkova, N. Ershova, M. Luchinina, and E. Kotelnikov, “Russian Text Corpus of Intimate Partner Violence: Annotation Through Crowdsourcing,” in 7th International Conference on Electronic Governance and Open Society: Challenges in Eurasia. EGOSE 2020, Springer, 2020, pp. 306–321.E. Mitiagina, M. Borodataya, E. Volchenkova, N. Ershova, M. Luchinina, and E. Kotelnikov, “Russian Text Corpus of Intimate Partner Violence: Annotation Through Crowdsourcing,” in 7th International Conference on Electronic Governance and Open Society: Challenges in Eurasia. EGOSE 2020, Springer, 2020, pp. 306–321.S. Mohammad, “A practical guide to sentiment annotation: Challenges and solutions,” in Proceedings of the 7th workshop on computational approaches to subjectivity, sentiment and social media analysis, 2016, pp. 174–179.S. Mohammad, “A practical guide to sentiment annotation: Challenges and solutions,” in Proceedings of the 7th workshop on computational approaches to subjectivity, sentiment and social media analysis, 2016, pp. 174–179.S. M. Mohammad, P. Sobhani, and S. Kiritchenko, “Stance and Sentiment in Tweets,” Special Section of the ACM Transactions on Internet Technology on Argumentation in Social Media, vol. 17, no. 3, pp. 1–23, 2017.S. M. Mohammad, P. Sobhani, and S. Kiritchenko, “Stance and Sentiment in Tweets,” Special Section of the ACM Transactions on Internet Technology on Argumentation in Social Media, vol. 17, no. 3, pp. 1–23, 2017.B. R. Chakravarthi, V. Muralidaran, R. Priyadharshini, and J. P. McCrae, “Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Text,” in Proceedings of the 1st Joint SLTU and CCURL Workshop (SLTU-CCURL 2020), 2020, pp. 202–210.B. R. Chakravarthi, V. Muralidaran, R. Priyadharshini, and J. P. McCrae, “Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Text,” in Proceedings of the 1st Joint SLTU and CCURL Workshop (SLTU-CCURL 2020), 2020, pp. 202–210.K. Krippendorff, Content analysis: an introduction to its methodology. Thousand Oaks, CA: SAGE Publications, Inc., 2013.K. Krippendorff, Content analysis: an introduction to its methodology. Thousand Oaks, CA: SAGE Publications, Inc., 2013.Y. Zhao, B. Qin, and T. Liu, “Creating a fine-grained corpus for chinese sentiment analysis,” IEEE Intelligent Systems, vol. 30, no. 1, pp. 36–43, 2014.Y. Zhao, B. Qin, and T. Liu, “Creating a fine-grained corpus for chinese sentiment analysis,” IEEE Intelligent Systems, vol. 30, no. 1, pp. 36–43, 2014.J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement, vol. 20, no. 1, pp. 37–46, 1960.J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement, vol. 20, no. 1, pp. 37–46, 1960.J. Bu et al., “ASAP: A Chinese Review Dataset Towards Aspect Category Sentiment Analysis and Rating Prediction,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 2069–2079.J. Bu et al., “ASAP: A Chinese Review Dataset Towards Aspect Category Sentiment Analysis and Rating Prediction,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 2069–2079.M. Navas-Loro, V. Rodr'iguez-Doncel, I. Santana-Perez, and A. S'anchez, “Spanish Corpus for Sentiment Analysis Towards Brands,” in Speech and Computer. SPECOM 2017, 2017, pp. 680–689.M. Navas-Loro, V. Rodr'iguez-Doncel, I. Santana-Perez, and A. S'anchez, “Spanish Corpus for Sentiment Analysis Towards Brands,” in Speech and Computer. SPECOM 2017, 2017, pp. 680–689.J. L. Fleiss, “Measuring nominal scale agreement among many raters,” Psychological bulletin, vol. 76, no. 5, p. 378, 1971.J. L. Fleiss, “Measuring nominal scale agreement among many raters,” Psychological bulletin, vol. 76, no. 5, p. 378, 1971.A. Rogers, A. Romanov, A. Rumshisky, S. Volkova, M. Gronas, and A. Gribov, “RuSentiment: An enriched sentiment analysis dataset for social media in Russian,” in Proceedings of the 27th international conference on computational linguistics, 2018, pp. 755–763.A. Rogers, A. Romanov, A. Rumshisky, S. Volkova, M. Gronas, and A. Gribov, “RuSentiment: An enriched sentiment analysis dataset for social media in Russian,” in Proceedings of the 27th international conference on computational linguistics, 2018, pp. 755–763.T. V. Zherebilo, Slovar lingvisticheskih terminov. Nazran: OOO Piligrim, 2010.T. V. Zherebilo, Slovar lingvisticheskih terminov. Nazran: OOO Piligrim, 2010.K. Krippendorff, “Computing Krippendorff's Alpha-Reliability.” 2008, Accessed: Jan. 17, 2023. [Online]. Available: https://repository.upenn.edu/asc_papers/43/.K. Krippendorff, “Computing Krippendorff's Alpha-Reliability.” 2008, Accessed: Jan. 17, 2023. [Online]. Available: https://repository.upenn.edu/asc_papers/43/.J. Hughes, “krippendorffsalpha: An R package for measuring agreement using Krippendorff's alpha coefficient,” The R Journal, vol. 13, no. 1, pp. 413–425, 2021.J. Hughes, “krippendorffsalpha: An R package for measuring agreement using Krippendorff's alpha coefficient,” The R Journal, vol. 13, no. 1, pp. 413–425, 2021.L. A. Jeni, J. F. Cohn, and F. De La Torre, “Facing imbalanced data--recommendations for the use of performance metrics,” in 2013 Humaine association conference on affective computing and intelligent interaction, 2013, pp. 245–251.L. A. Jeni, J. F. Cohn, and F. De La Torre, “Facing imbalanced data--recommendations for the use of performance metrics,” in 2013 Humaine association conference on affective computing and intelligent interaction, 2013, pp. 245–251.A. Y. Poletaev and I. V. Paramonov, “Recursive sentiment detection algorithm for Russian sentences,” Modelirovanie i Analiz Informatsionnykh Sistem, vol. 29, no. 2, pp. 134–147, 2022.A. Y. Poletaev and I. V. Paramonov, “Recursive sentiment detection algorithm for Russian sentences,” Modelirovanie i Analiz Informatsionnykh Sistem, vol. 29, no. 2, pp. 134–147, 2022.S. Smetanin and M. Komarov, “Deep transfer learning baselines for sentiment analysis in Russian,” Information Processing & Management, vol. 58, no. 3, p. 102484, 2021.S. Smetanin and M. Komarov, “Deep transfer learning baselines for sentiment analysis in Russian,” Information Processing & Management, vol. 58, no. 3, p. 102484, 2021.R. Artstein and M. Poesio, “Inter-coder agreement for computational linguistics,” Computational linguistics, vol. 34, no. 4, pp. 555–596, 2008.R. Artstein and M. Poesio, “Inter-coder agreement for computational linguistics,” Computational linguistics, vol. 34, no. 4, pp. 555–596, 2008.

The authors declare that there are no conflicts of interest present.