maisМоделирование и анализ информационных системModeling and Analysis of Information Systems1818-10152313-5417Yaroslavl State University10.18255/1818-1015-2023-3-202-213mais-1799Research ArticleTheory of DataКлассификация текстов по уровням CEFR с использованием методов машинного обучения и языковой модели BERTText classification by CEFR levels using machine learning methods and BERT language modelhttps://orcid.org/0000-0002-6137-8643ЛагутинаНадежда СтаниславовнаLagutinaNadezhda S.lagutinans@gmail.comhttps://orcid.org/0000-0002-1742-3240ЛагутинаКсения ВладимировнаLagutinaKsenia V.lagutinakv@mail.ruhttps://orcid.org/0009-0003-1741-0604БредерманАнастасия МихайловнаBredermanAnastasya M.anastasyabrederman@mail.ruhttps://orcid.org/0000-0002-6757-9622КасаткинаНаталья НиколаевнаKasatkinaNatalia N.ninet75@mail.ruЯрославский государственный университет им. П.Г. ДемидоваРоссияP.G. Demidov Yaroslavl State UniversityRussian Federation202317092023303202213Copyright © Лагутина Н.С., Лагутина К.В., Бредерман А.М., Касаткина Н.Н., 20232023Лагутина Н.С., Лагутина К.В., Бредерман А.М., Касаткина Н.Н.Lagutina N.S., Lagutina K.V., Brederman A.M., Kasatkina N.N.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://www.mais-journal.ru/jour/article/view/1799

В данной работе представлено исследование задачи автоматической классификации коротких связных текстов (эссе) на английском языке по уровням международной шкалы CEFR. Определение уровня текста на естественном языке является важной составляющей оценки знаний учащихся, в том числе для проверки открытых заданий в системах электронного обучения. Для решения этой задачи были рассмотрены векторные модели текста на основе стилометрических числовых характеристик уровня символов, слов, структуры предложения. Классификация полученных векторов осуществлялась стандартными классификаторами машинного обучения. В статье приведены результаты трёх наиболее успешных: Support Vector Classifier, Stochastic Gradient Descent Classifier, LogisticRegression. Оценкой качества послужили точность, полнота и F"=мера. Для экспериментов были выбраны два открытых корпуса текстов CEFR Levelled English Texts и BEA"=2019. Лучшие результаты классификации по шести уровням и подуровням CEFR от A1 до C2 показал Support Vector Classifier с F"=мерой 67 % для корпуса CEFR Levelled English Texts. Этот подход сравнивался с применением языковой модели BERT (шесть различных вариантов). Лучшая модель bert"=base"=cased обеспечила значение F"=меры 69 %. Анализ ошибок классификации показал, что большая их часть допущена между соседними уровнями, что вполне объяснимо с точки зрения предметной области. Кроме того, качество классификации сильно зависело от корпуса текстов, что продемонстрировало существенное различие F"=меры в ходе применения одинаковых моделей текста для разных корпусов. В целом, полученные результаты показали эффективность автоматического определения уровня текста и возможность его практического применения.

This paper presents a study of the problem of automatic classification of short coherent texts (essays) in English according to the levels of the international CEFR scale. Determining the level of text in natural language is an important component of assessing students knowledge, including checking open tasks in e-learning systems. To solve this problem, vector text models were considered based on stylometric numerical features of the character, word, sentence structure levels. The classification of the obtained vectors was carried out by standard machine learning classifiers. The article presents the results of the three most successful ones: Support Vector Classifier, Stochastic Gradient Descent Classifier, LogisticRegression. Precision, recall and F-score served as quality measures. Two open text corpora, CEFR Levelled English Texts and BEA-2019, were chosen for the experiments. The best classification results for six CEFR levels and sublevels from A1 to C2 were shown by the Support Vector Classifier with F-score 67 % for the CEFR Levelled English Texts. This approach was compared with the application of the BERT language model (six different variants). The best model, bert-base-cased, provided the F-score value of 69 %. The analysis of classification errors showed that most of them are between neighboring levels, which is quite understandable from the point of view of the domain. In addition, the quality of classification strongly depended on the text corpus, that demonstrated a significant difference in F-scores during application of the same text models for different corpora. In general, the obtained results showed the effectiveness of automatic text level detection and the possibility of its practical application.

автоматическая обработка текстаклассификация текстаCEFRBERTnatural language processingtext classificationCEFRBERTИсследование выполнено за счет средств Программы развития ЯрГУ до 2030 года, проект No GM-2023- 123061600058-4 «Разработка автоматизированной системы развития медиативной компетенции в языковом образовании».This study was supported by YarSU Development Program until 2030, project No. GM-2023-123061600058-4 “Develop- ment of an automated system for the development of mediative competence in language education”.ReferencesE. del Gobbo, A. Guarino, B. Cafarelli, L. Grilli, and P. Limone, “Automatic evaluation of open-ended questions for online learning. A systematic mapping,” Studies in Educational Evaluation, vol. 77, p. 101258, 2023.E. del Gobbo, A. Guarino, B. Cafarelli, L. Grilli, and P. Limone, “Automatic evaluation of open-ended questions for online learning. A systematic mapping,” Studies in Educational Evaluation, vol. 77, p. 101258, 2023.N. V. Galichev and P. S. Shirogorodskaya, “Problema avtomaticheskogo izmereniya slozhnyh konstruktov cherez otkrytye zadaniya,” in HXI Mezhdunarodnaya nauchno-prakticheskaya konferenciya molodyh issledovatelej obrazovaniya, 2022, pp. 695–697.N. V. Galichev and P. S. Shirogorodskaya, “Problema avtomaticheskogo izmereniya slozhnyh konstruktov cherez otkrytye zadaniya,” in HXI Mezhdunarodnaya nauchno-prakticheskaya konferenciya molodyh issledovatelej obrazovaniya, 2022, pp. 695–697.L. E. Adamova, O. V. Surikova, I. G. Bulatova, and O. O. Varlamov, “Application of the mivar expert system to evaluate the complexity of texts,” News of the Kabardin-Balkar scientific center of RAS, no. 2, pp. 11–29, 2021.L. E. Adamova, O. V. Surikova, I. G. Bulatova, and O. O. Varlamov, “Application of the mivar expert system to evaluate the complexity of texts,” News of the Kabardin-Balkar scientific center of RAS, no. 2, pp. 11–29, 2021.D. Ramesh and S. K. Sanampudi, “An automated essay scoring systems: a systematic literature review,” Artificial Intelligence Review, vol. 55, no. 3, pp. 2495–2527, 2022.D. Ramesh and S. K. Sanampudi, “An automated essay scoring systems: a systematic literature review,” Artificial Intelligence Review, vol. 55, no. 3, pp. 2495–2527, 2022.K. P. Yancey, G. Laflair, A. Verardi, and J. Burstein, “Rating Short L2 Essays on the CEFR Scale with GPT-4,” in Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), 2023, pp. 576–584.K. P. Yancey, G. Laflair, A. Verardi, and J. Burstein, “Rating Short L2 Essays on the CEFR Scale with GPT-4,” in Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), 2023, pp. 576–584.A. Gasparetto, M. Marcuzzo, A. Zangari, and A. Albarelli, “A survey on text classification algorithms: From text to predictions,” Information, vol. 13, no. 2, p. 83, 2022.A. Gasparetto, M. Marcuzzo, A. Zangari, and A. Albarelli, “A survey on text classification algorithms: From text to predictions,” Information, vol. 13, no. 2, p. 83, 2022.V. Ramnarain-Seetohul, V. Bassoo, and Y. Rosunally, “Similarity measures in automated essay scoring systems: A ten-year review,” Education and Information Technologies, vol. 27, no. 4, pp. 5573–5604, 2022.V. Ramnarain-Seetohul, V. Bassoo, and Y. Rosunally, “Similarity measures in automated essay scoring systems: A ten-year review,” Education and Information Technologies, vol. 27, no. 4, pp. 5573–5604, 2022.P. Yang, L. Li, F. Luo, T. Liu, and X. Sun, “Enhancing topic-to-essay generation with external commonsense knowledge,” in Proceedings of the 57th annual meeting of the association for computational linguistics, 2019, pp. 2002–2012.P. Yang, L. Li, F. Luo, T. Liu, and X. Sun, “Enhancing topic-to-essay generation with external commonsense knowledge,” in Proceedings of the 57th annual meeting of the association for computational linguistics, 2019, pp. 2002–2012.N. N. Mikheeva and E. V. Shulyndina, “Features of training written Internet communication in a non-linguistic university,” Tambov University Review. Series: Humanities, vol. 28, no. 2, pp. 405–414, 2023.N. N. Mikheeva and E. V. Shulyndina, “Features of training written Internet communication in a non-linguistic university,” Tambov University Review. Series: Humanities, vol. 28, no. 2, pp. 405–414, 2023.V. J. Schmalz and A. Brutti, “Automatic assessment of English CEFR levels using BERT embeddings,” 2021.V. J. Schmalz and A. Brutti, “Automatic assessment of English CEFR levels using BERT embeddings,” 2021.Y. Arase, S. Uchida, and T. Kajiwara, “CEFR-Based Sentence Difficulty Annotation and Assessment,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 6206–6219.Y. Arase, S. Uchida, and T. Kajiwara, “CEFR-Based Sentence Difficulty Annotation and Assessment,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 6206–6219.R. Jalota, P. Bourgonje, J. Van Sas, and H. Huang, “Mitigating Learnerese Effects for CEFR classification,” in Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), 2022, pp. 14–21.R. Jalota, P. Bourgonje, J. Van Sas, and H. Huang, “Mitigating Learnerese Effects for CEFR classification,” in Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), 2022, pp. 14–21.T. Gaillat et al., “Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach,” ReCALL, vol. 34, no. 2, pp. 130–146, 2022.T. Gaillat et al., “Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach,” ReCALL, vol. 34, no. 2, pp. 130–146, 2022.E. Kerz, D. Wiechmann, Y. Qiao, E. Tseng, and M. Str"obel, “Automated classification of written proficiency levels on the CEFR-scale through complexity contours and RNNs,” in Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, 2021, pp. 199–209.E. Kerz, D. Wiechmann, Y. Qiao, E. Tseng, and M. Str"obel, “Automated classification of written proficiency levels on the CEFR-scale through complexity contours and RNNs,” in Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, 2021, pp. 199–209.Y. Yang and J. Zhong, “Automated essay scoring via example-based learning,” in Web Engineering, 2021, pp. 201–208.Y. Yang and J. Zhong, “Automated essay scoring via example-based learning,” in Web Engineering, 2021, pp. 201–208.E. Mayfield and A. W. Black, “Should you fine-tune BERT for automated essay scoring?,” in Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, 2020, pp. 151–162.E. Mayfield and A. W. Black, “Should you fine-tune BERT for automated essay scoring?,” in Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, 2020, pp. 151–162.J. M. Imperial, “BERT Embeddings for Automatic Readability Assessment,” in Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), 2021, pp. 611–618.J. M. Imperial, “BERT Embeddings for Automatic Readability Assessment,” in Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), 2021, pp. 611–618.C. Bryant, M. Felice, O. E. Andersen, and T. Briscoe, “The BEA-2019 shared task on grammatical error correction,” in Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, 2019, pp. 52–75.C. Bryant, M. Felice, O. E. Andersen, and T. Briscoe, “The BEA-2019 shared task on grammatical error correction,” in Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, 2019, pp. 52–75.K. V. Lagutina and A. M. Manakhova, “Automated Search and Analysis of the Stylometric Features That Describe the Style of the Prose of 19th--21st Centuries,” Automatic Control and Computer Sciences, vol. 55, no. 7, pp. 866–876, 2021.K. V. Lagutina and A. M. Manakhova, “Automated Search and Analysis of the Stylometric Features That Describe the Style of the Prose of 19th--21st Centuries,” Automatic Control and Computer Sciences, vol. 55, no. 7, pp. 866–876, 2021.A. M. Manakhova and N. S. Lagutina, “Analysis of the impact of the stylometric characteristics of different levels for the verification of authors of the prose,” Modeling and Analysis of Information Systems, vol. 28, no. 3, pp. 260–279, 2021.A. M. Manakhova and N. S. Lagutina, “Analysis of the impact of the stylometric characteristics of different levels for the verification of authors of the prose,” Modeling and Analysis of Information Systems, vol. 28, no. 3, pp. 260–279, 2021.J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019, vol. 1, pp. 4171–4186.J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019, vol. 1, pp. 4171–4186.V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.” 2020.V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.” 2020.

The authors declare that there are no conflicts of interest present.