maisМоделирование и анализ информационных системModeling and Analysis of Information Systems1818-10152313-5417Yaroslavl State University10.18255/1818-1015-2025-3-298-310mais-1964Research ArticleArtificial IntelligenceСравнение современных моделей русскоязычных текстов для задачи классификации по уровням CEFRModern Russian-language texts models comparison for the task of CEFR levels classificationhttps://orcid.org/0000-0003-3147-077XЛавровскийВадим АлексеевичLavrovskiyVadim A.comado19@gmail.comhttps://orcid.org/0000-0002-6137-8643ЛагутинаНадежда СтаниславовнаLagutinaNadezhda S.lagutinans@rambler.ruhttps://orcid.org/0009-0000-1575-7098ЛавровскаяОльга БорисовнаLavrovskayaOlga B.o.lavrovskaya@uniyar.ac.ruЯрославский государственный университет им. П.Г. ДемидоваРоссияP.G. Demidov Yaroslavl State UniversityRussian Federation202524092025323298310Copyright © Лавровский В.А., Лагутина Н.С., Лавровская О.Б., 20252025Лавровский В.А., Лагутина Н.С., Лавровская О.Б.Lavrovskiy V.A., Lagutina N.S., Lavrovskaya O.B.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://www.mais-journal.ru/jour/article/view/1964

Разработка качественных инструментов автоматического определения уровней текстов по шкале CEFR позволяет создавать учебные и проверочные материалы более быстро и объективно. В данной работе авторы исследуют два типа современных моделей текста: лингвистические характеристики и эмбеддинги больших языковых моделей для задачи классификации русскоязычных текстов по шести уровням CEFR: A1—C2 и трём укрупнённым категориям A, B, C. Два вида моделей явным образом представляет текст в виде вектора числовых характеристик. При этом разделение текста на уровни рассматривается как обычная задача классификации в области компьютерной лингвистики. Эксперименты проводились с собственным корпусом из 1904 текстов. Лучшее качество достигается rubert-base-cased-conversational без дополнительной адаптации при определении как шести, так и трёх категорий текста. Максимальное значение F-меры для уровней A, B, C равно 0,77. Максимальное значение F-меры для прогнозирования шести категорий текста равно 0,67. Качество определения уровня текста больше зависит от модели, чем от алгоритма классификации машинного обучения. Результаты отличаются друг от друга не более чем на 0,01-0,02, особенно это касается ансамблевых методов.

The development of high-quality tools for automatic determination of text levels according to the CEFR scale allows creating educational and testing materials more quickly and objectively. In this paper, the authors examine two types of modern text models: linguistic characteristics and embeddings of large language models for the task of classifying Russian-language texts by six CEFR levels: A1-C2 and three broader categories A, B, C. The two types of models explicitly represent the text as a vector of numerical characteristics. In this case, dividing the text into levels is considered as a common classification task in the field of computational linguistics. The experiments were conducted with our own corpus of 1904 texts. The best quality is achieved by rubert-base-cased-conversational without additional adaptation when determining both six and three text categories. The maximum F-measure value for levels A, B, C is 0.77. The maximum F-measure value for predicting six text categories is 0.67. The quality of text level determination depends more on the model than on the machine learning classification algorithm. The results differ from each other by no more than 0.01-0.02, especially for ensemble methods.

автоматическая обработка текстаклассификация русскоязычных текстовлингвистические характеристикиэмбеддингиBERTGPTCEFRnatural language processingRussian-language texts classificationlinguistic characteristicsembeddingsBERTGPTCEFRРабота выполнена в рамках гранта министерства социальных коммуникаций и научно-технологическо- го развития Ярославской области по теме научного исследования «Автоматический анализ текстов», соглашение№17НП/ 2024 от 24 декабря 2024 годаThe work was carried out within the framework of a grant from the Ministry of Social Communications and Scientific and Technological Development of the Yaroslavl Region on the topic of scientific research ”Automatic text analysis”, agreement No. 17NP/2024 dated December 24, 2024.ReferencesN. V. Bordovskaya, E. A. Koshkina, M. A. Tikhomirova, and L. A. Melkaya, “Blended Educational Technologies in Higher Education: Systematic Review of Domestic Publications,” Vysshee obrazovanie v Rossii = Higher Education in Russia, vol. 31, no. 8-9, pp. 58–78, 2022, doi: 10.31992/0869-3617-2022-31-8-9-58-78.N. V. Bordovskaya, E. A. Koshkina, M. A. Tikhomirova, and L. A. Melkaya, “Blended Educational Technologies in Higher Education: Systematic Review of Domestic Publications,” Vysshee obrazovanie v Rossii = Higher Education in Russia, vol. 31, no. 8-9, pp. 58–78, 2022, doi: 10.31992/0869-3617-2022-31-8-9-58-78.A. W. Zaki and R. Darmi, “CEFR: Education towards 21st century of learning. Why matters,” Journal of Social Science and Humanities, vol. 4, no. 2, pp. 14–20, 2021, doi: 10.26666/rmp.jssh.2021.2.3.A. W. Zaki and R. Darmi, “CEFR: Education towards 21st century of learning. Why matters,” Journal of Social Science and Humanities, vol. 4, no. 2, pp. 14–20, 2021, doi: 10.26666/rmp.jssh.2021.2.3.T. Gaillat et al., “Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach,” ReCALL, vol. 34, no. 2, pp. 130–146, 2022, doi: 10.1017/S095834402100029X.T. Gaillat et al., “Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach,” ReCALL, vol. 34, no. 2, pp. 130–146, 2022, doi: 10.1017/S095834402100029X.I. Natova, “Estimating CEFR reading comprehension text complexity,” The Language Learning Journal, vol. 49, no. 6, pp. 699–710, 2021, doi: 10.1080/09571736.2019.1665088.I. Natova, “Estimating CEFR reading comprehension text complexity,” The Language Learning Journal, vol. 49, no. 6, pp. 699–710, 2021, doi: 10.1080/09571736.2019.1665088.G. Fakhretdinova, L. M. Zinnatullina, F. T. Galeeva, and E. Valeeva, “The CEFR for languages: research perspectives in foreign language teaching in Engineering university,” in International Conference on Interactive Collaborative Learning, 2021, pp. 225–232, doi: 10.1007/978-3-030-93907-6_24.G. Fakhretdinova, L. M. Zinnatullina, F. T. Galeeva, and E. Valeeva, “The CEFR for languages: research perspectives in foreign language teaching in Engineering university,” in International Conference on Interactive Collaborative Learning, 2021, pp. 225–232, doi: 10.1007/978-3-030-93907-6_24.A. Dmitrieva, A. Laposhina, and M. Lebedeva, “A comparative study of educational texts for native, foreign, and bilingual young speakers of russian: are simplified texts equally simple?,” Frontiers in Psychology, vol. 12, p. 703690, 2021, doi: 10.3389/fpsyg.2021.703690.A. Dmitrieva, A. Laposhina, and M. Lebedeva, “A comparative study of educational texts for native, foreign, and bilingual young speakers of russian: are simplified texts equally simple?,” Frontiers in Psychology, vol. 12, p. 703690, 2021, doi: 10.3389/fpsyg.2021.703690.V. J. Schmalz and A. Brutti, “Automatic Assessment of English CEFR Levels Using BERT Embeddings,” in Computational Linguistics CliC-it 2021, 2022, pp. 293–299.V. J. Schmalz and A. Brutti, “Automatic Assessment of English CEFR Levels Using BERT Embeddings,” in Computational Linguistics CliC-it 2021, 2022, pp. 293–299.N. S. Lagutina, K. V. Lagutina, A. M. Brederman, and N. N. Kasatkina, “Text classification by CEFR levels using machine learning methods and the BERT language model,” Automatic Control and Computer Sciences, vol. 58, no. 7, pp. 869–878, 2024, doi: 10.3103/S0146411624700329.N. S. Lagutina, K. V. Lagutina, A. M. Brederman, and N. N. Kasatkina, “Text classification by CEFR levels using machine learning methods and the BERT language model,” Automatic Control and Computer Sciences, vol. 58, no. 7, pp. 869–878, 2024, doi: 10.3103/S0146411624700329.R. M. S'anchez, D. Alfter, S. Dobnik, M. Szawerna, and E. Volodina, “Jingle BERT, Jingle BERT, Frozen All the Way: Freezing Layers to Identify CEFR Levels of Second Language Learners Using BERT,” in Swedish Language Technology Conference and NLP4CALL, 2024, pp. 137–152, doi: 10.3384/ecp211011.R. M. S'anchez, D. Alfter, S. Dobnik, M. Szawerna, and E. Volodina, “Jingle BERT, Jingle BERT, Frozen All the Way: Freezing Layers to Identify CEFR Levels of Second Language Learners Using BERT,” in Swedish Language Technology Conference and NLP4CALL, 2024, pp. 137–152, doi: 10.3384/ecp211011.K. Clark, U. Khandelwal, O. Levy, and C. D. Manning, “What Does BERT Look at? An Analysis of BERT’s Attention,” in Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2019, pp. 276–286, doi: 10.18653/v1/W19-4828.K. Clark, U. Khandelwal, O. Levy, and C. D. Manning, “What Does BERT Look at? An Analysis of BERT’s Attention,” in Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2019, pp. 276–286, doi: 10.18653/v1/W19-4828.N. Khallaf and S. Sharoff, “Automatic Difficulty Classification of Arabic Sentences,” in Proceedings of the Sixth Arabic Natural Language Processing Workshop, 2021, pp. 105–114.N. Khallaf and S. Sharoff, “Automatic Difficulty Classification of Arabic Sentences,” in Proceedings of the Sixth Arabic Natural Language Processing Workshop, 2021, pp. 105–114.D. Kogan et al., “Ace-CEFR--A Dataset for Automated Evaluation of the Linguistic Difficulty of Conversational Texts for LLM Applications.” 2025.D. Kogan et al., “Ace-CEFR--A Dataset for Automated Evaluation of the Linguistic Difficulty of Conversational Texts for LLM Applications.” 2025.H. Ma, J. Wang, and L. He, “Linguistic features distinguishing students’ writing ability aligned with CEFR levels,” Applied Linguistics, vol. 45, no. 4, pp. 637–657, 2024, doi: 10.1093/applin/amad054.H. Ma, J. Wang, and L. He, “Linguistic features distinguishing students’ writing ability aligned with CEFR levels,” Applied Linguistics, vol. 45, no. 4, pp. 637–657, 2024, doi: 10.1093/applin/amad054.V. Franzoni, G. Biondi, A. Milani, and others, “Morpho-Phraseological Based Classification of CEFR Italian L2 Learner Writing Proficiency,” IEEE ACCESS, vol. 12, pp. 156433–156441, 2024, doi: 10.1109/ACCESS.2024.3485988.V. Franzoni, G. Biondi, A. Milani, and others, “Morpho-Phraseological Based Classification of CEFR Italian L2 Learner Writing Proficiency,” IEEE ACCESS, vol. 12, pp. 156433–156441, 2024, doi: 10.1109/ACCESS.2024.3485988.A. N. Laposhina and M. Y. Lebedeva, “Textometr: an online tool for automated complexity level assessment of texts for Russian language learners,” Russian language studies, vol. 19, no. 3, pp. 331–345, 2021, doi: 10.22363/2618-8163-2021-19-3-331-345.A. N. Laposhina and M. Y. Lebedeva, “Textometr: an online tool for automated complexity level assessment of texts for Russian language learners,” Russian language studies, vol. 19, no. 3, pp. 331–345, 2021, doi: 10.22363/2618-8163-2021-19-3-331-345.I. Glivsi'c, C. L. Richter, and A. K. Ingason, “Testing relevant linguistic features in automatic CEFR skill level classification for Icelandic,” in Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), 2025, pp. 217–222.I. Glivsi'c, C. L. Richter, and A. K. Ingason, “Testing relevant linguistic features in automatic CEFR skill level classification for Icelandic,” in Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), 2025, pp. 217–222.N. Lagutina, K. Lagutina, A. Brederman, and N. Kasatkina, “Text classification by CEFR levels using machine learning methods and BERT language model,” Modeling and Analysis of Information Systems, vol. 30, no. 3, pp. 202–213, 2023, doi: 10.18255/1818-1015-2023-3-202-213.N. Lagutina, K. Lagutina, A. Brederman, and N. Kasatkina, “Text classification by CEFR levels using machine learning methods and BERT language model,” Modeling and Analysis of Information Systems, vol. 30, no. 3, pp. 202–213, 2023, doi: 10.18255/1818-1015-2023-3-202-213.P. Qi, Y. Zhang, Y. Zhang, J. Bolton, and C. D. Manning, “Stanza: A Python Natural Language Processing Toolkit for Many Human Languages,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020, pp. 101–108, doi: 10.18653/v1/2020.acl-demos.14.P. Qi, Y. Zhang, Y. Zhang, J. Bolton, and C. D. Manning, “Stanza: A Python Natural Language Processing Toolkit for Many Human Languages,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020, pp. 101–108, doi: 10.18653/v1/2020.acl-demos.14.

The authors declare that there are no conflicts of interest present.