Modern Russian-language texts models comparison for the task of CEFR levels classification

The development of high-quality tools for automatic determination of text levels according to the CEFR scale allows creating educational and testing materials more quickly and objectively. In this paper, the authors examine two types of modern text models: linguistic characteristics and embeddings of large language models for the task of classifying Russian-language texts by six CEFR levels: A1-C2 and three broader categories A, B, C. The two types of models explicitly represent the text as a vector of numerical characteristics. In this case, dividing the text into levels is considered as a common classification task in the field of computational linguistics. The experiments were conducted with our own corpus of 1904 texts. The best quality is achieved by rubert-base-cased-conversational without additional adaptation when determining both six and three text categories. The maximum F-measure value for levels A, B, C is 0.77. The maximum F-measure value for predicting six text categories is 0.67. The quality of text level determination depends more on the model than on the machine learning classification algorithm. The results differ from each other by no more than 0.01-0.02, especially for ensemble methods.

1. N. V. Bordovskaya, E. A. Koshkina, M. A. Tikhomirova, and L. A. Melkaya, “Blended Educational Technologies in Higher Education: Systematic Review of Domestic Publications,” Vysshee obrazovanie v Rossii = Higher Education in Russia, vol. 31, no. 8-9, pp. 58–78, 2022, doi: 10.31992/0869-3617-2022-31-8-9-58-78.

2. A. W. Zaki and R. Darmi, “CEFR: Education towards 21st century of learning. Why matters,” Journal of Social Science and Humanities, vol. 4, no. 2, pp. 14–20, 2021, doi: 10.26666/rmp.jssh.2021.2.3.

3. T. Gaillat et al., “Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach,” ReCALL, vol. 34, no. 2, pp. 130–146, 2022, doi: 10.1017/S095834402100029X.

4. I. Natova, “Estimating CEFR reading comprehension text complexity,” The Language Learning Journal, vol. 49, no. 6, pp. 699–710, 2021, doi: 10.1080/09571736.2019.1665088.

5. G. Fakhretdinova, L. M. Zinnatullina, F. T. Galeeva, and E. Valeeva, “The CEFR for languages: research perspectives in foreign language teaching in Engineering university,” in International Conference on Interactive Collaborative Learning, 2021, pp. 225–232, doi: 10.1007/978-3-030-93907-6_24.

6. A. Dmitrieva, A. Laposhina, and M. Lebedeva, “A comparative study of educational texts for native, foreign, and bilingual young speakers of russian: are simplified texts equally simple?,” Frontiers in Psychology, vol. 12, p. 703690, 2021, doi: 10.3389/fpsyg.2021.703690.

7. V. J. Schmalz and A. Brutti, “Automatic Assessment of English CEFR Levels Using BERT Embeddings,” in Computational Linguistics CliC-it 2021, 2022, pp. 293–299.

8. N. S. Lagutina, K. V. Lagutina, A. M. Brederman, and N. N. Kasatkina, “Text classification by CEFR levels using machine learning methods and the BERT language model,” Automatic Control and Computer Sciences, vol. 58, no. 7, pp. 869–878, 2024, doi: 10.3103/S0146411624700329.

9. R. M. S'anchez, D. Alfter, S. Dobnik, M. Szawerna, and E. Volodina, “Jingle BERT, Jingle BERT, Frozen All the Way: Freezing Layers to Identify CEFR Levels of Second Language Learners Using BERT,” in Swedish Language Technology Conference and NLP4CALL, 2024, pp. 137–152, doi: 10.3384/ecp211011.

10. K. Clark, U. Khandelwal, O. Levy, and C. D. Manning, “What Does BERT Look at? An Analysis of BERT’s Attention,” in Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2019, pp. 276–286, doi: 10.18653/v1/W19-4828.

11. N. Khallaf and S. Sharoff, “Automatic Difficulty Classification of Arabic Sentences,” in Proceedings of the Sixth Arabic Natural Language Processing Workshop, 2021, pp. 105–114.

12. D. Kogan et al., “Ace-CEFR--A Dataset for Automated Evaluation of the Linguistic Difficulty of Conversational Texts for LLM Applications.” 2025.

13. H. Ma, J. Wang, and L. He, “Linguistic features distinguishing students’ writing ability aligned with CEFR levels,” Applied Linguistics, vol. 45, no. 4, pp. 637–657, 2024, doi: 10.1093/applin/amad054.

14. V. Franzoni, G. Biondi, A. Milani, and others, “Morpho-Phraseological Based Classification of CEFR Italian L2 Learner Writing Proficiency,” IEEE ACCESS, vol. 12, pp. 156433–156441, 2024, doi: 10.1109/ACCESS.2024.3485988.

15. A. N. Laposhina and M. Y. Lebedeva, “Textometr: an online tool for automated complexity level assessment of texts for Russian language learners,” Russian language studies, vol. 19, no. 3, pp. 331–345, 2021, doi: 10.22363/2618-8163-2021-19-3-331-345.

16. I. Glivsi'c, C. L. Richter, and A. K. Ingason, “Testing relevant linguistic features in automatic CEFR skill level classification for Icelandic,” in Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), 2025, pp. 217–222.

17. N. Lagutina, K. Lagutina, A. Brederman, and N. Kasatkina, “Text classification by CEFR levels using machine learning methods and BERT language model,” Modeling and Analysis of Information Systems, vol. 30, no. 3, pp. 202–213, 2023, doi: 10.18255/1818-1015-2023-3-202-213.

18. P. Qi, Y. Zhang, Y. Zhang, J. Bolton, and C. D. Manning, “Stanza: A Python Natural Language Processing Toolkit for Many Human Languages,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020, pp. 101–108, doi: 10.18653/v1/2020.acl-demos.14.