maisМоделирование и анализ информационных системModeling and Analysis of Information Systems1818-10152313-5417Yaroslavl State University10.18255/1818-1015-2026-1-6-29mais-2000Research ArticleArtificial IntelligenceМногоуровневое лингвистическое конструирование признаков для классификации CEFR: комплексный анализ детерминированных и основанных на машинном обучении признаковMulti-tier linguistic feature engineering for CEFR classification: a comprehensive analysis of deterministic and machine learning-based featureshttps://orcid.org/0000-0002-9512-1256ЧикакеТендай МапунгванаChikakeTendai M.tendaichikake@phystech.eduhttps://orcid.org/0000-0002-6306-8892БазановаЕлена МихайловнаBazanovaElena M.bazanova.em@mipt.ruhttps://orcid.org/0009-0007-4151-3338ГоризонтоваАнна ВсеволодовнаGorizontovaAnna V.gorizontova.av@mipt.ruМосковский физико-технический институтРоссияMoscow Institute of Physics and TechnologyRussian Federation202616032026331629Copyright © Чикаке Т.М., Базанова Е.М., Горизонтова А.В., 20262026Чикаке Т.М., Базанова Е.М., Горизонтова А.В.Chikake T.M., Bazanova E.M., Gorizontova A.V.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://www.mais-journal.ru/jour/article/view/2000

Представлен анализ 133 лингвистических признаков для автоматической классификации уровня владения языком по шкале Common European Framework of Reference (CEFR) в двухуровневой архитектуре: детерминированные признаки Tier 1 (лексические, морфологические и синтаксические показатели) и признаки Tier 2, основанные на методах машинного обучения (семантическая связность, тематическая структура, когезия и сигналы ошибок). Эксперименты выполнены на корпусе из 3,205 текстов обучающихся из разнородных источников; валидация проводилась с триангуляцией по экспертно верифицированным подмножествам экзаменационных данных Cambridge. Материалы были собраны в 2022--2025 годах и включают существенный институциональный корпус из более чем 3,000 эссе и других письменных работ студентов Московского физико-технического института (МФТИ), изучающих английский язык как иностранный; уровень владения языком у них регулярно оценивается нашей интеллектуальной системой тестирования ISTOK (Intelligent System for Testing General Language Competencies). Признаковые матрицы стандартизировались после обработки пропусков (медианная импутация внутри фолда для кросс-валидации и заполнение нулями для отчётных экспериментов на отложенной выборке). В задаче контролируемой классификации лучшая модель Tier 1+2 достигает 66.72% точности (макро F1 = 0.69) и 94.53% смежной точности (ошибка не более чем на один уровень CEFR) на эталонном разбиении для 3,198 текстов с метками CEFR; расширенное сравнение с предварительными признаками Tier 3 достигает 67.50%. Неконтролируемый анализ выявляет структуру пространства признаков: для крайних уровней наблюдаются кластеры высокой чистоты (A1 99.5%; C2 82.4%), а на экспертных данных Cambridge получено умеренное согласование с профессиональными оценками (Adjusted Rand Index = 0.303). Приведены результаты блочных абляций и поиска компактных подмножеств; наибольшую информативность обеспечивают морфологическая сложность и лексическая продвинутость, а признаки ошибок дают устойчивый дополнительный выигрыш.

We analyzed 133 linguistic features for automated proficiency classification under the Common European Framework of Reference (CEFR) in a two-tier architecture: deterministic Tier 1 (lexical, morphological, and syntactic measures) and machine-learning-based Tier 2 (semantic coherence, topic structure, cohesion, and error-analysis signals). Experiments were conducted on a corpus of 3,205 learner texts from mixed sources, with triangulated validation against expert-verified Cambridge examination subsets. The materials were collected in 2022--2025 and included a substantial institutional corpus of over 3,000 essays and other writing texts produced by students of Moscow Institute of Physics and Technology (MIPT) studying English as a foreign language and regularly assessed by our AI-powered testing system ISTOK (Intelligent System for Testing General Language Competencies). Feature matrices were standardized after missing-value handling (fold-local median imputation for cross-validation and zero-fill for held-out reporting). In supervised evaluation, the best Tier 1+2 model reaches 66.72% exact accuracy (macro F1 = 0.69) and 94.53% adjacent accuracy (within one CEFR level) on a 3,198-sample CEFR-labeled benchmark split; an extended comparison including preliminary Tier 3 features achieves 67.50%. Unsupervised analyses show strong structure for extreme levels (A1 99.5% purity; C2 82.4% purity) and moderate alignment with professional Cambridge labels (Adjusted Rand Index = 0.303). We report block ablations and compact subset searches, with strongest signals from morphological complexity and lexical sophistication, and consistent incremental gains from error-based features. The results provide a validated, interpretable feature inventory and practical guidance for feature selection in automated language assessment systems.

классификация CEFRлингвистические признакимногоуровневая архитектураавтоматизированная оценкаинженерия признаковобработка естественного языкаCEFR classificationlinguistic featuresmulti-tier architectureautomated assessmentfeature engineeringnatural language processingМинистерство науки и высшего образования Российской Федерации (соглашение № 075-03-2026-305 от 16 января 2026 г., проект «Прикладные исследования по внедрению технологий искусственного интеллекта в высшем образовании», код проекта: ФСМГ-2025-0086)Ministry of Science and Higher Education of the Russian Federation (Agreement No. 075-03-2026-305, January 16, 2026, project “Applied Research on the Implementation of Artificial Intelligence Technologies in Higher Education”, project code: FSMG- 2025-0086)ReferencesCouncil of Europe, “Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEFR).” 2026, Accessed: Feb. 27, 2026. [Online]. Available: https://www.coe.int/en/web/common-european-framework-reference-languages.Council of Europe, “Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEFR).” 2026, Accessed: Feb. 27, 2026. [Online]. Available: https://www.coe.int/en/web/common-european-framework-reference-languages.J. M. Imperial et al., “UniversalCEFR: Enabling Open Multilingual Research on Language Proficiency Assessment,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, 2025, pp. 9714–9766, doi: 10.18653/v1/2025.emnlp-main.491.J. M. Imperial et al., “UniversalCEFR: Enabling Open Multilingual Research on Language Proficiency Assessment,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, 2025, pp. 9714–9766, doi: 10.18653/v1/2025.emnlp-main.491.K. North and M. Zampieri, “Features of lexical complexity: Insights from L1 and L2 speakers,” Frontiers in Artificial Intelligence, vol. 6, 2023, doi: 10.3389/frai.2023.1236963.K. North and M. Zampieri, “Features of lexical complexity: Insights from L1 and L2 speakers,” Frontiers in Artificial Intelligence, vol. 6, 2023, doi: 10.3389/frai.2023.1236963.T. Gaillat et al., “Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach,” ReCALL, vol. 34, no. 2, pp. 130–146, 2022, doi: 10.1017/S095834402100029X.T. Gaillat et al., “Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach,” ReCALL, vol. 34, no. 2, pp. 130–146, 2022, doi: 10.1017/S095834402100029X.C. Wood, M. Garcia-Salas, C. Schatschneider, and M. Torres-Chavarro, “Morphological complexity in writing: Implications for writing quality and patterns of change,” American Journal of Speech-Language Pathology, vol. 33, no. 3, pp. 1432–1442, 2024, doi: 10.1044/2024_AJSLP-23-00265.C. Wood, M. Garcia-Salas, C. Schatschneider, and M. Torres-Chavarro, “Morphological complexity in writing: Implications for writing quality and patterns of change,” American Journal of Speech-Language Pathology, vol. 33, no. 3, pp. 1432–1442, 2024, doi: 10.1044/2024_AJSLP-23-00265.A. Riemenschneider, Z. Weiss, P. Schr"oter, and D. Meurers, “The Interplay of Task Characteristics, Linguistic Complexity, and Language Proficiency in High-Stakes English as a Foreign Language Writing,” TESOL Quarterly, vol. 58, no. 2, pp. 775–801, 2024, doi: 10.1002/tesq.3254.A. Riemenschneider, Z. Weiss, P. Schr"oter, and D. Meurers, “The Interplay of Task Characteristics, Linguistic Complexity, and Language Proficiency in High-Stakes English as a Foreign Language Writing,” TESOL Quarterly, vol. 58, no. 2, pp. 775–801, 2024, doi: 10.1002/tesq.3254.P. Xu, “Reconsidering the syntactic complexity measures on L2 spoken English: A multi-dimensional perspective,” Heliyon, vol. 9, no. 6, p. e16856, 2023, doi: 10.1016/j.heliyon.2023.e16856.P. Xu, “Reconsidering the syntactic complexity measures on L2 spoken English: A multi-dimensional perspective,” Heliyon, vol. 9, no. 6, p. e16856, 2023, doi: 10.1016/j.heliyon.2023.e16856.A. Alzahrani, “Utility of Kolmogorov complexity measures: Analysis of L2 groups and L1 backgrounds,” PLOS ONE, vol. 19, no. 4, pp. 1–25, 2024, doi: 10.1371/journal.pone.0301806.A. Alzahrani, “Utility of Kolmogorov complexity measures: Analysis of L2 groups and L1 backgrounds,” PLOS ONE, vol. 19, no. 4, pp. 1–25, 2024, doi: 10.1371/journal.pone.0301806.M. Abdi Tabari, M. D. Johnson, and J. Gao, “Using automated indices of cohesion to explore the growth of cohesive features in L2 writing,” International Review of Applied Linguistics in Language Teaching, vol. 63, no. 3, pp. 2169–2200, 2025, doi: doi:10.1515/iral-2023-0185.M. Abdi Tabari, M. D. Johnson, and J. Gao, “Using automated indices of cohesion to explore the growth of cohesive features in L2 writing,” International Review of Applied Linguistics in Language Teaching, vol. 63, no. 3, pp. 2169–2200, 2025, doi: doi:10.1515/iral-2023-0185.X. Tang, H. Chen, D. Lin, and K. Li, “Incorporating Fine-Grained Linguistic Features and Explainable AI into Multi-Dimensional Automated Writing Assessment,” Applied Sciences, vol. 14, no. 10, p. 4182, 2024, doi: 10.3390/app14104182.X. Tang, H. Chen, D. Lin, and K. Li, “Incorporating Fine-Grained Linguistic Features and Explainable AI into Multi-Dimensional Automated Writing Assessment,” Applied Sciences, vol. 14, no. 10, p. 4182, 2024, doi: 10.3390/app14104182.J. F. Lohmann et al., “Neural Networks or Linguistic Features? - Comparing Different Machine-Learning Approaches for Automated Assessment of Text Quality Traits Among L1- and L2-Learners’ Argumentative Essays,” International Journal of Artificial Intelligence in Education, vol. 35, no. 3, pp. 1178–1217, 2025, doi: 10.1007/s40593-024-00426-w.J. F. Lohmann et al., “Neural Networks or Linguistic Features? - Comparing Different Machine-Learning Approaches for Automated Assessment of Text Quality Traits Among L1- and L2-Learners’ Argumentative Essays,” International Journal of Artificial Intelligence in Education, vol. 35, no. 3, pp. 1178–1217, 2025, doi: 10.1007/s40593-024-00426-w.M. Faseeh et al., “Hybrid Approach to Automated Essay Scoring: Integrating Deep Learning Embeddings with Handcrafted Linguistic Features for Improved Accuracy,” Mathematics, vol. 12, no. 21, p. 3416, 2024, doi: 10.3390/math12213416.M. Faseeh et al., “Hybrid Approach to Automated Essay Scoring: Integrating Deep Learning Embeddings with Handcrafted Linguistic Features for Improved Accuracy,” Mathematics, vol. 12, no. 21, p. 3416, 2024, doi: 10.3390/math12213416.E. Kerz, D. Wiechmann, Y. Qiao, E. Tseng, and M. Str"obel, “Automated classification of written proficiency levels on the CEFR-scale through complexity contours and RNNs,” in Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, 2021, pp. 199–209.E. Kerz, D. Wiechmann, Y. Qiao, E. Tseng, and M. Str"obel, “Automated classification of written proficiency levels on the CEFR-scale through complexity contours and RNNs,” in Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, 2021, pp. 199–209.N. Lagutina, K. Lagutina, A. Brederman, and N. Kasatkina, “Text classification by CEFR levels using machine learning methods and the BERT language model,” Automatic Control and Computer Sciences, vol. 58, no. 7, pp. 869–878, 2024, doi: 10.3103/S0146411624700329.N. Lagutina, K. Lagutina, A. Brederman, and N. Kasatkina, “Text classification by CEFR levels using machine learning methods and the BERT language model,” Automatic Control and Computer Sciences, vol. 58, no. 7, pp. 869–878, 2024, doi: 10.3103/S0146411624700329.A. Y. Uluslu, “Exploring Hybrid Linguistic Features for Turkish Text Readability,” in Proceedings of the 6th International Conference on Natural Language and Speech Processing ICNLSP 2023, Virtual Event, 16-17 December 2023, 2023, pp. 223–232.A. Y. Uluslu, “Exploring Hybrid Linguistic Features for Turkish Text Readability,” in Proceedings of the 6th International Conference on Natural Language and Speech Processing ICNLSP 2023, Virtual Event, 16-17 December 2023, 2023, pp. 223–232.S. Khan and others, “Analyzing Lexical Complexity in Learner Corpora: A Corpus-Driven Approach Using Part-of-Speech Tagging and Dependency Parsing,” Contemporary Journal of Social Science Review, vol. 3, no. 4, pp. 1143–1170, 2025, doi: 10.63878/cjssr.v3i4.1556.S. Khan and others, “Analyzing Lexical Complexity in Learner Corpora: A Corpus-Driven Approach Using Part-of-Speech Tagging and Dependency Parsing,” Contemporary Journal of Social Science Review, vol. 3, no. 4, pp. 1143–1170, 2025, doi: 10.63878/cjssr.v3i4.1556.E. M. Bazanova, A. V. Gorizontova, N. N. Gribova, T. M. Chikake, and A. V. Samosyuk, “Development and Prospects of National Intelligent System for Testing General Language Competencies Deployed Through Neural Network Solutions,” Higher Education in Russia, vol. 32, no. 8--9, pp. 147–166, 2023, doi: 10.31992/0869-3617-2023-32-8-9-147-166.E. M. Bazanova, A. V. Gorizontova, N. N. Gribova, T. M. Chikake, and A. V. Samosyuk, “Development and Prospects of National Intelligent System for Testing General Language Competencies Deployed Through Neural Network Solutions,” Higher Education in Russia, vol. 32, no. 8--9, pp. 147–166, 2023, doi: 10.31992/0869-3617-2023-32-8-9-147-166.K. Doi, K. Sudoh, S. Nakamura, and T. Watanabe, “Enhancing Automated Essay Scoring with Grammatical Features using Multi-task Learning and Item Response Theory,” Journal of Natural Language Processing, vol. 32, no. 2, pp. 438–479, 2025, doi: 10.5715/jnlp.32.438.K. Doi, K. Sudoh, S. Nakamura, and T. Watanabe, “Enhancing Automated Essay Scoring with Grammatical Features using Multi-task Learning and Item Response Theory,” Journal of Natural Language Processing, vol. 32, no. 2, pp. 438–479, 2025, doi: 10.5715/jnlp.32.438.Y. Bestgen, “Estimating lexical diversity using the moving average type-token ratio (MATTR): Pros and cons,” Research Methods in Applied Linguistics, vol. 4, no. 1, p. 100168, 2025, doi: 10.1016/j.rmal.2024.100168.Y. Bestgen, “Estimating lexical diversity using the moving average type-token ratio (MATTR): Pros and cons,” Research Methods in Applied Linguistics, vol. 4, no. 1, p. 100168, 2025, doi: 10.1016/j.rmal.2024.100168.K. North, M. Zampieri, and M. Shardlow, “Lexical complexity prediction: An overview,” ACM Computing Surveys, vol. 55, no. 9, pp. 1–42, 2023, doi: 10.1145/3557885.K. North, M. Zampieri, and M. Shardlow, “Lexical complexity prediction: An overview,” ACM Computing Surveys, vol. 55, no. 9, pp. 1–42, 2023, doi: 10.1145/3557885.M. Burtsev et al., “Deeppavlov: Open-source library for dialogue systems,” in Proceedings of ACL 2018, System Demonstrations, 2018, pp. 122–127.M. Burtsev et al., “Deeppavlov: Open-source library for dialogue systems,” in Proceedings of ACL 2018, System Demonstrations, 2018, pp. 122–127.D. Jurafsky and J. H. Martin, Part-of-Speech Tagging, 3Rd edition. Stanford University, 2026.D. Jurafsky and J. H. Martin, Part-of-Speech Tagging, 3Rd edition. Stanford University, 2026.

The authors declare that there are no conflicts of interest present.