Multi-tier linguistic feature engineering for CEFR classification: a comprehensive analysis of deterministic and machine learning-based features

We analyzed 133 linguistic features for automated proficiency classification under the Common European Framework of Reference (CEFR) in a two-tier architecture: deterministic Tier 1 (lexical, morphological, and syntactic measures) and machine-learning-based Tier 2 (semantic coherence, topic structure, cohesion, and error-analysis signals). Experiments were conducted on a corpus of 3,205 learner texts from mixed sources, with triangulated validation against expert-verified Cambridge examination subsets. The materials were collected in 2022--2025 and included a substantial institutional corpus of over 3,000 essays and other writing texts produced by students of Moscow Institute of Physics and Technology (MIPT) studying English as a foreign language and regularly assessed by our AI-powered testing system ISTOK (Intelligent System for Testing General Language Competencies). Feature matrices were standardized after missing-value handling (fold-local median imputation for cross-validation and zero-fill for held-out reporting). In supervised evaluation, the best Tier 1+2 model reaches 66.72% exact accuracy (macro F1 = 0.69) and 94.53% adjacent accuracy (within one CEFR level) on a 3,198-sample CEFR-labeled benchmark split; an extended comparison including preliminary Tier 3 features achieves 67.50%. Unsupervised analyses show strong structure for extreme levels (A1 99.5% purity; C2 82.4% purity) and moderate alignment with professional Cambridge labels (Adjusted Rand Index = 0.303). We report block ablations and compact subset searches, with strongest signals from morphological complexity and lexical sophistication, and consistent incremental gains from error-based features. The results provide a validated, interpretable feature inventory and practical guidance for feature selection in automated language assessment systems.

1. Council of Europe, “Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEFR).” 2026, Accessed: Feb. 27, 2026. [Online]. Available: https://www.coe.int/en/web/common-european-framework-reference-languages.

2. J. M. Imperial et al., “UniversalCEFR: Enabling Open Multilingual Research on Language Proficiency Assessment,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, 2025, pp. 9714–9766, doi: 10.18653/v1/2025.emnlp-main.491.

3. K. North and M. Zampieri, “Features of lexical complexity: Insights from L1 and L2 speakers,” Frontiers in Artificial Intelligence, vol. 6, 2023, doi: 10.3389/frai.2023.1236963.

4. T. Gaillat et al., “Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach,” ReCALL, vol. 34, no. 2, pp. 130–146, 2022, doi: 10.1017/S095834402100029X.

5. C. Wood, M. Garcia-Salas, C. Schatschneider, and M. Torres-Chavarro, “Morphological complexity in writing: Implications for writing quality and patterns of change,” American Journal of Speech-Language Pathology, vol. 33, no. 3, pp. 1432–1442, 2024, doi: 10.1044/2024_AJSLP-23-00265.

6. A. Riemenschneider, Z. Weiss, P. Schr"oter, and D. Meurers, “The Interplay of Task Characteristics, Linguistic Complexity, and Language Proficiency in High-Stakes English as a Foreign Language Writing,” TESOL Quarterly, vol. 58, no. 2, pp. 775–801, 2024, doi: 10.1002/tesq.3254.

7. P. Xu, “Reconsidering the syntactic complexity measures on L2 spoken English: A multi-dimensional perspective,” Heliyon, vol. 9, no. 6, p. e16856, 2023, doi: 10.1016/j.heliyon.2023.e16856.

8. A. Alzahrani, “Utility of Kolmogorov complexity measures: Analysis of L2 groups and L1 backgrounds,” PLOS ONE, vol. 19, no. 4, pp. 1–25, 2024, doi: 10.1371/journal.pone.0301806.

9. M. Abdi Tabari, M. D. Johnson, and J. Gao, “Using automated indices of cohesion to explore the growth of cohesive features in L2 writing,” International Review of Applied Linguistics in Language Teaching, vol. 63, no. 3, pp. 2169–2200, 2025, doi: doi:10.1515/iral-2023-0185.

10. X. Tang, H. Chen, D. Lin, and K. Li, “Incorporating Fine-Grained Linguistic Features and Explainable AI into Multi-Dimensional Automated Writing Assessment,” Applied Sciences, vol. 14, no. 10, p. 4182, 2024, doi: 10.3390/app14104182.

11. J. F. Lohmann et al., “Neural Networks or Linguistic Features? - Comparing Different Machine-Learning Approaches for Automated Assessment of Text Quality Traits Among L1- and L2-Learners’ Argumentative Essays,” International Journal of Artificial Intelligence in Education, vol. 35, no. 3, pp. 1178–1217, 2025, doi: 10.1007/s40593-024-00426-w.

12. M. Faseeh et al., “Hybrid Approach to Automated Essay Scoring: Integrating Deep Learning Embeddings with Handcrafted Linguistic Features for Improved Accuracy,” Mathematics, vol. 12, no. 21, p. 3416, 2024, doi: 10.3390/math12213416.

13. E. Kerz, D. Wiechmann, Y. Qiao, E. Tseng, and M. Str"obel, “Automated classification of written proficiency levels on the CEFR-scale through complexity contours and RNNs,” in Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, 2021, pp. 199–209.

14. N. Lagutina, K. Lagutina, A. Brederman, and N. Kasatkina, “Text classification by CEFR levels using machine learning methods and the BERT language model,” Automatic Control and Computer Sciences, vol. 58, no. 7, pp. 869–878, 2024, doi: 10.3103/S0146411624700329.

15. A. Y. Uluslu, “Exploring Hybrid Linguistic Features for Turkish Text Readability,” in Proceedings of the 6th International Conference on Natural Language and Speech Processing ICNLSP 2023, Virtual Event, 16-17 December 2023, 2023, pp. 223–232.

16. S. Khan and others, “Analyzing Lexical Complexity in Learner Corpora: A Corpus-Driven Approach Using Part-of-Speech Tagging and Dependency Parsing,” Contemporary Journal of Social Science Review, vol. 3, no. 4, pp. 1143–1170, 2025, doi: 10.63878/cjssr.v3i4.1556.

17. E. M. Bazanova, A. V. Gorizontova, N. N. Gribova, T. M. Chikake, and A. V. Samosyuk, “Development and Prospects of National Intelligent System for Testing General Language Competencies Deployed Through Neural Network Solutions,” Higher Education in Russia, vol. 32, no. 8--9, pp. 147–166, 2023, doi: 10.31992/0869-3617-2023-32-8-9-147-166.

18. K. Doi, K. Sudoh, S. Nakamura, and T. Watanabe, “Enhancing Automated Essay Scoring with Grammatical Features using Multi-task Learning and Item Response Theory,” Journal of Natural Language Processing, vol. 32, no. 2, pp. 438–479, 2025, doi: 10.5715/jnlp.32.438.

19. Y. Bestgen, “Estimating lexical diversity using the moving average type-token ratio (MATTR): Pros and cons,” Research Methods in Applied Linguistics, vol. 4, no. 1, p. 100168, 2025, doi: 10.1016/j.rmal.2024.100168.

20. K. North, M. Zampieri, and M. Shardlow, “Lexical complexity prediction: An overview,” ACM Computing Surveys, vol. 55, no. 9, pp. 1–42, 2023, doi: 10.1145/3557885.

21. M. Burtsev et al., “Deeppavlov: Open-source library for dialogue systems,” in Proceedings of ACL 2018, System Demonstrations, 2018, pp. 122–127.

22. D. Jurafsky and J. H. Martin, Part-of-Speech Tagging, 3Rd edition. Stanford University, 2026.