The impact of the size of training sets on quality of automatic short answers grading

The paper investigates the impact of training set size on the quality of automatic short answers grading, formulated as a classification task. The impact was evaluated using a method based on measuring the similarity between the assessed answer and a given reference answer, calculated via embedding vectors, in combination with a logistic regression classifier. Experiments were conducted on corpora of answers to questions in computer science, history, and software development using Qt framework. The sizes of the corpora were 547, 522, and 931 answers, respectively. Two experiments were conducted during the study. In the first experiment, the change in classification quality was assessed as the training set size was reduced. It showed that when the binary classification is utilized (an answer can be either correct or incorrect), reducing the size of the training set leads to a smaller decline in quality compared to ternary classification (which includes a class of partially correct answers). In the second experiment, the possibility of improving classification quality by expanding small‑sized training sets through data augmentation was investigated. It demonstrated that augmentation performed using the DeepSeek generative model can significantly improve results in several cases, which is important for practical applications under data scarcity conditions. Additionally, the experiments revealed that when different language models are used to generate embeddings, the magnitude of change in classification quality with varying training set sizes can differ significantly. Specifically, using certain models — such as rubert‑tiny2 and MiniLM‑L12‑v2 — to produce embeddings yields more stable results than using other models.

1. R. Gao, H. E. Merzdorf, S. Anwar, M. C. Hipwell, and A. R. Srinivasa, “Automatic assessment of text-based responses in post-secondary education: A systematic review,” Computers and Education: Artificial Intelligence, vol. 6, p. 100206, 2024, doi: 10.1016/j.caeai.2024.100206.

2. N. S. Lagutina and K. V. Lagutina, “A Survey of Models for Automatic Assessment of Similarity of Student’s Answer to the Reference Answer,” Automatic Control and Computer Sciences, vol. 59, no. 7, pp. 1152–1169, 2025, doi: 10.3103/S0146411625700427.

3. S. Kumar, S. Chakrabarti, and S. Roy, “Earth mover's distance pooling over Siamese LSTMs for automatic short answer grading,” in Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, 2017, pp. 2046–2052, doi: 10.24963/ijcai.2017/284.

4. T. H. Nguyen, H. H. Nguyen, Z. Ahmadi, T.-A. Hoang, and T.-N. Doan, “On the impact of dataset size: A Twitter classification case study,” in IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, 2021, pp. 210–217, doi: 10.1145/3486622.3493960.

5. A. R. Nair, R. P. Singh, D. Gupta, and P. Kumar, “Evaluating the impact of text data augmentation on text classification tasks using DistilBERT,” Procedia Computer Science, vol. 235, pp. 102–111, 2024, doi: 10.1016/j.procs.2024.04.013.

6. C. Shorten, T. M. Khoshgoftaar, and B. Furht, “Text data augmentation for deep learning,” Journal of Big Data, vol. 8, p. 101, 2021, doi: 10.1186/s40537-021-00492-0.

7. A. Poulton and S. Eliens, “Explaining transformer-based models for automatic short answer grading,” in Proceedings of the 5th International Conference on Digital Technology in Education, 2021, pp. 110–116, doi: 10.1145/3488466.3488479.

8. G. Kortemeyer, “Performance of the pre-trained large language model GPT-4 on automated short answer grading,” Discover Artificial Intelligence, vol. 4, no. 1, p. 47, 2024, doi: 10.1007/s44163-024-00147-y.

9. C. B. Minnegalieva, I. I. Kashapov, and O. D. Morozova, “Automated Grading of Students’ Short Answers Using Language Models,” Automatic Documentation and Mathematical Linguistics, vol. 58, no. Suppl 3, pp. S109–S114, 2024, doi: 10.3103/S0005105525700177.

10. H. Tan, C. Wang, Q. Duan, Y. Lu, H. Zhang, and R. Li, “Automatic short answer grading by encoding student responses via a graph convolutional network,” Interactive Learning Environments, vol. 31, no. 3, pp. 1636–1650, 2023, doi: 10.1080/10494820.2020.1855207.

11. J. Lun, J. Zhu, Y. Tang, and M. Yang, “Multiple data augmentation strategies for improving performance on automatic short answer scoring,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2020, vol. 34, no. 09, pp. 13389–13396, doi: 10.1609/aaai.v34i09.7062.

12. M. C. Wijanto and H.-S. Yong, “Combining balancing dataset and SentenceTransformers to improve short answer grading performance,” Applied Sciences, vol. 14, no. 11, p. 4532, 2024, doi: 10.3390/app14114532.

13. S. Zhong, J. Zeng, Y. Yu, and B. Lin, “Clustering Algorithms and RAG Enhancing Semi-Supervised Text Classification with Large LLMs,” International Journal of Data Science and Analytics, vol. 20, pp. 5377–5398, 2025, doi: 10.1007/s41060-025-00774-3.

14. R. Li et al., “Retrieval-augmented meta learning for low-resource text classification,” in Proceedings of the International Joint Conference on Neural Networks, 2024, pp. 1–8, doi: 10.1109/IJCNN60899.2024.10651119.

15. Y. Yu, Y. Zhuang, R. Zhang, Y. Meng, J. Shen, and C. Zhang, “ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval,” in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 11782–11805, doi: 10.18653/v1/2023.findings-acl.748.

16. H. Dai et al., “AugGPT: Leveraging ChatGPT for Text Data Augmentation,” IEEE Transactions on Big Data, vol. 11, pp. 907–918, 2025, doi: 10.1109/TBDATA.2025.3536934.

17. V. N. Kopnin and N. S. Lagutina, “Analysis of the Application of Large Language Models for Comparing Open Student Responses with Reference Answers,” in Matematicheskoe i Informacionnoe Modelirovanie: Materialy Vserossijskoj Konferencii Molodyh Uchenyh, 2025, vol. 23.

18. A. Liu et al., “DeepSeek-V3 Technical Report.” 2025, doi: 10.48550/arXiv.2412.19437v2.