Автоматическое определение семантического сходства ответов учащихся с эталонным с помощью современных моделей

В работе представлены результаты исследования современных моделей текста с целью выявления на их основе семантической близости текстов на английском языке. Задача определения семантического сходства текстов является важной составляющей многих областей обработки естественного языка: машинного перевода, поиска информации, систем вопросов и ответов, искусственного интеллекта в образовании. Авторы решали задачу классификации близости ответов учащихся к эталонному ответу учителя. Для исследования были выбраны нейросетевые языковые модели BERT и GPT, ранее применявшиеся к определению семантического сходства текстов, новая нейросетевая модель Mamba, а так же стилометрические характеристики текста. Эксперименты проводились с двумя корпусами текстов: корпус Text Similarity из открытых источников и собственный корпус, собранный с помощью филологов. Качество решения задачи оценивалось точностью, полнотой и F-мерой. Все нейросетевые языковые модели показали близкое качество F-меры около 86% для большего по размеру корпуса Text Similarity и 50–56% для собственного корпуса авторов. Совсем новым результатом оказалось успешное применение модели mamba. Однако, самым интересным достижением стало применение векторов стилометрических характеристик текста, показавшее 80% F-меры для авторского корпуса и одинаковое с нейросетевыми моделями качество решения задачи для другого корпуса.

1. R. Gao, H. E. Merzdorf, S. Anwar, M. C. Hipwell, and A. Srinivasa, “Automatic assessment of text-based responses in post-secondary education: A systematic review,” Computers and Education: Artificial Intelligence, vol. 6, p. 100206, 2024, doi: 10.1016/j.caeai.2024.100206.

2. J. Wang and Y. Dong, “Measurement of text similarity: a survey,” Information, vol. 11, no. 9, p. 421, 2020, doi: 10.3390/info11090421.

3. A. Rozeva and S. Zerkova, “Assessing semantic similarity of texts--methods and algorithms,” AIP Conference Proceedings, vol. 1910, no. 1, p. 060012, 2017, doi: 10.1063/1.5014006.

4. P. D. Wibisono, A. Asad, and A. Chintan, “Short text similarity measurement methods: a review,” Soft Computing, vol. 25, pp. 4699–4723, 2021, doi: 10.1007/s00500-020-05479-2.

5. N. S. Lagutina, M. V. Tihomirov, and N. K. Mastakova, “Algoritm avtomaticheskogo postroeniya yazykovogo profilya uchashchegosya,” Zametki po informatike i matematike, no. 15, pp. 58–65, 2023.

6. O. B. Mishunin, A. P. Savinov, and D. I. Firstov, “Sostoyanie i uroven' razrabotok sistem avtomaticheskoj ocenki svobodnyh otvetov na estestvennom yazyke,” Modern high technologies, no. 1, pp. 38–44, 2016.

7. L. Zahrotun, “Comparison Jaccard similarity, cosine similarity and combined both of the data clustering with Shared Nearest Neighbor method,” Computer Engineering and Applications Journal, vol. 5, no. 1, pp. 11–18, 2016, doi: 10.18495/comengapp.v5i1.160.

8. H. A. Abdeljaber, “Automatic Arabic short answers scoring using longest common subsequence and Arabic WordNet,” IEEE Access, vol. 9, pp. 76433–76445, 2021, doi: 10.1109/ACCESS.2021.3082408.

9. S. Sultana and I. Biskri, “Identifying similar sentences by using n-grams of characters,” in Recent Trends and Future Technology in Applied Intelligence: Proceedings of 31st International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, 2018, pp. 833–843, doi: 10.1007/978-3-319-92058-0_80.

10. S. Vij, D. Tayal, and A. Jain, “A machine learning approach for automated evaluation of short answers using text similarity based on WordNet graphs,” Wireless Personal Communications, vol. 111, pp. 1271–1282, 2020, doi: 10.1007/s11277-019-06913-x.

11. Y. Zhou, C. Li, G. Huang, Q. Guo, H. Li, and X. Wei, “A Short-Text Similarity Model Combining Semantic and Syntactic Information,” Electronics, vol. 12, no. 14, p. 3126, 2023, doi: 10.3390/electronics12143126.

12. M. Mohler and R. Mihalcea, “Text-to-text semantic similarity for automatic short answer grading,” in Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), 2009, pp. 567–575.

13. M. Han, X. Zhang, X. Yuan, J. Jiang, W. Yun, and C. Gao, “A survey on the techniques, applications, and performance of short text semantic similarity,” Concurrency and Computation: Practice and Experience, vol. 33, no. 5, p. e5971, 2021, doi: 10.1002/cpe.5971.

14. S. Roy, S. Dandapat, A. Nagesh, and Y. Narahari, “Wisdom of students: A consistent automatic short answer grading technique,” in Proceedings of the 13th International Conference on Natural Language Processing, 2016, pp. 178–187.

15. A. Ahmed, A. Joorabchi, and M. J. Hayes, “On Deep Learning Approaches to Automated Assessment: Strategies for Short Answer Grading,” in Proceedings of the 14th International Conference on Computer Supported Education, 2022, vol. 2, pp. 85–94, doi: 10.5220/0011082100003182.

16. A. Ahmed, A. Joorabchi, and M. J. Hayes, “On the application of sentence transformers to automatic short answer grading in blended assessment,” in Proceedings of the 33rd Irish Signals and Systems Conference (ISSC), 2022, pp. 1–6, doi: 10.1109/ISSC55427.2022.9826194.

17. L. Camus and A. Filighera, “Investigating transformers for automatic short answer grading,” in Proceedings of the 21st International Conference Artificial Intelligence in Education, Part II 21, 2020, pp. 43–48, doi: 10.1007/978-3-030-52240-7_8.

18. D. Viji and S. Revathy, “A hybrid approach of Weighted Fine-Tuned BERT extraction with deep Siamese Bi-LSTM model for semantic text similarity identification,” Multimedia Tools and Applications, vol. 81, no. 5, pp. 6131–6157, 2022, doi: 10.1007/s11042-021-11771-6.

19. D. Witschard, I. Jusufi, R. M. Martins, K. Kucher, and A. Kerren, “Interactive optimization of embedding-based text similarity calculations,” Information Visualization, vol. 21, no. 4, pp. 335–353, 2022, doi: 10.1177/14738716221114372.

20. T. Brown et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.

21. D. Shashavali et al., “Sentence similarity techniques for short vs variable length text using word embeddings,” Computaci'on y Sistemas, vol. 23, no. 3, pp. 999–1004, 2019, doi: 10.13053/cys-23-3-3273.

22. B. Hassan, S. E. Abdelrahman, R. Bahgat, and I. Farag, “UESTS: An unsupervised ensemble semantic textual similarity method,” IEEE Access, vol. 7, pp. 85462–85482, 2019, doi: 10.1109/ACCESS.2019.2925006.

23. I. Gagliardi and M. T. Artese, “Ensemble-Based Short Text Similarity: An Easy Approach for Multilingual Datasets Using Transformers and WordNet in Real-World Scenarios,” Big Data and Cognitive Computing, vol. 7, no. 4, p. 158, 2023, doi: 10.3390/bdcc7040158.

24. N. Lagutina, K. Lagutina, A. Brederman, and N. Kasatkina, “Text classification by CEFR levels using machine learning methods and BERT language model,” Modeling and Analysis of Information Systems, vol. 30, no. 3, pp. 202–213, 2023, doi: 10.18255/1818-1015-2023-3-202-213.

25. P. Qi, Y. Zhang, Y. Zhang, J. Bolton, and C. D. Manning, “Stanza: A Python Natural Language Processing Toolkit for Many Human Languages,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020, pp. 101–108, doi: 10.18653/v1/2020.acl-demos.14.