maisМоделирование и анализ информационных системModeling and Analysis of Information Systems1818-10152313-5417Yaroslavl State University10.18255/1818-1015-2024-2-194-205mais-1854Research ArticleArtificial IntelligenceАвтоматическое определение семантического сходства ответов учащихся с эталонным с помощью современных моделейAutomatic determination of semantic similarity of student answers with the standard one using modern modelshttps://orcid.org/0000-0002-6137-8643ЛагутинаНадежда СтаниславовнаLagutinaNadezhda S.lagutinans@gmail.comhttps://orcid.org/0000-0002-1742-3240ЛагутинаКсения ВладимировнаLagutinaKsenia V.lagutinakv@mail.ruhttps://orcid.org/0009-0007-5451-775XКопнинВладислав НиколаевичKopninVladislav N.vlad.kopnen@mail.ruЯрославский государственный университет им. П.Г. ДемидоваРоссияP.G. Demidov Yaroslavl State UniversityRussian Federation202413062024312194205Copyright © Лагутина Н.С., Лагутина К.В., Копнин В.Н., 20242024Лагутина Н.С., Лагутина К.В., Копнин В.Н.Lagutina N.S., Lagutina K.V., Kopnin V.N.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://www.mais-journal.ru/jour/article/view/1854

В работе представлены результаты исследования современных моделей текста с целью выявления на их основе семантической близости текстов на английском языке. Задача определения семантического сходства текстов является важной составляющей многих областей обработки естественного языка: машинного перевода, поиска информации, систем вопросов и ответов, искусственного интеллекта в образовании. Авторы решали задачу классификации близости ответов учащихся к эталонному ответу учителя. Для исследования были выбраны нейросетевые языковые модели BERT и GPT, ранее применявшиеся к определению семантического сходства текстов, новая нейросетевая модель Mamba, а так же стилометрические характеристики текста. Эксперименты проводились с двумя корпусами текстов: корпус Text Similarity из открытых источников и собственный корпус, собранный с помощью филологов. Качество решения задачи оценивалось точностью, полнотой и F-мерой. Все нейросетевые языковые модели показали близкое качество F-меры около 86% для большего по размеру корпуса Text Similarity и 50–56% для собственного корпуса авторов. Совсем новым результатом оказалось успешное применение модели mamba. Однако, самым интересным достижением стало применение векторов стилометрических характеристик текста, показавшее 80% F-меры для авторского корпуса и одинаковое с нейросетевыми моделями качество решения задачи для другого корпуса.

The paper presents the results of a study of modern text models in order to identify, on their basis, the semantic similarity of English-language texts. The task of determining semantic similarity of texts is an important component of many areas of natural language processing: machine translation, information retrieval, question and answer systems, artificial intelligence in education. The authors solved the problem of classifying the proximity of student answers to the teacher’s standard answer. The neural network language models BERT and GPT, previously used to determine the semantic similarity of texts, the new neural network model Mamba, as well as stylometric features of the text were chosen for the study. Experiments were carried out with two text corpora: the Text Similarity corpus from open sources and the custom corpus, collected with the help of philologists. The quality of the problem solution was assessed by precision, recall, and F-measure. All neural network language models showed a similar F-measure quality of about 86% for the larger Text Similarity corpus and 50–56% for the custom corpus. A completely new result was the successful application of the Mamba model. However, the most interesting achievement was the use of vectors of stylometric features of the text, which showed 80% F-measure for the custom corpus and the same quality of problem solving as neural network models for another corpus.

обработка естественного языкасходство текстовклассификация текстовнейросетевые языковые моделиоценка открытых ответов учащихсяискусственный интеллект в образованииnatural language processingtext similaritytext classificationneural network language modelsassessing students' open responsesartificial intelligence in educationЯрГУ (проект VIP-016).Yaroslavl State University (project VIP-016).ReferencesR. Gao, H. E. Merzdorf, S. Anwar, M. C. Hipwell, and A. Srinivasa, “Automatic assessment of text-based responses in post-secondary education: A systematic review,” Computers and Education: Artificial Intelligence, vol. 6, p. 100206, 2024, doi: 10.1016/j.caeai.2024.100206.R. Gao, H. E. Merzdorf, S. Anwar, M. C. Hipwell, and A. Srinivasa, “Automatic assessment of text-based responses in post-secondary education: A systematic review,” Computers and Education: Artificial Intelligence, vol. 6, p. 100206, 2024, doi: 10.1016/j.caeai.2024.100206.J. Wang and Y. Dong, “Measurement of text similarity: a survey,” Information, vol. 11, no. 9, p. 421, 2020, doi: 10.3390/info11090421.J. Wang and Y. Dong, “Measurement of text similarity: a survey,” Information, vol. 11, no. 9, p. 421, 2020, doi: 10.3390/info11090421.A. Rozeva and S. Zerkova, “Assessing semantic similarity of texts--methods and algorithms,” AIP Conference Proceedings, vol. 1910, no. 1, p. 060012, 2017, doi: 10.1063/1.5014006.A. Rozeva and S. Zerkova, “Assessing semantic similarity of texts--methods and algorithms,” AIP Conference Proceedings, vol. 1910, no. 1, p. 060012, 2017, doi: 10.1063/1.5014006.P. D. Wibisono, A. Asad, and A. Chintan, “Short text similarity measurement methods: a review,” Soft Computing, vol. 25, pp. 4699–4723, 2021, doi: 10.1007/s00500-020-05479-2.P. D. Wibisono, A. Asad, and A. Chintan, “Short text similarity measurement methods: a review,” Soft Computing, vol. 25, pp. 4699–4723, 2021, doi: 10.1007/s00500-020-05479-2.N. S. Lagutina, M. V. Tihomirov, and N. K. Mastakova, “Algoritm avtomaticheskogo postroeniya yazykovogo profilya uchashchegosya,” Zametki po informatike i matematike, no. 15, pp. 58–65, 2023.N. S. Lagutina, M. V. Tihomirov, and N. K. Mastakova, “Algoritm avtomaticheskogo postroeniya yazykovogo profilya uchashchegosya,” Zametki po informatike i matematike, no. 15, pp. 58–65, 2023.O. B. Mishunin, A. P. Savinov, and D. I. Firstov, “Sostoyanie i uroven' razrabotok sistem avtomaticheskoj ocenki svobodnyh otvetov na estestvennom yazyke,” Modern high technologies, no. 1, pp. 38–44, 2016.O. B. Mishunin, A. P. Savinov, and D. I. Firstov, “Sostoyanie i uroven' razrabotok sistem avtomaticheskoj ocenki svobodnyh otvetov na estestvennom yazyke,” Modern high technologies, no. 1, pp. 38–44, 2016.L. Zahrotun, “Comparison Jaccard similarity, cosine similarity and combined both of the data clustering with Shared Nearest Neighbor method,” Computer Engineering and Applications Journal, vol. 5, no. 1, pp. 11–18, 2016, doi: 10.18495/comengapp.v5i1.160.L. Zahrotun, “Comparison Jaccard similarity, cosine similarity and combined both of the data clustering with Shared Nearest Neighbor method,” Computer Engineering and Applications Journal, vol. 5, no. 1, pp. 11–18, 2016, doi: 10.18495/comengapp.v5i1.160.H. A. Abdeljaber, “Automatic Arabic short answers scoring using longest common subsequence and Arabic WordNet,” IEEE Access, vol. 9, pp. 76433–76445, 2021, doi: 10.1109/ACCESS.2021.3082408.H. A. Abdeljaber, “Automatic Arabic short answers scoring using longest common subsequence and Arabic WordNet,” IEEE Access, vol. 9, pp. 76433–76445, 2021, doi: 10.1109/ACCESS.2021.3082408.S. Sultana and I. Biskri, “Identifying similar sentences by using n-grams of characters,” in Recent Trends and Future Technology in Applied Intelligence: Proceedings of 31st International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, 2018, pp. 833–843, doi: 10.1007/978-3-319-92058-0_80.S. Sultana and I. Biskri, “Identifying similar sentences by using n-grams of characters,” in Recent Trends and Future Technology in Applied Intelligence: Proceedings of 31st International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, 2018, pp. 833–843, doi: 10.1007/978-3-319-92058-0_80.S. Vij, D. Tayal, and A. Jain, “A machine learning approach for automated evaluation of short answers using text similarity based on WordNet graphs,” Wireless Personal Communications, vol. 111, pp. 1271–1282, 2020, doi: 10.1007/s11277-019-06913-x.S. Vij, D. Tayal, and A. Jain, “A machine learning approach for automated evaluation of short answers using text similarity based on WordNet graphs,” Wireless Personal Communications, vol. 111, pp. 1271–1282, 2020, doi: 10.1007/s11277-019-06913-x.Y. Zhou, C. Li, G. Huang, Q. Guo, H. Li, and X. Wei, “A Short-Text Similarity Model Combining Semantic and Syntactic Information,” Electronics, vol. 12, no. 14, p. 3126, 2023, doi: 10.3390/electronics12143126.Y. Zhou, C. Li, G. Huang, Q. Guo, H. Li, and X. Wei, “A Short-Text Similarity Model Combining Semantic and Syntactic Information,” Electronics, vol. 12, no. 14, p. 3126, 2023, doi: 10.3390/electronics12143126.M. Mohler and R. Mihalcea, “Text-to-text semantic similarity for automatic short answer grading,” in Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), 2009, pp. 567–575.M. Mohler and R. Mihalcea, “Text-to-text semantic similarity for automatic short answer grading,” in Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), 2009, pp. 567–575.M. Han, X. Zhang, X. Yuan, J. Jiang, W. Yun, and C. Gao, “A survey on the techniques, applications, and performance of short text semantic similarity,” Concurrency and Computation: Practice and Experience, vol. 33, no. 5, p. e5971, 2021, doi: 10.1002/cpe.5971.M. Han, X. Zhang, X. Yuan, J. Jiang, W. Yun, and C. Gao, “A survey on the techniques, applications, and performance of short text semantic similarity,” Concurrency and Computation: Practice and Experience, vol. 33, no. 5, p. e5971, 2021, doi: 10.1002/cpe.5971.S. Roy, S. Dandapat, A. Nagesh, and Y. Narahari, “Wisdom of students: A consistent automatic short answer grading technique,” in Proceedings of the 13th International Conference on Natural Language Processing, 2016, pp. 178–187.S. Roy, S. Dandapat, A. Nagesh, and Y. Narahari, “Wisdom of students: A consistent automatic short answer grading technique,” in Proceedings of the 13th International Conference on Natural Language Processing, 2016, pp. 178–187.A. Ahmed, A. Joorabchi, and M. J. Hayes, “On Deep Learning Approaches to Automated Assessment: Strategies for Short Answer Grading,” in Proceedings of the 14th International Conference on Computer Supported Education, 2022, vol. 2, pp. 85–94, doi: 10.5220/0011082100003182.A. Ahmed, A. Joorabchi, and M. J. Hayes, “On Deep Learning Approaches to Automated Assessment: Strategies for Short Answer Grading,” in Proceedings of the 14th International Conference on Computer Supported Education, 2022, vol. 2, pp. 85–94, doi: 10.5220/0011082100003182.A. Ahmed, A. Joorabchi, and M. J. Hayes, “On the application of sentence transformers to automatic short answer grading in blended assessment,” in Proceedings of the 33rd Irish Signals and Systems Conference (ISSC), 2022, pp. 1–6, doi: 10.1109/ISSC55427.2022.9826194.A. Ahmed, A. Joorabchi, and M. J. Hayes, “On the application of sentence transformers to automatic short answer grading in blended assessment,” in Proceedings of the 33rd Irish Signals and Systems Conference (ISSC), 2022, pp. 1–6, doi: 10.1109/ISSC55427.2022.9826194.L. Camus and A. Filighera, “Investigating transformers for automatic short answer grading,” in Proceedings of the 21st International Conference Artificial Intelligence in Education, Part II 21, 2020, pp. 43–48, doi: 10.1007/978-3-030-52240-7_8.L. Camus and A. Filighera, “Investigating transformers for automatic short answer grading,” in Proceedings of the 21st International Conference Artificial Intelligence in Education, Part II 21, 2020, pp. 43–48, doi: 10.1007/978-3-030-52240-7_8.D. Viji and S. Revathy, “A hybrid approach of Weighted Fine-Tuned BERT extraction with deep Siamese Bi-LSTM model for semantic text similarity identification,” Multimedia Tools and Applications, vol. 81, no. 5, pp. 6131–6157, 2022, doi: 10.1007/s11042-021-11771-6.D. Viji and S. Revathy, “A hybrid approach of Weighted Fine-Tuned BERT extraction with deep Siamese Bi-LSTM model for semantic text similarity identification,” Multimedia Tools and Applications, vol. 81, no. 5, pp. 6131–6157, 2022, doi: 10.1007/s11042-021-11771-6.D. Witschard, I. Jusufi, R. M. Martins, K. Kucher, and A. Kerren, “Interactive optimization of embedding-based text similarity calculations,” Information Visualization, vol. 21, no. 4, pp. 335–353, 2022, doi: 10.1177/14738716221114372.D. Witschard, I. Jusufi, R. M. Martins, K. Kucher, and A. Kerren, “Interactive optimization of embedding-based text similarity calculations,” Information Visualization, vol. 21, no. 4, pp. 335–353, 2022, doi: 10.1177/14738716221114372.T. Brown et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.T. Brown et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.D. Shashavali et al., “Sentence similarity techniques for short vs variable length text using word embeddings,” Computaci'on y Sistemas, vol. 23, no. 3, pp. 999–1004, 2019, doi: 10.13053/cys-23-3-3273.D. Shashavali et al., “Sentence similarity techniques for short vs variable length text using word embeddings,” Computaci'on y Sistemas, vol. 23, no. 3, pp. 999–1004, 2019, doi: 10.13053/cys-23-3-3273.B. Hassan, S. E. Abdelrahman, R. Bahgat, and I. Farag, “UESTS: An unsupervised ensemble semantic textual similarity method,” IEEE Access, vol. 7, pp. 85462–85482, 2019, doi: 10.1109/ACCESS.2019.2925006.B. Hassan, S. E. Abdelrahman, R. Bahgat, and I. Farag, “UESTS: An unsupervised ensemble semantic textual similarity method,” IEEE Access, vol. 7, pp. 85462–85482, 2019, doi: 10.1109/ACCESS.2019.2925006.I. Gagliardi and M. T. Artese, “Ensemble-Based Short Text Similarity: An Easy Approach for Multilingual Datasets Using Transformers and WordNet in Real-World Scenarios,” Big Data and Cognitive Computing, vol. 7, no. 4, p. 158, 2023, doi: 10.3390/bdcc7040158.I. Gagliardi and M. T. Artese, “Ensemble-Based Short Text Similarity: An Easy Approach for Multilingual Datasets Using Transformers and WordNet in Real-World Scenarios,” Big Data and Cognitive Computing, vol. 7, no. 4, p. 158, 2023, doi: 10.3390/bdcc7040158.N. Lagutina, K. Lagutina, A. Brederman, and N. Kasatkina, “Text classification by CEFR levels using machine learning methods and BERT language model,” Modeling and Analysis of Information Systems, vol. 30, no. 3, pp. 202–213, 2023, doi: 10.18255/1818-1015-2023-3-202-213.N. Lagutina, K. Lagutina, A. Brederman, and N. Kasatkina, “Text classification by CEFR levels using machine learning methods and BERT language model,” Modeling and Analysis of Information Systems, vol. 30, no. 3, pp. 202–213, 2023, doi: 10.18255/1818-1015-2023-3-202-213.P. Qi, Y. Zhang, Y. Zhang, J. Bolton, and C. D. Manning, “Stanza: A Python Natural Language Processing Toolkit for Many Human Languages,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020, pp. 101–108, doi: 10.18653/v1/2020.acl-demos.14.P. Qi, Y. Zhang, Y. Zhang, J. Bolton, and C. D. Manning, “Stanza: A Python Natural Language Processing Toolkit for Many Human Languages,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020, pp. 101–108, doi: 10.18653/v1/2020.acl-demos.14.

The authors declare that there are no conflicts of interest present.