maisМоделирование и анализ информационных системModeling and Analysis of Information Systems1818-10152313-5417Yaroslavl State University10.18255/1818-1015-2023-4-418-428NJRIIFmais-1829Research ArticleArtificial IntelligenceГенерация ключевых слов для русскоязычных научных текстов с помощью модели mT5Keyphrase generation for the Russian-language scientific texts using mT5https://orcid.org/0000-0001-8409-6457ГлазковаАнна ВалерьевнаGlazkovaAnna V.a.v.glazkova@utmn.ruhttps://orcid.org/0000-0003-4464-1355МорозовДмитрий АлексеевичMorozovDmitry A.morozowdm@gmail.comhttps://orcid.org/0000-0002-1508-4089ВоробьеваМарина СергеевнаVorobevaMarina S.m.s.vorobeva@utmn.ruhttps://orcid.org/0000-0001-5201-1260СтупниковАндрей АнатольевичStupnikovAndreya.a.stupnikov@utmn.ruТюменский государственный университет;Институт проблем передачи информации РАН им. А. А. ХаркевичаРоссияUniversity of Tyumen;Institute for Information Transmission Problems (Kharkevich Institute)Russian FederationНовосибирский национальный исследовательский государственный университет;Институт проблем передачи информации РАН им. А. А. ХаркевичаРоссияNovosibirsk National Research State University;Institute for Information Transmission Problems (Kharkevich Institute)Russian FederationТюменский государственный университетРоссияUniversity of TyumenRussian Federation202311122023304418428Copyright © Глазкова А.В., Морозов Д.А., Воробьева М.С., Ступников А.А., 20232023Глазкова А.В., Морозов Д.А., Воробьева М.С., Ступников А.А.Glazkova A.V., Morozov D.A., Vorobeva M.S., Stupnikov A.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://www.mais-journal.ru/jour/article/view/1829

Авторами предлагается подход к генерации ключевых слов для русскоязычных научных текстов с помощью модели mT5 (multilingual text-to-text transformer), дообученнной на материале текстового корпуса Keyphrases CS&Math Russian. Автоматический подбор ключевых слов является актуальной задачей обработки естественного языка, поскольку ключевые слова помогают читателям осуществлять поиск статей и облегчают систематизацию научных текстов. В данной работе задача подбора ключевых слов рассматривается как задача автоматического реферирования текстов. Дообучение mT5 осуществлялась на текстах аннотаций русскоязычных научных статей. В качестве входных и выходных данных выступали тексты аннотаций и списки ключевых слов, разделенных запятыми, соответственно. Результаты, полученные с помощью mT5, были сравнены с результатами нескольких базовых методов: TopicRank, YAKE!, RuTermExtract, и KeyBERT. Для представления результатов использовались следующие метрики: F-мера, ROUGE-1, BERTScore. Лучшие результаты на тестовой выборке были получены с помощью mT5 и RuTermExtract. Наиболее высокое значение F-меры продемонстрировала модель mT5 (11.24 %), превзойдя RuTermExtract на 0.22 %. RuTermExtract показал лучший результат по метрике ROUGE-1 (15.12 %). Лучшие результаты по BERTScore также были достигнуты этими двумя методами: mT5 — 76.89 % (BERTScore, использующая модель mBERT), RuTermExtract — 75.8 % (BERTScore на основе ruSciBERT). Также авторами была оценена возможность mT5 генерировать ключевые слова, отсутствующие в исходном тексте. К ограничениям предложенного подхода относятся необходимость формирования обучающей выборки для дообучения модели и, вероятно, ограниченная применимость дообученной модели для текстов других предметных областей. Преимущества генерации ключевых слов с помощью mT5 — отсутствие необходимости задавать фиксированные значения длины и количества ключевых слов, необходимости проводить нормализацию, что особенно важно для флективных языков, и возможность генерировать ключевые слова, в явном виде отсутствующие в тексте.

In this work, we applied the multilingual text-to-text transformer (mT5) to the task of keyphrase generation for Russian scientific texts using the Keyphrases CS&Math Russian corpus. The automatic selection of keyphrases is a relevant task of natural language processing since keyphrases help readers find the article easily and facilitate the systematization of scientific texts. In this paper, the task of keyphrase selection is considered as a text summarization task. The mT5 model was fine-tuned on the texts of abstracts of Russian research papers. We used abstracts as an input of the model and lists of keyphrases separated with commas as an output. The results of mT5 were compared with several baselines, including TopicRank, YAKE!, RuTermExtract, and KeyBERT. The results are reported in terms of the full-match F1-score, ROUGE-1, and BERTScore. The best results on the test set were obtained by mT5 and RuTermExtract. The highest F1-score is demonstrated by mT5 (11,24 %), exceeding RuTermExtract by 0,22 %. RuTermextract shows the highest score for ROUGE-1 (15,12 %). According to BERTScore, the best results were also obtained using these methods: mT5 — 76,89 % (BERTScore using mBERT), RuTermExtract — 75,8 % (BERTScore using ruSciBERT). Moreover, we evaluated the capability of mT5 for predicting the keyphrases that are absent in the source text. The important limitations of the proposed approach are the necessity of having a training sample for fine-tuning and probably limited suitability of the fine-tuned model in cross-domain settings. The advantages of keyphrase generation using pre-trained mT5 are the absence of the need for defining the number and length of keyphrases and normalizing produced keyphrases, which is important for flective languages, and the ability to generate keyphrases that are not presented in the text explicitly.

автоматическое реферированиеподбор ключевых словmT5automatic text summarizationselecting keyphrasesmT5Работа выполнена в рамках проекта №МК-3118.2022.4, поддержанного грантом Президента Россий- ской Федерации для молодых ученых — кандидатов наук.The work was carried out within the framework of project No. MK-3118.2022.4, supported by a grant from the President of the Russian Federation for young scientists — candidates of science.ReferencesN. S. Lagutina, K. V. Lagutina, A. S. Adrianov, and I. V. Paramonov, “Russian Language Thesauri: Automated Construction and Application For Natural Language Processing Tasks,” Modeling and Analysis of Information Systems, vol. 25, no. 4, pp. 435–458, 2018.N. S. Lagutina, K. V. Lagutina, A. S. Adrianov, and I. V. Paramonov, “Russian Language Thesauri: Automated Construction and Application For Natural Language Processing Tasks,” Modeling and Analysis of Information Systems, vol. 25, no. 4, pp. 435–458, 2018.S. Beliga, “Keyword extraction: a review of methods and approaches.” 2014.S. Beliga, “Keyword extraction: a review of methods and approaches.” 2014.E. cCano and O. Bojar, “Keyphrase generation: A multi-aspect survey,” in 25th Conference of Open Innovations Association (FRUCT), 2019, pp. 85–94.E. cCano and O. Bojar, “Keyphrase generation: A multi-aspect survey,” in 25th Conference of Open Innovations Association (FRUCT), 2019, pp. 85–94.R. Campos, V. Mangaravite, A. Pasquali, A. Jorge, C. Nunes, and A. Jatowt, “YAKE! Keyword extraction from single documents using multiple local features,” Information Sciences, vol. 509, pp. 257–289, 2020.R. Campos, V. Mangaravite, A. Pasquali, A. Jorge, C. Nunes, and A. Jatowt, “YAKE! Keyword extraction from single documents using multiple local features,” Information Sciences, vol. 509, pp. 257–289, 2020.S. R. El-Beltagy and A. Rafea, “KP-Miner: A keyphrase extraction system for English and Arabic documents,” Information systems, vol. 34, no. 1, pp. 132–144, 2009.S. R. El-Beltagy and A. Rafea, “KP-Miner: A keyphrase extraction system for English and Arabic documents,” Information systems, vol. 34, no. 1, pp. 132–144, 2009.A. Bougouin, F. Boudin, and B. Daille, “TopicRank: Graph-based topic ranking for keyphrase extraction,” in International joint conference on natural language processing (IJCNLP), 2013, pp. 543–551.A. Bougouin, F. Boudin, and B. Daille, “TopicRank: Graph-based topic ranking for keyphrase extraction,” in International joint conference on natural language processing (IJCNLP), 2013, pp. 543–551.R. Mihalcea and P. Tarau, “TextRank: Bringing order into text,” in Proceedings of the 2004 conference on empirical methods in natural language processing, 2004, pp. 404–411.R. Mihalcea and P. Tarau, “TextRank: Bringing order into text,” in Proceedings of the 2004 conference on empirical methods in natural language processing, 2004, pp. 404–411.I. H. Witten, G. W. Paynter, E. Frank, C. Gutwin, and C. G. Nevill-Manning, “KEA: Practical automatic keyphrase extraction,” in Proceedings of the fourth ACM conference on Digital libraries, 1999, pp. 254–255.I. H. Witten, G. W. Paynter, E. Frank, C. Gutwin, and C. G. Nevill-Manning, “KEA: Practical automatic keyphrase extraction,” in Proceedings of the fourth ACM conference on Digital libraries, 1999, pp. 254–255.M. Grootendorst, “KeyBERT: Minimal keyword extraction with BERT.” Zenodo, 2020, doi: 10.5281/zenodo.4461265.M. Grootendorst, “KeyBERT: Minimal keyword extraction with BERT.” Zenodo, 2020, doi: 10.5281/zenodo.4461265.F. Boudin and Y. Gallina, “Redefining Absent Keyphrases and their Effect on Retrieval Effectiveness,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 4185–4193.F. Boudin and Y. Gallina, “Redefining Absent Keyphrases and their Effect on Retrieval Effectiveness,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 4185–4193.R. Meng, S. Zhao, S. Han, D. He, P. Brusilovsky, and Y. Chi, “Deep Keyphrase Generation,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017, pp. 582–592.R. Meng, S. Zhao, S. Han, D. He, P. Brusilovsky, and Y. Chi, “Deep Keyphrase Generation,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017, pp. 582–592.E. Cano and O. Bojar, “Keyphrase Generation: A Text Summarization Struggle,” in Proceedings of NAACL-HLT, 2019, pp. 666–672.E. Cano and O. Bojar, “Keyphrase Generation: A Text Summarization Struggle,” in Proceedings of NAACL-HLT, 2019, pp. 666–672.J. Zhao and Y. Zhang, “Incorporating linguistic constraints into keyphrase generation,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 5224–5233.J. Zhao and Y. Zhang, “Incorporating linguistic constraints into keyphrase generation,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 5224–5233.R. Liu, Z. Lin, and W. Wang, “Keyphrase Prediction With Pre-trained Language Model.” 2020.R. Liu, Z. Lin, and W. Wang, “Keyphrase Prediction With Pre-trained Language Model.” 2020.M. Kulkarni, D. Mahata, R. Arora, and R. Bhowmik, “Learning Rich Representation of Keyphrases from Text,” in Findings of the Association for Computational Linguistics: NAACL 2022, 2022, pp. 891–906.M. Kulkarni, D. Mahata, R. Arora, and R. Bhowmik, “Learning Rich Representation of Keyphrases from Text,” in Findings of the Association for Computational Linguistics: NAACL 2022, 2022, pp. 891–906.A. Vaswani et al., “Attention is All You Need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, pp. 6000–6010.A. Vaswani et al., “Attention is All You Need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, pp. 6000–6010.M. F. M. Chowdhury, G. Rossiello, M. Glass, N. Mihindukulasooriya, and A. Gliozzo, “Applying a Generic Sequence-to-Sequence Model for Simple and Effective Keyphrase Generation.” 2022.M. F. M. Chowdhury, G. Rossiello, M. Glass, N. Mihindukulasooriya, and A. Gliozzo, “Applying a Generic Sequence-to-Sequence Model for Simple and Effective Keyphrase Generation.” 2022.A. V. Glazkova and D. A. Morozov, “Applying transformer-based text summarization for keyphrase generation,” Lobachevskii Journal of Mathematics, vol. 44, no. 1, pp. 123–136, 2023.A. V. Glazkova and D. A. Morozov, “Applying transformer-based text summarization for keyphrase generation,” Lobachevskii Journal of Mathematics, vol. 44, no. 1, pp. 123–136, 2023.A. Glazkova and D. Morozov, “Multi-task fine-tuning for generating keyphrases in a scientific domain,” in IX International Conference on Information Technology and Nanotechnology (ITNT), 2023, pp. 1–5.A. Glazkova and D. Morozov, “Multi-task fine-tuning for generating keyphrases in a scientific domain,” in IX International Conference on Information Technology and Nanotechnology (ITNT), 2023, pp. 1–5.D. Wu, W. U. Ahmad, and K.-W. Chang, “Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study.” 2022.D. Wu, W. U. Ahmad, and K.-W. Chang, “Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study.” 2022.E. G. Sokolova and O. Mitrofanova, “Automatic Keyphrase Extraction by applying KEA to Russian texts,” in Computational linguistics and computing ontologies, 2017, pp. 157–165.E. G. Sokolova and O. Mitrofanova, “Automatic Keyphrase Extraction by applying KEA to Russian texts,” in Computational linguistics and computing ontologies, 2017, pp. 157–165.M. V. Sandul and E. G. Mikhailova, “Keyword extraction from single Russian document,” in Proceedings of the Third Conference on Software Engineering and Information Management, 2018, pp. 30–36.M. V. Sandul and E. G. Mikhailova, “Keyword extraction from single Russian document,” in Proceedings of the Third Conference on Software Engineering and Information Management, 2018, pp. 30–36.E. Sokolova, A. Moskvina, and O. Mitrofanova, “Keyphrase Extraction from the Russian Corpus on Linguistics by Means of KEA and RAKE Algorithms,” in Data analytics and management in data-intensive domains, 2018, pp. 369–372.E. Sokolova, A. Moskvina, and O. Mitrofanova, “Keyphrase Extraction from the Russian Corpus on Linguistics by Means of KEA and RAKE Algorithms,” in Data analytics and management in data-intensive domains, 2018, pp. 369–372.O. A. Mitrofanova and D. A. Gavrilic, “Experiments on automatic keyphrase extraction in stylistically heterogeneous corpus of Russian texts,” Terra Linguistica, vol. 50, no. 4, pp. 22–40, 2022.O. A. Mitrofanova and D. A. Gavrilic, “Experiments on automatic keyphrase extraction in stylistically heterogeneous corpus of Russian texts,” Terra Linguistica, vol. 50, no. 4, pp. 22–40, 2022.D. A. Morozov, A. V. Glazkova, M. A. Tyutyulnikov, and B. L. Iomdin, “Keyphrase Generation for Abstracts of the Russian-Language Scientific Articles,” NSU Vestnik. Series: Linguistics and Intercultural Communication, vol. 21, no. 1, pp. 54–66, 2023.D. A. Morozov, A. V. Glazkova, M. A. Tyutyulnikov, and B. L. Iomdin, “Keyphrase Generation for Abstracts of the Russian-Language Scientific Articles,” NSU Vestnik. Series: Linguistics and Intercultural Communication, vol. 21, no. 1, pp. 54–66, 2023.B. Koloski, S. Pollak, B. vSkrlj, and M. Martinc, “Extending Neural Keyword Extraction with TF-IDF tagset matching,” in Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation, 2021, pp. 22–29.B. Koloski, S. Pollak, B. vSkrlj, and M. Martinc, “Extending Neural Keyword Extraction with TF-IDF tagset matching,” in Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation, 2021, pp. 22–29.D. Morozov and A. Glazkova, “Keyphrases CS&Math Russian.” Mendeley Data, 2022, doi: 10.17632/dv3j9wc59v.1.D. Morozov and A. Glazkova, “Keyphrases CS&Math Russian.” Mendeley Data, 2022, doi: 10.17632/dv3j9wc59v.1.L. Xue et al., “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 483–498.L. Xue et al., “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 483–498.K. Grashchenkov, A. Grabovoy, and I. Khabutdinov, “A Method of Multilingual Summarization For Scientific Documents,” in Ivannikov Ispras Open Conference (ISPRAS), 2022, pp. 24–30.K. Grashchenkov, A. Grabovoy, and I. Khabutdinov, “A Method of Multilingual Summarization For Scientific Documents,” in Ivannikov Ispras Open Conference (ISPRAS), 2022, pp. 24–30.A. Gryaznov, R. Rybka, I. Moloshnikov, A. Selivanov, and A. Sboev, “Influence of the duration of training a deep neural network model on the quality of text summarization task,” AIP Conference Proceedings, vol. 2849, no. 1, p. 400006, 2023.A. Gryaznov, R. Rybka, I. Moloshnikov, A. Selivanov, and A. Sboev, “Influence of the duration of training a deep neural network model on the quality of text summarization task,” AIP Conference Proceedings, vol. 2849, no. 1, p. 400006, 2023.A. A. Pechnikov, “Comparative analysis of scientometrics indicators of journals Math-Net.ru and Elibrary.ru,” Vestnik Tomskogo gosudarstvennogo universiteta, no. 56, pp. 112–121, 2021.A. A. Pechnikov, “Comparative analysis of scientometrics indicators of journals Math-Net.ru and Elibrary.ru,” Vestnik Tomskogo gosudarstvennogo universiteta, no. 56, pp. 112–121, 2021.Y. Kuratov and M. Arkhipov, “Adaptation of deep bidirectional multilingual transformers for Russian language,” in Komp'juternaja Lingvistika i Intellektual'nye Tehnologii, 2019, pp. 333–339.Y. Kuratov and M. Arkhipov, “Adaptation of deep bidirectional multilingual transformers for Russian language,” in Komp'juternaja Lingvistika i Intellektual'nye Tehnologii, 2019, pp. 333–339.C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485–5551, 2020.C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485–5551, 2020.L. Page, S. Brin, R. Motwani, and T. Winograd, “The PageRank citation ranking: Bringing order to the web: Stanford InfoLab,” in Navigation, findability and the usage of cultural heritage on the web: an exploratory study, 1999, p. 1508503.L. Page, S. Brin, R. Motwani, and T. Winograd, “The PageRank citation ranking: Bringing order to the web: Stanford InfoLab,” in Navigation, findability and the usage of cultural heritage on the web: an exploratory study, 1999, p. 1508503.M. Korobov, “Morphological analyzer and generator for Russian and Ukrainian languages,” in Analysis of Images, Social Networks and Texts: 4th International Conference, AIST 2015, Yekaterinburg, Russia, April 9--11, 2015, Revised Selected Papers 4, 2015, pp. 320–332.M. Korobov, “Morphological analyzer and generator for Russian and Ukrainian languages,” in Analysis of Images, Social Networks and Texts: 4th International Conference, AIST 2015, Yekaterinburg, Russia, April 9--11, 2015, Revised Selected Papers 4, 2015, pp. 320–332.J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171–4186.J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171–4186.F. Boudin, “PKE: an open source python-based keyphrase extraction toolkit,” in Proceedings of COLING 2016, the 26th international conference on computational linguistics: system demonstrations, 2016, pp. 69–73.F. Boudin, “PKE: an open source python-based keyphrase extraction toolkit,” in Proceedings of COLING 2016, the 26th international conference on computational linguistics: system demonstrations, 2016, pp. 69–73.N. A. Gerasimenko, A. S. Chernyavsky, and M. A. Nikiforova, “ruSciBERT: a transformer language model for obtaining semantic embeddings of scientific texts in Russian,” in Doklady Mathematics, 2022, vol. 106, no. Suppl 1, pp. S95–S96.N. A. Gerasimenko, A. S. Chernyavsky, and M. A. Nikiforova, “ruSciBERT: a transformer language model for obtaining semantic embeddings of scientific texts in Russian,” in Doklady Mathematics, 2022, vol. 106, no. Suppl 1, pp. S95–S96.C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text summarization branches out, 2004, pp. 74–81.C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text summarization branches out, 2004, pp. 74–81.T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “BERTScore: Evaluating Text Generation with BERT.” 2020.T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “BERTScore: Evaluating Text Generation with BERT.” 2020.

The authors declare that there are no conflicts of interest present.