maisМоделирование и анализ информационных системModeling and Analysis of Information Systems1818-10152313-5417Yaroslavl State University10.18255/1818-1015-2025-3-282-297mais-1963Research ArticleArtificial IntelligenceМетодология иерархического многозадачного обучения нейронных сетей типа ERNIE 3 для анализа и генерации русскоязычных текстовHierarchical multi-task learning methodology for ERNIE-3-Type neural networks in Russian-language text analysis and generationhttps://orcid.org/0009-0000-7413-1503ТотминаЕкатерина ВадимовнаTotminaEkaterina V.e.totmina@g.nsu.ruhttps://orcid.org/0000-0002-0457-0698БондаренкоИванBondarenkoIvani.bondarenko@g.nsu.ruhttps://orcid.org/0009-0007-6873-7868СередкинАлександр ВалерьевичSeredkinAleksandr V.a.seredkin@g.nsu.ruНовосибирский национальный исследовательский государственный университетРоссияNovosibirsk National Research State UniversityRussian Federation202524092025323282297Copyright © Тотмина Е.В., Бондаренко И., Середкин А.В., 20252025Тотмина Е.В., Бондаренко И., Середкин А.В.Totmina E.V., Bondarenko I., Seredkin A.V.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://www.mais-journal.ru/jour/article/view/1963

Статья посвящена разработке методологии иерархического многозадачного обучения нейронных сетей, основанной на принципах архитектуры ERNIE 3, и экспериментальной апробации данной методологии на базе модели FRED-T5 для задач анализа и генерации текстов на русском языке. Иерархическое многозадачное обучение является перспективным подходом к созданию универсальных языковых моделей, способных эффективно решать разнообразные задачи обработки естественного языка (NLP). Предложенная методология объединяет преимущества специализированных энкодерных блоков для задач понимания текста (NLU) и общего декодера для генеративных задач (NLG), что позволяет повысить производительность модели и снизить вычислительные затраты. В работе проведён сравнительный анализ эффективности разработанной методологии на открытом бенчмарке Russian SuperGLUE с использованием предварительно обученной русскоязычной модели FRED-T5-1.7B. Экспериментальные результаты подтвердили существенное улучшение качества модели в режимах zero-shot и few-shot по сравнению с базовой конфигурацией. Дополнительно рассмотрены возможности практического применения разработанного подхода в решении реальных NLP-задач, а также даны рекомендации по дальнейшему развитию методологии и её интеграции в прикладные системы обработки русскоязычных текстов.

The article addresses the development of a methodology for hierarchical multi-task learning of neural networks, inspired by the ERNIE 3 architecture, and its experimental validation using the FRED-T5 model for Russian-language text analysis and generation tasks. Hierarchical multi-task learning represents a promising approach for creating universal language models capable of efficiently solving a variety of natural language processing (NLP) tasks. The proposed methodology integrates specialized encoder blocks for natural language understanding (NLU) tasks with a shared decoder for natural language generation (NLG) tasks, thus improving model performance and reducing computational costs. This paper presents a comparative analysis of the developed methodology’s performance using the open Russian SuperGLUE benchmark and the pre-trained Russian-language model FRED-T5-1.7B. Experimental results confirm a significant improvement in model quality in both zero-shot and few-shot scenarios compared to the baseline configuration. Additionally, the paper explores practical applications of the developed approach in real NLP tasks and provides recommendations for further advancement of the methodology and its integration into applied systems for processing Russian-language texts.

иерархическое многозадачное обучениеFRED-T5обработка естественного языканейронные сетигенерация текстаанализ текстаzero-shot обучениеfew-shot обучениеseq2seq моделиhierarchical multi-task learningFRED-T5natural language processingneural networkstext generationtext analysiszero-shot learningfew-shot learningseq2seq modelsReferencesT. Brown et al., “Language models are few-shot learners,” Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901, 2020.T. Brown et al., “Language models are few-shot learners,” Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901, 2020.H. Touvron et al., “LLaMA: Open and Efficient Foundation Language Models.” 2023.H. Touvron et al., “LLaMA: Open and Efficient Foundation Language Models.” 2023.A. Chowdhery et al., “Palm: Scaling language modeling with pathways,” Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023.A. Chowdhery et al., “Palm: Scaling language modeling with pathways,” Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023.C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020.C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020.Y. Zhu et al., “Can Large Language Models Understand Context?,” in Findings of the Association for Computational Linguistics: EACL 2024, 2024, pp. 2004–2018.Y. Zhu et al., “Can Large Language Models Understand Context?,” in Findings of the Association for Computational Linguistics: EACL 2024, 2024, pp. 2004–2018.D. Khurana, A. Koli, K. Khatter, and S. Singh, “Natural language processing: state of the art, current trends, challenges,” Multimedia Tools, Applications, vol. 82, pp. 3713–3744, 2023, doi: 10.1007/s11042-022-13428-4.D. Khurana, A. Koli, K. Khatter, and S. Singh, “Natural language processing: state of the art, current trends, challenges,” Multimedia Tools, Applications, vol. 82, pp. 3713–3744, 2023, doi: 10.1007/s11042-022-13428-4.D. Hupkes et al., “A taxonomy, review of generalization research in NLP,” Nature Machine Intelligence, vol. 5, pp. 1161–1174, 2023, doi: 10.1038/s42256-023-00729-y.D. Hupkes et al., “A taxonomy, review of generalization research in NLP,” Nature Machine Intelligence, vol. 5, pp. 1161–1174, 2023, doi: 10.1038/s42256-023-00729-y.P. P. Ray, “ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope,” Internet of Things and Cyber-Physical Systems, vol. 3, pp. 121–154, 2023.P. P. Ray, “ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope,” Internet of Things and Cyber-Physical Systems, vol. 3, pp. 121–154, 2023.Y. Yang and Z. Xue, “Training Heterogeneous Features in Sequence to Sequence Tasks: Latent Enhanced Multi-filter Seq2Seq Model,” in Intelligent Systems, Applications, 2023, pp. 103–117.Y. Yang and Z. Xue, “Training Heterogeneous Features in Sequence to Sequence Tasks: Latent Enhanced Multi-filter Seq2Seq Model,” in Intelligent Systems, Applications, 2023, pp. 103–117.Y. Sun et al., “ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation.” 2021.Y. Sun et al., “ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation.” 2021.D. Zmitrovich et al., “A Family of Pretrained Transformer Language Models for Russian,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 507–524.D. Zmitrovich et al., “A Family of Pretrained Transformer Language Models for Russian,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 507–524.M. Song and Y. Zhao, “Enhance RNNLMs with Hierarchical Multi-Task Learning for ASR,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 6102–6106.M. Song and Y. Zhao, “Enhance RNNLMs with Hierarchical Multi-Task Learning for ASR,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 6102–6106.A. Vaswani et al., “Attention Is All You Need,” in Advances in Neural Information Processing Systems, 2017, vol. 30, pp. 5998–6008.A. Vaswani et al., “Attention Is All You Need,” in Advances in Neural Information Processing Systems, 2017, vol. 30, pp. 5998–6008.V. Sanh et al., “Multitask Prompted Training Enables Zero-Shot Task Generalization.” 2022.V. Sanh et al., “Multitask Prompted Training Enables Zero-Shot Task Generalization.” 2022.Y. Tay et al., “UL2: Unifying Language Learning Paradigms.” 2023.Y. Tay et al., “UL2: Unifying Language Learning Paradigms.” 2023.Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum Learning,” in Proceedings of the 26th International Conference on Machine Learning, 2009, pp. 41–48.Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum Learning,” in Proceedings of the 26th International Conference on Machine Learning, 2009, pp. 41–48.I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, “Cross-stitch networks for multi-task learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 3994–4003.I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, “Cross-stitch networks for multi-task learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 3994–4003.Y. Sun et al., “ERNIE 2.0: A continual pre-training framework for language understanding,” in Proceedings of the AAAI conference on Artificial Intelligence, 2020, vol. 34, no. 05, pp. 8968–8975.Y. Sun et al., “ERNIE 2.0: A continual pre-training framework for language understanding,” in Proceedings of the AAAI conference on Artificial Intelligence, 2020, vol. 34, no. 05, pp. 8968–8975.L. Xue et al., “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 483–498, doi: 10.18653/v1/2021.naacl-main.41.L. Xue et al., “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 483–498, doi: 10.18653/v1/2021.naacl-main.41.S. Wang et al., “ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation.” 2021.S. Wang et al., “ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation.” 2021.J. Pfeiffer, A. Kamath, A. R"uckl'e, K. Cho, and I. Gurevych, “AdapterFusion: Non-Destructive Task Composition for Transfer Learning,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 2021, pp. 487–503, doi: 10.18653/v1/2021.eacl-main.39.J. Pfeiffer, A. Kamath, A. R"uckl'e, K. Cho, and I. Gurevych, “AdapterFusion: Non-Destructive Task Composition for Transfer Learning,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 2021, pp. 487–503, doi: 10.18653/v1/2021.eacl-main.39.W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022.W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022.S. Longpre et al., “The flan collection: Designing data and methods for effective instruction tuning,” in Proceedings of the International Conference on Machine Learning, 2023, pp. 22631–22648.S. Longpre et al., “The flan collection: Designing data and methods for effective instruction tuning,” in Proceedings of the International Conference on Machine Learning, 2023, pp. 22631–22648.N. Houlsby et al., “Parameter-efficient transfer learning for NLP,” in Proceedings of the International Conference on Machine Learning, 2019, pp. 2790–2799.N. Houlsby et al., “Parameter-efficient transfer learning for NLP,” in Proceedings of the International Conference on Machine Learning, 2019, pp. 2790–2799.H. A. A. Al-Khamees, M. E. Manaa, Z. H. Obaid, and N. A. Mohammedali, “Implementing Cyclical Learning Rates in Deep Learning Models for Data Classification,” in Proceedings of the International Conference on Forthcoming Networks and Sustainability in the AIoT Era, 2024, pp. 205–215.H. A. A. Al-Khamees, M. E. Manaa, Z. H. Obaid, and N. A. Mohammedali, “Implementing Cyclical Learning Rates in Deep Learning Models for Data Classification,” in Proceedings of the International Conference on Forthcoming Networks and Sustainability in the AIoT Era, 2024, pp. 205–215.A. Koloskova, H. Hendrikx, and S. U. Stich, “Revisiting gradient clipping: Stochastic bias and tight convergence guarantees,” in Proceedings of the International Conference on Machine Learning, 2023, pp. 17343–17363.A. Koloskova, H. Hendrikx, and S. U. Stich, “Revisiting gradient clipping: Stochastic bias and tight convergence guarantees,” in Proceedings of the International Conference on Machine Learning, 2023, pp. 17343–17363.

The authors declare that there are no conflicts of interest present.