References

mais

Моделирование и анализ информационных систем

Modeling and Analysis of Information Systems

1818-10152313-5417

Yaroslavl State University

10.18255/1818-1015-2022-4-286-314

mais-1748

Research Article

Theory of Data

Анализ академической успеваемости студентов с использованием журналов событий электронной образовательной среды

Analysis of Students’ Academic Performance using LMS Event Logs

https://orcid.org/0000-0003-3843-5379

Шаимов

Никита Денисович

Shaimov

Nikita D.

nshaimov@hse.ru

https://orcid.org/0000-0002-9420-3751

Ломазова

Ирина Александровна

Lomazova

Irina A.

ilomazova@hse.ru

https://orcid.org/0000-0003-2352-3384

Мицюк

Алексей Александрович

Mitsyuk

Alexey A.

amitsyuk@hse.ru

https://orcid.org/0000-0002-3063-4640

Самоненко

Илья Юрьевич

Samonenko

Ilya Yu.

isamonenko@hse.ru

Национальный исследовательский университет «Высшая школа экономики»РоссияHSE UniversityRussian Federation

2022

18122022

294286314

2022

Шаимов Н.Д., Ломазова И.А., Мицюк А.А., Самоненко И.Ю.

Shaimov N.D., Lomazova I.A., Mitsyuk A.A., Samonenko I.Y.

Данная работа распространяется под лицензией Creative Commons Attribution 4.0.

This work is licensed under a Creative Commons Attribution 4.0 License.

https://www.mais-journal.ru/jour/article/view/1748

Современный образовательный процесс предполагает использование электронных образовательных сред. Это специальные информационные системы, которые являются как средством для хранения учебных материалов, так и инструментом для проведения проверочных работ, сбора домашних заданий, ведения журнала оценок, совместной работы. Такие среды производят большое количество данных о поведении учащихся и преподавателей в рамках учебного процесса. В данной работе предлагается подход, позволяющий анализировать такие данные, извлекать из них типичные траектории учащихся, которые ведут к успешным или неудачным результатам обучения. Показано, как для построения моделей образовательного процесса на основе имеющихся данных могут быть использованы алгоритмы process mining. Также показано, как можно оценить, насколько синтезированная модель отражает реальное поведение системы, записанное в журналах событий. Работа содержит не только описание предлагаемого подхода, но и пример его применения к реальному набору данных для образовательной программы бакалавриата. Наглядно показано, как с использованием нашего подхода можно выяснить, какие факторы приводят к формированию успешных и неудачных траекторий студентов. Выявлены узкие места образовательного процесса, а также ошибки в данных, свидетельствующие о некорректной работе системы. В результате анализа выявлены точки особого внимания для администраторов образовательной программы, а также определены некоторые сигнальные события, появление которых в индивидуальной траектории студента может быть тревожным сигналом. Применение подхода предполагает использование только свободных программных инструментов с открытым исходным кодом, что дополнительно облегчает его внедрение в самых разных образовательных организациях.

Modern educational process involves the use of electronic educational environments. These are special information systems that are both a means for storing educational materials and a tool for conducting tests, collecting homework, keeping a grade book, and working together. Such environments produce a large amount of data containing the recorded behavior of students and teachers within the educational process. This paper proposes an approach that allows one to analyze such data and discover typical student trajectories that lead to successful or unsuccessful learning outcomes. It is shown how process mining can be used to build models of the educational process based on the available data. We also show how you can evaluate the extent to which the synthesized model reflects the actual behavior of the system recorded in event logs. The paper contains not only a description of the proposed approach, but also a case study with its application to a real data set for an undergraduate educational program. It is clearly shown how, using our approach, it is possible to find out what factors lead to the formation of successful and unsuccessful student trajectories. The bottlenecks of the educational process were identified, as well as errors in the data, indicating the incorrect operation of the system. As a result of the analysis, points of special attention for administrators of the educational program were identified, as well as some signal events, the appearance of which in a student’s individual trajectory can be an alarm. The application of the approach involves the use of free open source software, which further facilitates its deployment in a variety of educational organizations.

моделирование процессовизвлечение и анализ моделей процессовэлектронная образовательная средажурналы событий

process analysisprocess mininglearning management systemsevent logs

References1

R. Jaakonmaki, J. vom Brocke, S. Dietze, H. Drachsler, A. Fortenbacher, R. Helbig, M. D. Kickmeier-Rust, I. Marenzi, A. Suarez, and H. Yun, Learning Analytics Cookbook - How to Support Learning Processes Through Data Analytics and Visualization, ser. Springer Briefs in Business Process Management. Springer, 2020.

W. van der Aalst, Process Mining - Data Science in Action, Second Edition. Springer, 2016, isbn: 978-3-662-49850-7. doi: 10.1007/978-3-662-49851-4. [Online]. Available: https://doi.org/10.1007/978-3--49851-4.

J. Carmona, B. van Dongen, A. Solti, and M. Weidlich, Conformance Checking - Relating Processes and Models. Springer, 2018, isbn: 978-3-319-99413-0. doi: 10.1007/978-3-319-99414-7. [Online]. Available: https://doi.org/10.1007/978-3-319-99414-7.

S. Suriadi, M. T. Wynn, C. Ouyang, A. H. M. ter Hofstede, and N. J. van Dijk, «Understanding Process Behaviours in a Large Insurance Company in Australia: A Case Study», in CAiSE, ser. Lecture Notes in Computer Science, vol. 7908, Springer, 2013, pp. 449-464.

M. Mittal and A. Sureka, «Process mining software repositories from student projects in an undergraduate software engineering course», in ICSE Companion, ACM, 2014, pp. 344-353.

A. Mitsyuk, A. Kalenkova, S. Shershakov, and W. van der Aalst, «Using process mining for the analysis of an e-trade system: A case study», Biznes-informatika, no. 3 (29), pp. 15-27, 2014.

S.-k. Lee, B. Kim, M. Huh, S. Cho, S. Park, and D. Lee, «Mining transportation logs for understanding the after-assembly block manufacturing process in the shipbuilding industry», Expert Syst. Appl., vol. 40, no. 1, pp. 83-95, 2013.

A. Valencia-Parra, B. Ramos-Gutierrez, A. J. Varela-Vaca, M. T. G. Lopez, and A. G. Bernal, «Enabling process mining in aircraft manufactures: extracting event logs and discovering processes from complex data», in BPM (Industry Forum), ser. CEUR Workshop Proceedings, vol. 2428, CEUR-WS.org, 2019, pp. 166-177.

K. Smit and J. Mens, «Process Mining in The Rail Industry: A Qualitative Analysis of Success Factors and Remaining Challenges», in Bled eConference, University of Maribor Press / Association for Information Systems, 2019, p. 25.

J. Munoz-Gama, N. Martin, C. Fernandez-Llatas, O. A. Johnson, M. Sepulveda, E. Helm, V. Galvez-Yanjari, E. Rojas, A. Martinez-Millana, D. Aloini, I. A. Amantea, R. Andrews, M. Arias, I. Beerepoot, E. Benevento, A. Burattin, D. Capurro, J. Carmona, M.Comuzzi, B. Dalmas, R. de la Fuente, C. D. Francescomarino, C. D. Ciccio, R. Gatta, C. Ghidini, F. Gonzalez-Lopez, G. Ibanez-Sanchez, H. B. Klasky, A. P. Kurniati, X. Lu, F. Mannhardt, R. Mans, M. Marcos, R. M. de Carvalho, M. Pegoraro, S. K. Poon, L. Pufahl, H. A. Reijers, S. Remy, S. Rinderle-Ma, L. Sacchi, F. Seoane, M. Song, A. Stefanini, E. Sulis, A. H. M. ter Hofstede, P. J. Toussaint, V. Traver, Z. Valero-Ramon, I. van de Weerd, W. van der Aalst, R. J. B. Vanwersch, M. Weske, M. T. Wynn, and F. Zerbato, «Process mining for healthcare: Characteristics and challenges», J. Biomed. Informatics, vol. 127, p. 103 994, 2022.

A. Guzzo, A.Rullo, and E. Vocaturo, «Process mining applications in the healthcare domain: A comprehensive review», WIREs Data Mining Knowl. Discov., vol. 12, no. 2, 2022.

M. R. Dallagassa, C. dos Santos Garcia, E. E. Scalabrin, S. O. Ioshii, and D. R. Carvalho, «Opportunities and challenges for applying process mining in healthcare: a systematic mapping study», J. Ambient Intell. Humaniz.Comput., vol. 13, no. 1, pp. 165-182, 2022.

C. dos Santos Garcia, A. Meincheim, E. R. F. Junior, M. R. Dallagassa, D. M. V. Sato, D. R. Carvalho, E. A. P. Santos, and E. E. Scalabrin, «Process mining techniques and applications - A systematic mapping study», Expert Syst. Appl., vol. 133, pp. 260-295, 2019.

M. Dumas and F. M. Maggi, «Enabling Process Innovation via Deviance Mining and Predictive Monitoring», in BPM - Driving Innovation in a Digital World, J. vom Brocke and T. Schmiedel, Eds., Springer, 2015, pp. 145-154.

I. Teinemaa, M. Dumas, F. M. Maggi, and C. D. Francescomarino, «Predictive Business Process Monitoring with Structured and Unstructured Data», in Business Process Management - 14th International Conference, BPM 2016, Rio de Janeiro, Brazil, September 18-22, 2016. Proceedings, M. L. Rosa, P. Loos, and O. Pastor, Eds., ser. Lecture Notes in Computer Science, vol. 9850, Springer, 2016, pp. 401-417.

I. Teinemaa, N. Tax, M. de Leoni, M. Dumas, and F. M. Maggi, «Alarm-Based Prescriptive Process Monitoring», in Business Process Management Forum - BPM Forum 2018, Sydney, NSW, Australia, September 9-14, 2018, Proceedings, M. Weske, M. Montali, I. Weber, and J. vom Brocke, Eds., ser. Lecture Notes in Business Information Processing, vol. 329, Springer, 2018, pp. 91-107.

W. van der Aalst, «Business Process Simulation Survival Guide», in Handbook on Business Process Management 1, Introduction, Methods, and Information Systems, 2nd Ed, ser.International Handbooks on Information Systems, J. vom Brocke and M. Rosemann, Eds., Springer, 2015, pp. 337-370.

A. A. Mitsyuk, I. S. Shugurov, A. A. Kalenkova, and W. van der Aalst, «Generating event logs for high-level process models», Simul. Model. Pract. Theory, vol. 74, pp. 1-16, 2017. doi: 10.1016/j.simpat. 2017.01.003. [Online]. Available: https://doi.org/10.1016/j.simpat.2017.01.003.

W. van der Aalst, «Process mining and simulation: a match made in heaven!», in Proceedings of the 50th Computer Simulation Conference, SummerSim 2018, Bordeaux, France, July 09-12, 2018, ACM, 2018, 4:1-4:12. [Online]. Available: http://dl.acm.org/citation.cfm?id=3275386.

W. van der Aalst, «Process-Aware Information Systems: Design, Enactment, and Analysis», in Wiley Encyclopedia of Computer Science and Engineering, B. W. Wah, Ed., John Wiley & Sons, Inc., 2008.

G. Acampora, A. Vitiello, B. N. D. Stefano, W. van der Aalst, C. W. Gunther, and E. Verbeek, «IEEE 1849: The XES Standard: The Second IEEE Standard Sponsored by IEEE Computational Intelligence Society [Society Briefs]», IEEE Comput.Intell. Mag., vol. 12, no. 2, pp. 4-8, 2017.

A. J. M. M. Weijters and W. van der Aalst, «Rediscovering workflow models from event-based data using little thumb», Integr.Comput. Aided Eng., vol. 10, no. 2, pp. 151-162, 2003. doi: 10.3233/ica-2003-10205. [Online]. Available: https://doi.org/10.3233/ica-2003-10205.

A. Berti and W. van der Aalst, «A Novel Token-Based Replay Technique to Speed Up Conformance Checking and Process Enhancement», Trans. Petri Nets Other Model. Concurr., vol. 15, pp. 1-26, 2021.

J. Munoz-Gama and J. Carmona, «A Fresh Look at Precision in Process Conformance», vol. 6336, Sep. 2010, pp. 211-226, isbn: 978-3-642-15617-5. doi: 10.1007/978-3-642-15618-2_16.

J. Buijs, B. Dongen, and W. Aalst, Quality Dimensions in Process Discovery: The Importance of Fitness, Precision, Generalization and Simplicity», International Journal of Cooperative Information Systems, vol. 23, p. 1 440 001, Mar. 2014. doi: 10.1142/S0218843014400012.

A. Adriansyah, J. Munoz-Gama, J. Carmona, B. Dongen, and W. Aalst, «Measuring precision of modeled behavior», Information Systems and e-Business Management, vol. 13, Jan. 2014. doi: 10.1007/ s10257-014-0234-7.

E. Verbeek, J. C. A. M. Buijs, B. F. van Dongen, and W. van der Aalst, «ProM 6: The Process Mining Toolkit», in Proceedings of the Business Process Management 2010 Demonstration Track, Hoboken, NJ, USA, September 14-16, 2010, M. L. Rosa, Ed., ser. CEUR Workshop Proceedings, vol. 615, CEUR-WS.org, 2010. [Online]. Available: http://ceur-ws.org/Vol-615/paper13.pdf.

A. Berti, S. J. van Zelst, and W. van der Aalst, «Process Mining for Python (PM4Py): Bridging the Gap Between Process- and Data Science», CoRR, vol. abs/1905.06169, 2019. arXiv: 1905.06169. [Online]. Available: http://arxiv.org/abs/1905.06169.

A. Berti, M. P. Nghia, and W. van der Aalst, «PM4Py-GPU: A High-Performance General-Purpose Library for Process Mining», in Research Challenges in Information Science - 16th International Conference, RCIS 2022, Barcelona, Spain, May 17-20, 2022, Proceedings, R. S. S. Guizzardi, J. Ralyte, and X. Franch, Eds., ser. Lecture Notes in Business Information Processing, vol. 446, Springer, 2022, pp. 727-734.

S. A. Shershakov, «VTMine for Visio: A Graphical Tool for Modeling in Process Mining», Autom. Control.Comput. Sci., vol. 55, no. 7, pp. 847-865, 2021. doi: 10.3103/S0146411621070282. [Online]. Available: https://doi.org/10.3103/S0146411621070282.

I. S. Shugurov and A. A. Mitsyuk, «Applying MapReduce to conformance checking», Proceedings of ISPRAS, vol. 28, no. 3, pp. 103-122, 2016. [Online]. Available: https://ispranproceedings.elpub.ru/jour/issue/download/9/17#page=104.

A. Bogarin, R. Cerezo, and C. Romero, «A survey on educational process mining», WIREs Data Mining Knowl. Discov., vol. 8, no. 1, 2018. doi: 10.1002/widm.1230. [Online]. Available: https://doi.org/10.1002/widm.1230.

J. C. Vidal, B. Vaezquez-Barreiros, M. Lama, and M. Mucientes, «Recompiling learning processes from event logs», Knowl. Based Syst., vol. 100, pp. 160-174, 2016. doi: 10.1016/j.knosys.2016.03.003. [Online]. Available: https://doi.org/10.1016Zj.knosys.2016.03.003.

A. Bogarin, R. Cerezo, and C. Romero, «Discovering learning processes using Inductive Miner: A case study with Learning Management Systems (LMSs).», Psicothema, vol. 30 3, pp. 322-329, 2018.

H. Al-Qaheri and M. Panda, «An Education Process Mining Framework: Unveiling Meaningful Information for Understanding Students’ Learning Behavior and Improving Teaching Quality», Inf., vol. 13, no. 1, p. 29, 2022. doi: 10.3390/info13010029. [Online]. Available: https://doi.org/10.3390/info13010029.

E. M. Real, E. P. Pimentel, L. V. de Oliveira, J. C. Braga, and I. Stiubiener, «Educational Process Mining for Verifying Student Learning Paths in an Introductory Programming Course», in IEEE Frontiers in Education Conference, FIE 2020, Uppsala, Sweden, October 21-24, 2020, IEEE, 2020, pp. 1-9. doi: 10.1109/ FIE44824.2020.9274125. [Online]. Available: https://doi.org/10.1109/FIE44824.2020.9274125.

J. P. Salazar-Fernandez, M. Sepulveda, J. Munoz-Gama, and M. Nussbaum, «Curricular Analytics to Characterize Educational Trajectories in High-Failure Rate Courses That Lead to Late Dropout», Applied Sciences, vol. 11, no. 4, 2021, issn: 2076-3417. doi: 10.3390/app11041436. [Online]. Available: https://www.mdpi.com/2076-3417/11/4/1436.

J. P. Salazar-Fernandez, J. Munoz-Gama, J. Maldonado-Mahauad, D. Bustamante, and M. Sepulveda, «Backpack Process Model (BPPM): A Process Mining Approach for Curricular Analytics», Applied Sciences, vol. 11, no. 9, 2021, issn: 2076-3417.

I. A. Lomazova, A. A. Mitsyuk, and A. M. Sharipova, ModelingMOOC learnflow with Petri net extensions, 2021. doi: 10.48550/ARXIV.2111.04419. [Online]. Available: https://arxiv.org/abs/2111.04419.

L. Juhanak, J. Zounek, and L. Rohlikova, «Using process mining to analyze students’ quiz-taking behavior patterns in a learning management system», Comput. Hum. Behav., vol. 92, pp. 496-506, 2019. doi: 10.1016/j.chb.2017.12.015. [Online]. Available: https://doi.org/10.1016/j.chb.2017.12.015.

V. Southavilay, K. Yacef, and R. A. Calvo, «Process Mining to Support Students’ Collaborative Writing», in EDM, www.educationaldatamining.org, 2010, pp. 257-266.

G. Deeva and J. D. Weerdt, «Understanding Automated Feedback in Learning Processes by Mining Local Patterns», in Business Process Management Workshops, ser. Lecture Notes in Business Information Processing, vol. 342, Springer, 2018, pp. 56-68.

D. Codish, E. Rabin, and G. Ravid, «User behavior pattern detection in unstructured processes - a learning management system case study», Interact. Learn. Environ., vol. 27, no. 5-6, pp. 699-725, 2019.

J. Maldonado-Mahauad, M. Perez-Sanagustin, R. F. Kizilcec, N. Morales, and J. Munoz-Gama, «Mining theory-based patterns from Big data: Identifying self-regulated learning strategies in Massive Open Online Courses», Comput. Hum. Behav., vol. 80, pp. 179-196, 2018.

W. Hachicha, L. Ghorbel, R. Champagnat, C. A. Zayani, and I. Amous, «Using Process Mining for Learning Resource Recommendation: A Moodle Case Study», in Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 25th International Conference KES-2021, Virtual Event / Szczecin, Poland, 8-10 September 2021, J. Watrobski, W. Salabun, C. Toro, C. Zanni-Merk, R. J. Howlett, and L. C. Jain, Eds., ser. Procedia Computer Science, vol. 192, Elsevier, 2021, pp. 853-862. doi: 10.1016/j.procs.2021.08.088. [Online]. Available: https://doi.org/10.1016Zj.procs.2021.08.088.

G. Sedrakyan, J. D. Weerdt, and M. Snoeck, «Process-mining enabled feedback: ’’Tell me what I did wrong” vs. ’’tell me how to do it right”», Comput. Hum. Behav., vol. 57, pp. 352-376, 2016.

M. L. van Eck, X. Lu, S. J. J. Leemans, and W. van der Aalst, «PM2: Process Mining Project Methodology», in CAiSE, ser. Lecture Notes in Computer Science, vol. 9097, Springer, 2015, pp. 297-313.

A. Weijters, W. Aalst, and A. Medeiros, Process Mining with the Heuristics Miner-algorithm. Jan. 2006, vol. 166, pp. 1-34.

The authors declare that there are no conflicts of interest present.