Towards LLM-Assisted Business Process Modeling in an Industrial Modeling Tool: An Experience Report
Abstract
Business Process Modeling (BPM) is a key and complex activity in Software and Systems Engineering. As a consequence, any relevant assistance integrated into the BPM tooling can be highly beneficial. Historically, Natural Language Processing (NLP) techniques have already been used in this context. However, the recent developments around Large Language Models (LLMs) offer promising new perspectives. In this paper, we report on our practical experience of integrating state-of-the-art LLMs within the industrial modeling tool Modelio. In particular, we experiment with the automated generation of BPMN models from natural-language instructions. To this end, we consider two different solutions: 1) The direct generation of BPMN creation scripts that call the modeling tool API; 2) The generation of intermediate representations that use a BPMN helper library. We first validate our approach with 7 industrial partners on 24 scenarios from the MATISSE collaborative project. In addition, we evaluate this approach with 3 different LLMs across 55 BPM scenarios from an open dataset. We demonstrate that using an intermediate representation (expressed in a compact DSL) globally improves performance, especially in more complex cases. We also share practical lessons learned and guidelines for the potential adoption of our approach by practitioners and researchers.