Specializing Large-Language Models for Collaboration Workflow Generation in Software Engineering

Abstract

Automated problem solving through teams of Large Language Model (LLM)-based agents has recently made remarkable progress, demonstrating the potential of collaborative approaches for software engineering tasks. Existing approaches generally follow one of two paradigms: they either solve tasks directly without generating an explicit collaboration workflow artifact, or they rely on simple lightweight collaboration artifacts. We hypothesize that generating expressive and formal collaboration workflow artifacts with rich control-flow semantics can improve the orchestration of specialized AI agents for complex software engineering tasks. This thesis investigates methods for specializing LLMs to automatically generate such workflow artifacts from natural language task descriptions. As a first step, we adopt SysML v2 action flows as a formal workflow representation and construct a large-scale dataset of approximately 75,000 collaboration workflows by transforming heterogeneous workflow representations into SysML v2 models. This dataset provides the foundation for specializing and evaluating LLMs on workflow generation tasks. Building on this work, the current research focuses on developing a benchmark for collaboration workflow generation and defining evaluation metrics that compare semantically equivalent workflows despite structural differences. The final objective of this thesis is to develop specialized LLMs capable of reliably generating formal collaboration workflow artifacts that orchestrate teams of AI agents for complex software engineering tasks.