This work proposes and develops a multi-model unified platform to generate and execute code based on natural language prompts and presents practitioners feedback and insights into the use of LLMs in software development, including their strengths and weaknesses, key aspects overlooked by benchmarks and metrics.
Z. Rasheed, Muhammad Waseem, Kai-Kristian Kemell et al.· arXiv.org· 18 citations· ⚡2
A system that uses Large Language Models (LLMs) to automate the API-first development of RESTful microservices and assists in creating OpenAPI specification, generating server code from it, and refining the code through a feedback loop that analyzes execution logs and error messages is presented.
Saurabh Chauhan, Z. Rasheed, Malik Abdul Sami et al.· International Conference on...· 16 citations· ⚡1
A large language models based multi-agent system enables precise task execution and inter-agent collaboration, addressing the challenges of refactoring in functional programming.
Shahbaz Siddeeq, Z. Rasheed, Malik Abdul Sami et al.· arXiv.org· 1 citation
The recent advancements of Small Language Models (SLMs) have opened new possibilities for efficient code generation. SLMs offer lightweight and cost-effective alternatives to Large Language Models (LLMs), making them attractive for use in resource-constrained environments. However, empirical understanding of SLMs, particularly their capabilities, limitations, and performance trade-offs in code generation remains limited. This study presents a comprehensive empirical evaluation of 20 open-source SLMs ranging from 0.4B to 10B parameters on five diverse code-related benchmarks (HumanEval, MBPP, Mercury, HumanEvalPack, and CodeXGLUE). The models are assessed along three dimensions: i) functional correctness of generated code, ii) computational efficiency and iii) performance across multiple programming languages. The findings of this study reveal that several compact SLMs achieve competitive results while maintaining a balance between performance and efficiency, making them viable for deployment in resource-constrained environments. However, achieving further improvements in accuracy requires switching to larger models. These models generally outperform their smaller counterparts, but they require much more computational power. We observe that for 10% performance improvements, models can require nearly a 4x increase in VRAM consumption, highlighting a trade-off between effectiveness and scalability. Besides, the multilingual performance analysis reveals that SLMs tend to perform better in languages such as Python, Java, and PHP, while exhibiting relatively weaker performance in Go, C++, and Ruby. However, statistical analysis suggests these differences are not significant, indicating a generalizability of SLMs across programming languages. Based on the findings, this work provides insights into the design and selection of SLMs for real-world code generation tasks.
Mahade Hasan, Muhammad Waseem, Kai-Kristian Kemell et al.· Journal of Systems and Softw...· 16 citations
Reach audiences
Advertise in front of researchers, engineers, and readers.
The use of Large Language Models (LLMs) for autonomous code generation is gaining attention in emerging technologies. As LLM capabilities expand, they offer new possibilities such as code refactoring, security enhancements, and legacy application upgrades. Many outdated web applications pose security and reliability challenges, yet companies continue using them due to the complexity and cost of upgrades. To address this, we propose an LLM-based multi-agent system that autonomously upgrades legacy web applications to the latest versions. The system distributes tasks across multiple phases, updating all relevant files. To evaluate its effectiveness, we employed Zero-Shot Learning (ZSL) and One-Shot Learning (OSL) prompts, applying identical instructions in both cases. The evaluation involved updating view files and measuring the number and types of errors in the output. For complex tasks, we counted the successfully met requirements. The experiments compared the proposed system with standalone LLM execution, repeated multiple times to account for stochastic behavior. Results indicate that our system maintains context across tasks and agents, improving solution quality over the base model in some cases. This study provides a foundation for future model implementations in legacy code updates. Additionally, findings highlight LLMs' ability to update small outdated files with high precision, even with basic prompts. The source code is publicly available on GitHub: https://github.com/alasalm1/Multi-agent-pipeline.
Valtteri Ala-Salmi, Z. Rasheed, Malik Abdul Sami et al.· International Conference on...· 4 citations
This article analyzes the flow-debt tradeoffs associated with VC and identifies and explains how current model, platform, and hardware limitations contribute to these issues, and proposes countermeasures to address them, informing research and practice towards more sustainable VC approaches.
Muhammad Waseem, Aakash Ahmad, Kai-Kristian Kemell et al.· arXiv.org· 4 citations
This is one of the first reviews to integrate peer-reviewed and grey literature on vibe coding under a single documented protocol and is strongest for prototyping and user-interface work and weakest for production, data-intensive, and safety-critical use, and tool visibility does not imply effectiveness.
Shahbaz Siddeeq, Muhammad Waseem, Kai-Kristian Kemell et al.· arXiv.org· 0 citations
The results indicate that the proposed approach reduces the effort required for manual dataset construction while preserving data quality through mandatory human validation, and highlights the effectiveness of hybrid automation workflows in accelerating fine-tuning dataset preparation without compromising reliability.
Niklas Suvitie, Mika Saari, Pekka Abrahamsson· International Convention on...· 1 citation
A variability-preserving imputation method is introduced that augments linear interpolation with locally adaptive stochastic noise, retaining physiological dynamics essential for accurate forecasting in short-term Heart Rate Variability forecasting.
Luukas Peräkylä, F. Sohrab, Ville Hautamäki et al.· arXiv.org· 0 citations
An experience report from a small full-stack team that applied contextual prompting and explicit architectural constraints to build a multi-project agent learning platform designed for sustained, production-oriented use and an academic retrieval-augmented generation system is presented.
Md Nasir Uddin Shuvo, M. Islam, Mahade Hasan et al.· arXiv.org· 0 citations
Automated authoring of Gherkin Behavior-Driven Development (BDD) acceptance criteria remains a manual bottleneck in requirements engineering. This study investigates whether epic-organized LLM-generated Gherkin produces higher quality and coverage than requirement-aligned generation. We compare our Timeless (an epic-organized LLM pipeline) approach against a naive large language model (LLM) baseline on four requirements documents (107 requirements) from the PURE dataset. Evaluation covers structural metrics, automated requirement coverage via TF-IDF and dense embeddings, and blind expert assessment by four researchers. In our evaluation, the JSON-constrained pipeline produced structurally valid scenarios across all generated outputs, while the zero-shot baseline achieved 99% structural validity. Semantic coverage was comparable to the baseline, with Timeless achieving 94.3% semantic Requirement Coverage Rate compared with 92.9% for the baseline. TF-IDF produced lower coverage scores for the epic-organized output, suggesting that lexical metrics may miss coverage when scenarios paraphrase requirements at a higher level of abstraction. Expert raters prefer the epic-organized strategy on Correctness (4.61 vs 4.14), Executability (4.61 vs 4.07), and Completeness (4.31 vs 3.50). Overall, the results suggest that epic-organized generation can improve perceived Gherkin quality while maintaining comparable semantic coverage, although broader replication is needed before generalizing this finding.
Shahbaz Siddeeq, M. Abbasi, Jussi Rasku et al.· 0 citations
The use of large language models to automatically improve the user story quality in Austrian Post Group IT agile teams is explored, with a reference model for an Autonomous LLM-based Agent System developed and implemented at the company.
Zheying Zhang, M. Rayhan, Tomas Herda et al.· International Conference on...· 48 citations· ⚡4
What if pathology foundation models could do more with less? GigaPath-Flash and GigaTIME-Flash cut computational demands while maintaining strong performance, opening the door to larger studies and broader exploration. The post GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models appeared first on Microsoft Research.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
