Skip to content
Log in
Sponsored Have an interesting project or opportunity? → Hire me ↗
Sponsored Artificial Intelligence: A Modern Approach, Global Edition Artificial Intelligence: A Modern Approach, Global Edition
#large language models Dataset Open access

Replication Package for Understanding Architectural Degradation. A Large Language Model-assisted Multivocal Literature Review

Noman Ahmad Ruoyu Su Matteo Esposito Valentina Lenarduzzi Andrea Janes Davide Taibi
Sep 2026 · Zenodo (CERN European Organization for Nuclear Research)

Abstract

Replication Package The package contains the datasets, source code, validation results, and generated outputs used throughout the study. It enables researchers to reproduce the literature review, data extraction, coding, validation, and analysis procedures reported in the paper. Note: All the hyperlinks referring to the shared files only work in the local version (downloading the replication package). The online version will not work because the browser cannot locate the referenced local files. License All generated data is provided under DATA_LICENSE Creative Commons 4.0 Attribution License. All scripts are provided under the Script_LICENSE MIT License. Package Structure Replication_Package/ │ ├── README.md ├── Install.md ├── requirements.txt ├── Data License.md ├── Script License.md ├── Understanding Full read and analysis.xlsx ├── Validation understanding architectural degradation.xlsx │ └── Code and outputs/ ├── Title Abstract Round Code and output/ ├── T-A Validation code and Outputs/ ├── Full Read code and outputs/ ├── Validation Full Read outputs/ ├── (Final Round code and outputs) Understanding MLR analysis/ ├── Validation for Final Round (code and outputs)/ └── Figures (Sankey & Heatmap) Data Files 1. Understanding Full read and analysis.xlsx This workbook contains the extracted data and the final coded datasets used throughout the study. Workbook Contents Sheet Name Description Process Documents the complete study workflow, including screening, queries, extraction, and coding procedures followed during the literature review. White (title/abstract) Contains the raw results obtained after the title and abstract screening stage for the white literature (peer-reviewed studies). Grey (title/abstract) Contains the raw results obtained after the title and abstract screening stage for the grey literature. Full read (accepted) Data extraction form containing information extracted from all accepted primary studies during the full-text review. Who Consolidated coding table of the stakeholders associated with software architecture degradation. What Consolidated coding table of the drivers and motivations of software architecture degradation. When Consolidated coding table of the software development lifecycle stages and situations in which software architecture degradation emerges. Where Consolidated coding table of the locations within software systems where software architecture degradation occurs. Why Consolidated coding table of the reasons why practitioners and organizations tolerate or introduce software architecture degradation. How Consolidated coding table of how the identified motivations manifest and evolve into software architecture degradation. 2. Validation understanding architectural degradation.xlsx This workbook contains the validation datasets used to evaluate the reliability and consistency of the coding process. Workbook Contents Sheet Name Description Stats Summary statistics, agreement rates, disagreement counts, and validation results reported in the study. Title abstract (Validation) Validation dataset for the title and abstract screening stage, including sampled studies, validation decisions, and adjudicated outcomes. Who (full read) Validation of the full-text review coding for Who by LLM models and human evaluators. What (full read) Validation of the full-text review coding for What by LLM models and human evaluators. When (full read) Validation of the full-text review coding for When by LLM models and human evaluators. Where (full read) Validation of the full-text review coding for Where by LLM models and human evaluators. Why (full read) Validation of the full-text review coding for Why by LLM models and human evaluators. How (full read) Validation of the full-text review coding for How by LLM models and human evaluators. Who (final round validation) Final-round validation dataset for the coding of stakeholders (Who), including LLM and human validation What (final round validation) Final-round validation dataset for the coding of drivers and motivations (What), including LLM and human validation When (final round validation) Final-round validation dataset for the coding of lifecycle stages (When), including LLM and human validation Where (final round validation) Final-round validation dataset for the coding of locations (Where), including LLM and human validation Why (final round validation) Final-round validation dataset for the coding of motivations (Why), including LLM and human validation How (final round validation) Final-round validation dataset for the coding of manifestations (How), including LLM and human validation Code and Outputs The Code and outputs directory contains the source code, validation scripts, intermediate files, and generated outputs used throughout the study. Folder Description Title Abstract Round Code and output Source code used for the title and abstract screening stage and all the raw outputs. T-A Validation code and Outputs Validation scripts and generated outputs for the title and abstract screening stage. Full Read code and outputs Source code used during the full-text review, data extraction, coding, and analysis. Validation Full Read outputs Validation scripts and generated outputs for the full-text review stage. (Final Round code and outputs) understanding MLR analysis Source code used to generate the final coded datasets, descriptive statistics, tables, and figures reported in the manuscript. Validation for Final Round (code and outputs) Scripts and outputs used for the final validation of the coded datasets before producing the final study results. Figrues (Sankey & Heatmap) Scripts and outputs used in paper for sankey and heatmap diagrams. Replication of the Results This section describes the procedure for reproducing the title and abstract screening, full-text analysis, LLM-based extraction and coding, validation, and final analysis performed in this study. The workflow uses Marco-o1 as the primary model and Mistral, Qwen, and Llama as independent validation models. The computationally intensive LLM experiments were originally executed on a supercomputer. Setting Up the Environment Follow the instructions in INSTALL.md to configure the required software environment. The required Python dependencies are listed in requirements.txt. After configuring the environment, execute the replication stages below in the specified order. 1. Title and Abstract Screening Use the scripts and input data provided in: Code and outputs/Title Abstract Round Code and output/ The scripts process the retrieved white and grey literature and generate the title and abstract screening decisions. The corresponding outputs are provided in the White (title/abstract) and Grey (title/abstract) sheets of: Understanding Full read and analysis.xlsx Studies accepted at this stage proceed to validation and full-text review. 2. Title and Abstract Validation Use the scripts provided in: Code and outputs/T-A Validation code and Outputs/ The title and abstract screening decisions are independently validated using Mistral, Qwen, and Llama. Human validation is performed on the selected validation sample as described in the paper. The resulting validation data are provided in the Title abstract (Validation) sheet of: Validation understanding architectural degradation.xlsx After validation and adjudication, the accepted studies are used as input to the full-text analysis. 3. Preparing the Full-Text Input The full-text analysis requires the PDF files of the studies accepted after title and abstract screening and validation. The PDFs are processed using the scripts provided in: Code and outputs/Full Read code and outputs/ The scripts extract the textual content of each study and provide it to the LLM analysis pipeline. Note: The full-text PDFs are not distributed with the replication package. Researchers wishing to reproduce this stage must obtain the corresponding papers independently and place them in the input location expected by the scripts. 4. Full-Text Analysis with Marco-o1 Use the scripts provided in: Code and outputs/Full Read code and outputs/ Marco-o1 is used as the primary model to analyze the accepted studies and extract information for the six dimensions investigated in the study: Who, What, When, Where, Why, and How. For the original supercomputer execution, the general command structure is: sbatch "MODEL_NAME" The exact Python and batch scripts used in the experiments are provided in the same directory. The resulting extracted data are consolidated in the Full read (accepted) sheet of: Understanding Full read and analysis.xlsx 5. Full-Text Validation After generating the Marco-o1 full-text results, independently validate them using Mistral, Qwen, and Llama. The validation scripts and generated outputs are provided in: Code and outputs/Validation Full Read outputs/ Each validator is executed separately using the corresponding model configuration. For supercomputer execution, the general command structure is: sbatch "MODEL_NAME" The LLM validation results are combined with the human validation decisions according to the validation and adjudication procedure described in the paper. The resulting data are available in the corresponding Who, What, When, Where, Why, and How (full read) sheets of: Validation understanding architectural degradation.xlsx The corresponding agreement statistics are reported in the Stats sheet. 6. Final-Round Analysis After completing the full-text analysis and validation, use the validated results as input to the final coding stage. The scripts and outputs are provided in: Code and outputs/(Final Round code and outputs) Understanding MLR analysis/ Marco-o1 is used to consolidate the

DOI
View source

Similar papers

#computer vision Open access Jun 2016

Software Development in Startup Companies: The Greenfield Startup Model

Software startups are newly created companies with no operating history and oriented towards producing cutting-edge products. However, despite the increasing importance of startups in the economy, few scientific studies attempt to address software engineering issues, especially for early-stage startups. If anything, startups need engineering practices of the same level or better than those of larger companies, as their time and resources are more scarce, and one failed project can put them out of business. In this study we aim to improve understanding of the software development strategies employed by startups. We performed this state-of-practice investigation using a grounded theory approach. We packaged the results in the Greenfield Startup Model (GSM), which explains the priority of startups to release the product as quickly as possible. This strategy allows startups to verify product and market fit, and to adjust the product trajectory according to early collected user feedback. The need to shorten time-to-market, by speeding up the development through low-precision engineering activities, is counterbalanced by the need to restructure the product before targeting further growth. The resulting implications of the GSM outline challenges and gaps, pointing out opportunities for future research to develop and validate engineering practices in the startup context.

Carmine Giardino, Nicolò Paternoster, M. Unterkalmsteiner et al. · 179 citations · ⚡14
#computer vision Open access Oct 2016

Software Startups - A Research Agenda

Software startup companies develop innovative, software-intensive products within limited timeframes and with few resources, searching for sustainable and scalable business models. Software startup ...

M. Unterkalmsteiner, P. Abrahamsson, Xiaofeng Wang et al. · 157 citations · ⚡17
#machine learning Review Open access Oct 2016

“Failures” to be celebrated: an analysis of major pivots of software startups

In the context of software startups, project failure is embraced actively and considered crucial to obtain validated learning that can lead to pivots. A pivot is the strategic change of a business concept, product or the different elements of a business model. A better understanding is needed on different types of pivots and different factors that lead to failures and trigger pivots, for software entrepreneurial teams to make better decisions under chaotic and unpredictable environment. Due to the nascent nature of the topic, the existing research and knowledge on the pivots of software startups are very limited. In this study, we aimed at identifying the major types of pivots that software startups make during their startup processes, and highlighting the factors that fail software projects and trigger pivots. To achieve this, we conducted a case survey study based on the secondary data of the major pivots happened in 49 software startups. 10 pivot types and 14 triggering factors were identified. The findings show that customer need pivot is the most common among all pivot types. Together with customer segment pivot, they are common market related pivots. The major product related pivots are zoom-in and technology pivots. Several new pivot types were identified, including market zoom-in, complete and side project pivots. Our study also demonstrates that negative customer reaction and flawed business model are the most common factors that trigger pivots in software startups. Our study extends the research knowledge on software startup pivot types and pivot triggering factors. Meanwhile it provides practical knowledge to software startups, which they can utilize to guide their effective decisions on pivoting.

Sohaib Shahid Bajwa, Xiaofeng Wang, Anh Nguyen-Duc et al. · 127 citations · ⚡15
#computer vision Review Open access May 2015

A survey study on major technical barriers affecting the decision to adopt cloud services

In the context of cloud computing, risks associated with underlying technologies, risks involving service models and outsourcing, and enterprise readiness have been recognized as potential barriers for the adoption. To accelerate cloud adoption, the concrete barriers negatively influencing the adoption decision need to be identified. Our study aims at understanding the impact of technical and security-related barriers on the organizational decision to adopt the cloud. We analyzed data collected through a web survey of 352 individuals working for enterprises consisting of decision makers as well as employees from other levels within an organization. The comparison of adopter and non-adopter sample reveals three potential adoption inhibitor, security, data privacy, and portability. The result from our logistic regression analysis confirms the criticality of the security concern, which results in an up to 26-fold increase in the non-adoption likelihood. Our study underlines the importance of the technical and security perspectives for research investigating the adoption of technology.

Nattakarn Phaphoom, Xiaofeng Wang, S. Samuel et al. · 111 citations · ⚡8
#computer vision Conference Open access Dec 2013

Affordable and Energy-Efficient Cloud Computing Clusters: The Bolzano Raspberry Pi Cloud Cluster Experiment

We present our ongoing work building a Raspberry Pi cluster consisting of 300 nodes. The unique characteristics of this single board computer pose several challenges, but also offer a number of interesting opportunities. On the one hand, a single Raspberry Pi can be purchased cheaply and has a low power consumption, which makes it possible to create an affordable and energy-efficient cluster. On the other hand, it lacks in computing power, which makes it difficult to run computationally intensive software on it. Nevertheless, by combining a large number of Raspberries into a cluster, this drawback can be (partially) offset. Here we report on the first important steps of creating our cluster: how to set up and configure the hardware and the system software, and how to monitor and maintain the system. We also discuss potential use cases for our cluster, the two most important being an inexpensive and green test bed for cloud computing research and a robust and mobile data center for operating in adverse environments.

P. Abrahamsson, S. Helmer, Nattakarn Phaphoom et al. · 110 citations · ⚡7
#computer vision Book Open access Mar 2017

On the Unhappiness of Software Developers

The happy-productive worker thesis states that happy workers are more productive. Recent research in software engineering supports the thesis, and the ideal of flourishing happiness among software developers is often expressed among industry practitioners. However, the literature suggests that a cost-effective way to foster happiness and productivity among workers could be to limit unhappiness. Psychological disorders such as job burnout and anxiety could also be reduced by limiting the negative experiences of software developers. Simultaneously, a baseline assessment of (un)happiness and knowledge about how developers experience it are missing. In this paper, we broaden the understanding of unhappiness among software developers in terms of (1) the software developer population distribution of (un)happiness, and (2) the causes of unhappiness while developing software. We conducted a large-scale quantitative and qualitative survey, incorporating a psychometrically validated instrument for measuring (un)happiness, with 2 220 developers, yielding a rich and balanced sample of 1318 complete responses. Our results indicate that software developers are a slightly happy population, but the need for limiting the unhappiness of developers remains. We also identified 219 factors representing causes of unhappiness while developing software. Our results, which are available as open data, can act as guidelines for practitioners in management positions and developers in general for fostering happiness on the job. We suggest considering happiness in future studies of both human and technical aspects in software engineering.

D. Graziotin, Fabian Fagerholm, Xiaofeng Wang et al. · 84 citations · ⚡6

Related blog posts