Skip to content
Log in
#diffusion models #protein folding Dataset Open access

Assessing State-Specific Accuracy of Cofolding Models for Kinases and GPCRs

Leon Obendorf Niklas Piet Doering Petra Knaus Gerhard Wolber
Aug 2026 · Zenodo (CERN European Organization for Nuclear Research)

Abstract

# Cofolding benchmark: structures, alignments and analysis scripts Supplementary data for *Benchmarking Protein–Ligand Cofolding Models: Correct LigandPlacement Is Decoupled from Accurate Protein Conformation*. Seven protein–ligand complexes, three class A GPCRs and four kinase systems, were predictedwith AlphaFold3, Boltz-2, Chai-1 and RoseTTAFold3 under different MSA and template settingsand compared with the experimental structures. Everything needed to repeat that comparison ishere: the experimental references, the predicted structures, the state-specific alignmentsused to bias the predictions, and the analysis scripts. Tables and figures are not deposited, only the inputs they are made from. The superposedstructures are included, since every reported RMSD was measured on them. ## Contents ```0_reference_structures/ experimental structure of each complex, and the ligand SMILES1_Structures/ the predicted complexes, per target and method2_input_preparation/ scripts that build the model inputs; MSAs/ holds the alignments3_predictions/ how each model was run4_alignment/ superposition onto the reference, and the superposed structures5_ligand_rmsd/ ligand RMSD in the protein reference frame6_extract_metrics/ RMSD, pLDDT, ipTM and PAE collected into per-target tables7_core_analysis_plots/ Figure 48_rmsd_scatter_plots/ Figure 5, Figure S4, and the correlations quoted in the text9_plddt_plots/ Figures S1 and S210_multiseed/ Figure S3, the five-seed repetition11_interaction_fingerprints/ Figures S5-S10, interaction fingerprints and pose validity``` ### Reference structures `0_reference_structures/` holds one folder per target with the experimental complex as mmCIFand as the PDB file the analysis reads, together with the ligand SMILES. The entries are 8ZMG(5-HT2A), 8UGW (A2AAR), 8Y45 (DOR), 9L04 (ALK2 with RK-783), 6UNQ (ALK2 with AMPPNP), 9DMI(LRRK2) and 8TSD (PI3K). ### Predicted structures `1_Structures/` is arranged as target, method, condition: ```1_Structures/GPCR_5HT2a/Alphafold/5ht2a_inactive_custom_templates_custommsa/ seed-0_sample-0/ ... seed-0_sample-4/ one prediction each, .cif and confidences``` The same layout holds for `GPCR_AA2A`, `GPCR_DOR`, `Kinase_ALK2_ATP`, `Kinase_ALK2_RK783`,`Kinase_LRRK2`, `Kinase_PI3K`, and for `boltz`, `Chai1` and `RF3`. Every condition contributesfive diffusion samples of one seed. These are the predictions as the models wrote them; thesuperposed copies live in `4_alignment/superposed_structures/` and, for the five-seed set, in`10_multiseed/superposed_structures/`. ### Biasing alignments `2_input_preparation/MSAs/` holds one alignment per target, named for the conformational stateit biases towards: | file | target | sequences ||---|---|---|| `5HT2A_inactive.a3m` | 5-HT2A | 208 || `A2AAR_active.a3m` | A2AAR | 301 || `DOR_active.a3m` | DOR | 292 || `ALK2_active.a3m` | ALK2, both ligands | 1000 || `LRRK2_dfgout.a3m` | LRRK2 | 669 || `PI3K_dfgout.a3m` | PI3K | 2 | The custom templates are public PDB entries, listed by accession code in Table S2 of theSupporting Information. The scripts beside the alignments turn them into the input files eachmodel expects: AlphaFold3 job JSONs (`create_alphafold3_jobs_new_a3m.py`, and`create_aa2a_active_json.py` for A2AAR), template mmCIFs with the index arrays AlphaFold3needs (`fix_templates_bio.py`), Chai-1 template M8 files (`create_m8_gpcr_chai1.py`), RF3 jobsderived from the AlphaFold3 ones (`create_rf3_jsons_from_af3.py`) and the A3M rewriting RF3requires (`normalize_a3m_for_rf3.py`). ## Requirements `requirements.txt` lists the python packages. The analysis also needs PyMOL for thesuperposition and MDAnalysis for the ligand RMSD; folder 11 additionally needs ProLIF, RDKit,PoseBusters, Open Babel, PDBFixer and OpenMM. The cofolding models themselves are only neededto repeat the predictions, not to reproduce the analysis. ## How to run it Folders 4 to 11 find their inputs inside this deposit, so no paths need editing. Each stepwrites what the next one reads, so run them in this order. Steps 4 and 5 rebuild thesuperpositions and RMSD logs; to work from the deposited superpositions instead, start atstep 6. ```bash# 4-5: superpose each prediction onto its reference, then measure the ligandpython 4_alignment/alignment_pymol.py --savepython 5_ligand_rmsd/lig_rmsd.py # 6: collect RMSD, pLDDT and the confidence metrics into the per-target tablespython 6_extract_metrics/extract_pdb_data.py # the three GPCRspython 6_extract_metrics/extract_kinase_data.py # LRRK2 and PI3Kpython 6_extract_metrics/extract_alk2_data.py # ALK2 with ATP and with RK-783python 6_extract_metrics/extract_confidence.py # ipTM, pTM, interface PAE, overall PAE # 7: Figure 4, and the per-model tables that steps 8 and 9 read.# AXIS_BREAK=3.0 gives the axis break of the published figure; the GPCR panels of# Figure 4 additionally colour best and worst by ligand RMSD.AXIS_BREAK=3.0 python 7_core_analysis_plots/plot_core_analysis_subpart_final.pyAXIS_BREAK=3.0 BEST_BY_LIGAND_RMSD=1 PLOT_OUTDIR=plots_ligbest \ python 7_core_analysis_plots/plot_core_analysis_subpart_final.py # 8: Figure 5, Figure S4, and the correlations quoted in the textpython 8_rmsd_scatter_plots/plot_template_vs_ligand_rmsd.py \ --after-dir 6_extract_metrics/result_tables/after-plotting \ --output-dir plots/scatter --rmsd-cutoff 2.0python 8_rmsd_scatter_plots/plot_plddt_vs_protein_rmsd.pypython 8_rmsd_scatter_plots/plot_plddt_combined.pypython 8_rmsd_scatter_plots/correlation_table.py # 9: Figures S1 and S2python 9_plddt_plots/plot_atomwise_plddt_from_after_csv.py \ --after-dir 6_extract_metrics/result_tables/after-plotting # 10: Figure S3, the five-seed repetitionpython 10_multiseed/aggregate_from_derived.pypython 10_multiseed/create_comparison_heatmaps.py # 11: Figures S5-S10python 11_interaction_fingerprints/prolif_fingerprints.pypython 11_interaction_fingerprints/plot_interaction_heatmaps.py``` Figures land in `plots/`, the multi-seed ones in `10_multiseed/plots/`, and the tables in`6_extract_metrics/result_tables/` and `10_multiseed/result_tables/`. Before step 10, set ` ` at the top of `aggregate_from_derived.py` to the PyMOLexecutable. Without it the script still writes its tables, but the protein RMSD column staysempty. ## How the numbers are defined Predictions are superposed onto the reference on backbone atoms (N, CA, C, O) of a fixedselection, with PyMOL's iterative outlier rejection switched off, so the RMSD covers the wholeselection. For the GPCRs and LRRK2 that selection is the folded core, which leaves out theflexible termini. The ligand RMSD is then measured in that same frame, without superposing theligands onto one another, so it reports where the ligand sits rather than how similar itsinternal geometry is. The extraction scripts add the mean protein and ligand pLDDT from themodel files, and read ipTM, pTM and PAE out of each method's own output. Everything after thatis plotting. ## Steps that were commands rather than scripts The alignments in `2_input_preparation/MSAs/` were built from FASTA sets of structures sharingone conformational state: ```bashmmseqs createdb mmseqs createdb mmseqs search --max-seqs 1000 --threads 8mmseqs result2msa --msa-format-mode 5reformat.pl -M first -r a3m a3m ``` RF3 predictions: ```bashrf3 fold inference_engine=rf3 inputs= ckpt_path= \ diffusion_batch_size=5 seed=0``` Boltz-2 predictions, where the MSA route depends on the condition: `--use_msa_server` for thedefault runs, an explicit A3M for the biased ones, and no MSA at all for the third: ```bashboltz predict --use_msa_server --diffusion_samples 5``` ## Placeholders The scripts in folders 2 and 3 read and write wherever the predictions are run, so they carryplaceholders in angle brackets that need to be filled in. Each script names the ones it usesin a comment under its docstring. | placeholder | what belongs there ||---|---|| ` ` | directory the predictions are written to || ` `, ` `, ` ` | AlphaFold3 job inputs, outputs and model weights || ` `, ` ` | AlphaFold3 sequence databases and the GPU to use || ` `, ` ` | working directory and conda installation of the run scripts || ` ` | PyMOL executable, in `10_multiseed/aggregate_from_derived.py` || ` ` | interpreter of the environment with ProLIF, RDKit and PoseBusters | ## Relation to the earlier version The alignment, ligand RMSD, metric extraction and plotting scripts were revised during peerreview: the superposition runs without iterative outlier rejection on fixed atom selections,and the ligand RMSD is measured in the protein reference frame without refitting. EveryRMSD-derived number and figure in the paper comes from these versions, and the earliervariants are not part of this deposit.

DOI
View source

Similar papers

#computer vision Open access Jun 2016

Software Development in Startup Companies: The Greenfield Startup Model

Software startups are newly created companies with no operating history and oriented towards producing cutting-edge products. However, despite the increasing importance of startups in the economy, few scientific studies attempt to address software engineering issues, especially for early-stage startups. If anything, startups need engineering practices of the same level or better than those of larger companies, as their time and resources are more scarce, and one failed project can put them out of business. In this study we aim to improve understanding of the software development strategies employed by startups. We performed this state-of-practice investigation using a grounded theory approach. We packaged the results in the Greenfield Startup Model (GSM), which explains the priority of startups to release the product as quickly as possible. This strategy allows startups to verify product and market fit, and to adjust the product trajectory according to early collected user feedback. The need to shorten time-to-market, by speeding up the development through low-precision engineering activities, is counterbalanced by the need to restructure the product before targeting further growth. The resulting implications of the GSM outline challenges and gaps, pointing out opportunities for future research to develop and validate engineering practices in the startup context.

Carmine Giardino, Nicolò Paternoster, M. Unterkalmsteiner et al. · 179 citations · ⚡14
#computer vision Open access Oct 2016

Software Startups - A Research Agenda

Software startup companies develop innovative, software-intensive products within limited timeframes and with few resources, searching for sustainable and scalable business models. Software startup ...

M. Unterkalmsteiner, P. Abrahamsson, Xiaofeng Wang et al. · 157 citations · ⚡17
#machine learning Review Open access Oct 2016

“Failures” to be celebrated: an analysis of major pivots of software startups

In the context of software startups, project failure is embraced actively and considered crucial to obtain validated learning that can lead to pivots. A pivot is the strategic change of a business concept, product or the different elements of a business model. A better understanding is needed on different types of pivots and different factors that lead to failures and trigger pivots, for software entrepreneurial teams to make better decisions under chaotic and unpredictable environment. Due to the nascent nature of the topic, the existing research and knowledge on the pivots of software startups are very limited. In this study, we aimed at identifying the major types of pivots that software startups make during their startup processes, and highlighting the factors that fail software projects and trigger pivots. To achieve this, we conducted a case survey study based on the secondary data of the major pivots happened in 49 software startups. 10 pivot types and 14 triggering factors were identified. The findings show that customer need pivot is the most common among all pivot types. Together with customer segment pivot, they are common market related pivots. The major product related pivots are zoom-in and technology pivots. Several new pivot types were identified, including market zoom-in, complete and side project pivots. Our study also demonstrates that negative customer reaction and flawed business model are the most common factors that trigger pivots in software startups. Our study extends the research knowledge on software startup pivot types and pivot triggering factors. Meanwhile it provides practical knowledge to software startups, which they can utilize to guide their effective decisions on pivoting.

Sohaib Shahid Bajwa, Xiaofeng Wang, Anh Nguyen-Duc et al. · 127 citations · ⚡15
#computer vision Review Open access May 2015

A survey study on major technical barriers affecting the decision to adopt cloud services

In the context of cloud computing, risks associated with underlying technologies, risks involving service models and outsourcing, and enterprise readiness have been recognized as potential barriers for the adoption. To accelerate cloud adoption, the concrete barriers negatively influencing the adoption decision need to be identified. Our study aims at understanding the impact of technical and security-related barriers on the organizational decision to adopt the cloud. We analyzed data collected through a web survey of 352 individuals working for enterprises consisting of decision makers as well as employees from other levels within an organization. The comparison of adopter and non-adopter sample reveals three potential adoption inhibitor, security, data privacy, and portability. The result from our logistic regression analysis confirms the criticality of the security concern, which results in an up to 26-fold increase in the non-adoption likelihood. Our study underlines the importance of the technical and security perspectives for research investigating the adoption of technology.

Nattakarn Phaphoom, Xiaofeng Wang, S. Samuel et al. · 111 citations · ⚡8
#computer vision Open access Feb 2018

Lean Internal Startups for Software Product Innovation in Large Companies: Enablers and Inhibitors

To compete in this age of disruption, large companies cannot rely on cost efficiency, lead time reduction and quality improvement. They are now looking for ways to innovate like startups. Meanwhile, the awareness and use of the Lean startup approach have grown rapidly amongst the software startup community in recent years. This study investigates how Lean internal startup facilitates software product innovation in large companies and identifies its enablers and inhibitors. A multiple case study approach is followed in the investigation. Two software product innovation projects from two large companies are examined, using a conceptual framework that is based on the method-in-action framework and extended with the previously developed Lean-Internal Corporate Venture model. Seven face-to-face in-depth interviews of the employees with different roles are conducted. Within-case analysis and cross-case comparison are applied to draw the findings from the cases. A generic process flow summarises the common key processes of Lean internal startups. The findings suggest that an internal startup that is initiated management or employees faces different challenges. A list of enablers of applying Lean startup in large companies are identified, including top management support and cross-functional team. Both cases face different inhibitors due to the different process of inception, objective of the team and type of the product. Our contributions are threefold. First, this study is one of the first attempt to investigate the use of Lean startup approach in large companies empirically. Second, the study shows the potential of the method-in-action framework to investigate the Lean startup approach in non-startup context. The third is a general process of Lean internal startup and the evidence of the enablers and inhibitors of implementing it, which are both theory-informed and empirically grounded.

Henry Edison, Nina M. Smørsgård, Xiaofeng Wang et al. · 78 citations · ⚡6
#computer vision Conference Sep 2010

Exploring the Sources of Waste in Kanban Software Development Projects

The application of agile software methods and more recently the integration of Lean practices contribute to the trend of continuous improvement in the software industry. One such area warranting proper empirical evidence is a project’s operational efficiency when using the Kanban method. This short paper takes a new angle and explores waste in the Kanban-driven software development project context. A preliminary research model is presented for helping the consequent replication of the study. The results from the empirical analysis suggest Kanban can be an effective method in visualizing and organizing the current work, but does not prevent waste from creeping in, although the overall project outcome may be successful.

Marko Ikonen, Petri Kettunen, Nilay V. Oza et al. · 67 citations · ⚡9

Related blog posts