Cancer-ACPNet: A Two-Stage ESM-2 and Capsule Network Framework for Anticancer Peptide Screening and Cancer-Type Activity Prediction.

Abstract

Anticancer peptides (ACPs) are being widely studied as potential peptide-based anticancer agents because of their ability to interact selectively with cancer cells. Despite this promise, computational identification of ACPs and prediction of their cancer-type activity remain difficult, mainly because available peptide datasets are small, imbalanced, and biologically heterogeneous. In this study, we propose a two-stage framework for ACP identification and functional cancer-type annotation. In Stage 1, ESM-2 sequence embeddings are combined with BLOSUM62 and AAIndex-based descriptors and processed through a capsule-enhanced classifier to distinguish ACPs from non-ACPs. In Stage 2, the learned ACP representation is reused for one-vs-all prediction across seven cancer types using an ensemble of MLP heads with meta-stacking, calibration, class-aware thresholds, and MC-dropout inference. The Cancer-ACPNet Stage-1 model achieved accuracies of 84.08%, 96.74%, and 84.75% on Set 1, Set 2, and Set 3, respectively, with balanced F1-score and MCC values. For Stage 2, the model achieved strong seven cancer-type activity prediction, with macro-average accuracy, F1-score, MCC, and AUC values of 89.37%, 88.18%, 77.79%, and 92.77%, respectively. Model explanation analyses based on saliency, SHAP, residue enrichment, mutation validation, meta-stacker coefficients, and end-to-end residue perturbation identified residue regions and physicochemical patterns that influenced the predictions. These results are interpreted as model-level explanations requiring further experimental validation. In general, this framework provides an efficient means of screening potential ACPs and assigning probable cancer-type activities, which helps to prioritize candidate peptides to be experimentally tested.