Robust Transcription Factor Binding Site Prediction and Explainability Using a Heterogeneous Mixture of Experts Architecture
Transcription Factor Binding Site (TFBS) prediction is central to understanding gene regulation and various biological processes. This study introduces HetMoE, a heterogeneous, embedding-gated Mixture-of-Experts for TFBS prediction. A gating network operates on the embeddings produced by a pool of complementary expert backbones (a modified-DeepBIND convolutional network, DeepSEA, and DanQ, with a fine-tuned DNABERT-6 genomic language model as an optional expert), so that models of different architectures are combined and weighted on a per-input basis. Models are trained against GC- and repeat-matched real genomic negatives, a fair protocol that avoids the dinucleotide-shuffle artifact, and evaluated with a balanced-test-set protocol (deterministic inference, B=1000 paired bootstrap and Analysis of Variance (ANOVA)) on in-distribution and out-of-distribution (OOD) factors. HetMoE attains the best in-distribution performance (mean Area Under the Curve (AUC) 0.881) and, on a held-out set stratified by DNA-binding-domain family, surpasses fine-tuned DNABERT-6 on the motif-bearing OOD mean across three random seeds (0.821±0.005 vs. 0.799±0.008, a gain present in every seed), most strongly on the sequence-specific and within-family factors. The advantage comes from the gating mechanism rather than from ensembling: input-dependent gating exceeds a static average of the same experts by 0.073 AUC and the best single expert by 0.088, and the configuration selected on in-distribution data is a pretraining-free pool of convolutional experts. We further show that the common dinucleotide-shuffle negative protocol inflates the apparent margin (to a mean of 0.864), which shows the importance of fair, genomically matched negatives. We also introduce an attribution method (ShiftSmooth) that improves interpretability by averaging the gradient over small shifts of the input sequence, giving more reliable attribution for motif discovery and localization than the Vanilla Gradient. Together these provide an efficient and interpretable approach to TFBS prediction that can support further study of genome regulation.