Style–morphology decomposition for disentangling structural and staining effects on MSI prediction changes. A, Representative examples of counterfactual manipulation between MSIH and non-MSIH classes. For each original image x and its counterfactual xcf, style-hybrid (xstyle) and morphology-hybrid (xmorph) images were generated using Vahadane stain transfer. Each hybrid isolates the effect of either stain or morphology while controlling for the other. The right-hand bars show the Shapley-style decomposition of the logit change (Δf) into stain (φstyle) and morphology (φmorph) contributions, demonstrating that morphologic differences dominate the model’s predictions. Grad-CAM visualizations below provide region-level attribution under MIL. In contrast, MoPaDi produces class-directed “what-if” edits that offer a complementary view of candidate morphologic and style changes associated with prediction shifts. Scale bar applies to all images within the panel. B, Decomposition results across test-set patients, showing median contributions of φstyle, φmorph, and total (Δf) for manipulations toward (↑) and away from (↓) each class. C, Scatter plot of morphology versus style contributions per patient, illustrating consistent dominance of morphologic effects across both manipulation directions and classes.
Laura Žigutytė, Tim Lenz, Tianyu Han et al.· 0 citations
Counterfactual image examples generated for the lung and breast cancer–type classifiers. A, Representative examples of LUSC tile transitioning to its counterfactual lung adenocarcinoma (LUAD) image and vice versa. B, Morphologic feature prevalence in original and counterfactual image pairs (N = 32 transitions; 16 tiles for each class). Horizontal bars show the percentage of image pairs in which at least one of three raters (three board-certified pathologists) identified each morphologic feature as present in the original (dark gray) or counterfactual (light gray) image. Right, Mean pairwise inter-rater agreement (Cohen κ) per feature, computed across all pairs and both directions combined. C, Counterfactual image generation effectiveness, measured as the percentage of generated images predicted as the opposite class across varying manipulation amplitudes. D, Representative examples of ILC tile transitioning to its counterfactual IDC image and vice versa. Difference maps display pixelwise differences between the original and the synthetic tile. Scale bar applies to all images within the panel unless otherwise indicated.
Laura Žigutytė, Tim Lenz, Tianyu Han et al.· 0 citations
Counterfactual image examples generated for the liver cancer type classifier [hepatocellular carcinoma (HCC) vs. cholangiocarcinoma (CCA)]. A, Representative examples of counterfactual transitions generated with two approaches (linear and MIL) and a comparison of corresponding segmentation masks. B, Counterfactual image generation effectiveness, measured as the percentage of generated images predicted as the opposite class across varying manipulation amplitudes. HCC-to-CCA required higher amplitudes to achieve similar results, which likely reflects class imbalance in the dataset. Scale bar applies to all images within the panel.
Laura Žigutytė, Tim Lenz, Tianyu Han et al.· 0 citations
Independent classifier validation and representative bidirectional counterfactual transformations in a multiclass setting. A, External classifier responses to counterfactual morphing. Counterfactual images were generated at increasing morphing amplitudes (α) with MoPaDi and then encoded with three foundation models (UNI2, CONCH, and Virchow2). Independent classifiers were trained on the corresponding encoders’ features extracted from all TCGA-CRC tiles and then used to predict the target probability P(target) on both the original and counterfactual tiles. The resulting change ΔP(target) reflects how strongly the morphing affected class evidence. Bars show the median ΔP(target) with percentile-based variability across tiles. B, Representative examples of bidirectional counterfactual explanations for MSIL patients. We defined MSIL by fitting a two-component Gaussian mixture model to the log-transformed distribution of total MSI events and using the intersection of the two components as the cutoff separating MSIL from MSIH samples.
Laura Žigutytė, Tim Lenz, Tianyu Han et al.· 0 citations