Benchmarking supervised classifiers within a design-of-experiments framework: Robust statistical inference in text mining

Abstract

Supervised classification based on Bag-of-Words representations is widely used in literary text mining, yet benchmarking practices often remain methodologically fragile. Common problems include feature selection before train/test separation, comparisons based on non-shared resampling splits, and inferential conclusions drawn from average accuracy alone. These choices may produce misleading significance and overstate marginal performance differences between classifiers. This article presents a reproducible workflow for benchmarking supervised classifiers within a design-of-experiments framework for text analysis in R, using Dante’s Divina Commedia as an empirical testbed. The workflow combines leakage-free preprocessing, shared Monte Carlo train/test splits, paired statistical comparison, pilot-based power assessment, and validation through deliberately mispaired designs. Elastic-net multinomial logistic regression and linear support vector machine are used as competing classifiers. The objective is not to propose a new classifier, but to show how established methods can be applied under statistically coherent conditions. Shared train/test splits should be treated as part of the inferential design rather than as a technical detail of model fitting. Leakage-free preprocessing remains essential even when bias appears numerically small, because leakage can alter classifier ranking and distort inferential interpretation. Small performance differences should be interpreted alongside model transparency, inferential stability, and practical relevance, rather than through accuracy alone.