Feb 2025· Machine-mediated learning· Vol 115· 0 citations· 68 references
Computer Science
TL;DR
This work experimentally explores the question: is it always optimal to train the assessor for the target metric, or could it be better to train for a different metric and then map predictions back to the target metric?
Abstract
An AI assessor is an external, ideally independent system that predicts an indicator, e.g., a loss value, of another AI system. Assessors can leverage information from the test results of many other AI systems and have the flexibility of being trained on any loss function or scoring rule: from squared error to toxicity metrics. Here we address the question: is it always optimal to train the assessor for the target metric? Or could it be better to train for a different metric and then map predictions back to the target metric? Using twenty regression and classification problems with tabular data, we experimentally explore this question for, respectively, regression losses and classification scores with monotonic and nonmonotonic mappings and find that, contrary to intuition, optimising for more informative metrics (i.e., yielding a better-conditioned supervision signal) is not universally preferred. Surprisingly, some monotonic transformations are promising. For example, logistic loss is useful for minimising absolute or quadratic errors in regression, and logarithmic score helps maximise quadratic or spherical scores in classification.
In regulated domains such as finance, a model that cannot be explained cannot be deployed, yet many interpretable classifiers defeat their own purpose by producing formulas with dozens of features that no regulator could read. We take the reverse direction. Starting from an interpretable classifier expressed as a single equation over the input features, we progressively simplify it into more readable forms, including a pruned monomial, a directional if--then rule, and the integer scorecards and tallies that finance already deploys. Because the equation is itself the predictive model rather than a post-hoc explanation we can directly quantify what is lost under each simplification. Across four financial datasets, we find that pruning is nearly free and that fidelity can erode faster than predictive performance, allowing simpler rules to remain effective classifiers without faithfully reproducing the original model. A human assessment shows that simplification improves perceived readability, while preferences for different representations vary by professional background. Beyond measuring these losses empirically, we show that some can be anticipated from the original model: we derive a bound on the change caused by pruning and predict how faithfully a rule retaining only the direction of each feature's effect preserves the original ranking.
This brief argues that market-based governance fails: individual property rights over data cannot address AI’s harms and benefits, because machine learning is fundamentally about data externalities, and because platform network effects are artificially maintained.
Maximilian Kasy· Journal of Economic Interact...· 0 citations
This work shows that pooled refitting recomputes the inverse-Gram geometry used to weight source evidence, which can reverse shared preferences, and derive exact and approximate preservation conditions, and develops a three-stage audit that traces strict pairwise reversals through decision changes to task-defined utility loss.
Yanli Yan, Yuanzheng Li, Yong Zhao et al.· 0 citations
This work defines the evidential ceiling of an evaluation as the largest factor by which one result can move belief under a fixed testing budget, derive it in closed form for the benchmark null result, and uses it to locate that boundary exactly.
This work defines a grouping metric, specify a harness, and shows how tracking a human-AI pair's grouping over time yields the compounding signal that Paper 1's field study requires.
It is shown that a key factor is the error correlation structure between human and AI predictions, and when the AI's prediction errors are negatively correlated with those of the human, the decision maker can construct robust strategies which guarantee improvements in expected utility.