Making good decisions with counterfactuals

Abstract

Making a good decision often requires knowing something counterfactual: whether a person would be better off if we acted than if we didn't. Formal work shows how both experimental and observational data ought to be combined to inform good decisions. Here, we ask whether people's intuitions follow this normative standard. In three experiments (N = 354), participants decided whether or not to act based on observational and experimental data. Participants performed better than chance but far from optimally. Trial-by-trial feedback about the consequences of their choices, including what would have happened had they chosen otherwise, didn't improve performance. The breakdown was at the inference step: participants' estimates of whether a person would benefit from the treatment, be harmed by it, or be left unaffected were almost unrelated to normative predictions. Yet, participants' decisions followed coherently from those (incorrect) estimates. Experts in statistics, economics, and data science made better decisions than novices, but their counterfactual inferences were no more accurate. Counterfactual reasoning is critical for making good decisions from experimental and observational data, but its logic doesn't come naturally to people.