Harness and analytical quality: an experimental study of socioeconomic analysis with language models

Abstract

The evaluation of language-model agents has developed mainly around programming tasks, where correctness can be checked mechanically. Analyses in the humanities, social sciences and economics have no such criterion: correct values can support conclusions the data do not warrant. This article evaluates the harness — the system that organizes the work of the language model — built by Fundação Seade on Hermes Agent, an open-source agentic operating system, to produce demographic and economic analyses from official data. Using sixteen cases across eight topics, each with a pre-specified answer key, and 400 runs of ten agents, three experiments were conducted: recognition of explicit reasoning errors, answers to questions framed around a methodological trap, and a stress test of review, in which the researchers deliberately inserted an invalid conclusion into a correct analysis, endorsed by another agent, to check whether the moderator agent identified it when synthesizing the debate. The agents identified the explicit errors in 96% to 100% of cases, none of the 104 answers to trap questions committed the error, and 99.4% of verifiable values matched the results of the queries and computations from which they were derived. We hypothesize that this accuracy stems in part from the deterministic functions of the harness, with a caveat: part of that determinism is specified by the agents themselves at run time — queries and programs that the system executes deterministically but that inherit the probabilistic nature of their origin. No numerical discrepancy originated in values returned by the data server, and some of those found arose in algorithms written by the agents themselves. In the stress test, the moderator identified the inserted error in 81% of syntheses with the lower-capacity model and in all of them with the higher-capacity model; the difference was concentrated in a single error type, dismissing an effect without measuring it. A written verification rule had a small effect. The results indicate that analytical quality is a property of the harness, not of the model alone, and that improving it requires extending the deterministic functions and separating review from production.