User study result and dataset

Abstract

This record contains the human annotation study underpinning the MSc thesis"Exploring Semantic Consistency in Image Super-Resolution" (Master in ComputerVision, Universitat Autònoma de Barcelona / Computer Vision Center). Super-resolution is normally evaluated on fidelity and perceptual quality — howclose an output is to its reference, and how natural it looks. Neither questionasks whether the reconstruction still means the same thing. A generative modelcan invent a plausible face, produce readable text where the reference had onlya smudge, or turn one object into another, and score well on both axes whiledoing it. This dataset was collected to study that failure mode directly, byasking people what actually changed. It contains 104 super-resolution image pairs and 4,700 annotation records from41 participants, and it is self-contained: every statistic reported in thethesis can be recomputed from these files alone. THE TASK Participants saw a reference image and a super-resolved reconstruction side byside, and marked every way the reconstruction altered the meaning of the scene: C1 — details ("details"): fine details altered, invented or lost C2 — local structures ("local"): local structures deformed or restructured C3 — holistic semantics ("holistic"): the overall meaning of the scene changes C4 — no change ("none"): nothing meaningful changed C1–C3 may co-occur. C4 is exclusive. Annotators honoured that constraint withouta single exception across all 3,497 judgements. Because one judgement can carryseveral categories, a row is one selected label, not one judgement: a judgementis the set of rows sharing a (user_id, image_id) pair. THE IMAGES 100 real pairs plus 4 attention controls. Each control presents a byte-identicalpair — the same file shown twice — so the only defensible answer is C4. This isverifiable rather than asserted: exactly the four rows flagged is_control = truehave reference_identical_to_reconstruction = true, and no real pair does. The 100 real pairs are balanced 52/52 between two public corpora, LSDIR andFoundIR, and spread across 15 super-resolution models at six or seven pairseach: BSRGAN, DiffBIR, DiT4SR, DiT-SR, FaithDiff, HAT, OSEDiff, PiSA-SR, RAP-SR,RealESRGAN, SeeSR, SUPIR, SwinIR, TSD-SR and TVT. FoundIR pairs span fourdegradation regimes. PARTICIPATION AND QUALITY CONTROL 41 participants started, 29 completed all 104 pairs, and 24 of those passed allfour controls. Controls were graded silently: participants received no feedbackand failing one did not end the session. Nobody who finished scored below two offour, and a missed control was almost always answered "details" — the signatureof someone reading compression noise as real change rather than clicking atrandom. The 24 who passed everything form the clean pool used for the headline results:24 × 100 = 2,400 judgements, exactly 24 per real pair. Every one of thoseparticipants answered every pair, so no result can be an artefact of unevencoverage. CONTENTS annotations.csv 4,700 annotation records manifest_104.csv the 104 pairs with provenance, model, degradation and control flags images/reference/ 104 reference images images/reconstruction/ 104 reconstructions; filename equals image_id README.md column reference, agreement formula, runnable reproduction snippet SHA256SUMS checksums for every file PRIVACY Participants are pseudonymous. IP addresses were removed entirely rather thanhashed, because the IPv4 space is small enough to invert any hash by exhaustivesearch. Timestamps are reduced to dates, with a seconds_from_user_start columnpreserving response-time information without the absolute times that wouldpermit linkage. Country and language values held by fewer than threeparticipants are generalised, so no single row can identify a participant. PROVENANCE AND LICENSING Reference images derive from the public LSDIR and FoundIR datasets and remainunder their original licences; manifest_104.csv records the exact source ofevery file. The reconstructions were generated by the author using the publiccheckpoints of the 15 models listed above. Annotations, manifest anddocumentation are released under CC BY 4.0.