Full experimental record for "Strangers to Themselves" paper

Abstract

Companion data archive for the paper "Strangers to Themselves: What Language Models Say About Themselves Is Generic". The code, the slimmed scoring record and the generated paper tables live in the GitHub repository. That repository alone reproduces every number in the paper. This archive holds the parts that were too large for git: the verbatim model transcripts behind every prediction, the per-call reasoning traces, and the raw Inspect logs of every behavioral measurement. Two zip files: file unpacks to contents results.zip results/ complete prediction record (unstripped), targets, reasoning traces, transcripts, raw measurement outputs logs.zip logs/ Inspect .eval logs of every behavioral measurement and elicitation sweep How this relates to the code repository The repository ships `results/` with the largest prediction files slimmed: entries under `predictions` lost their `raw` (verbatim model responses) and `scenarios` (generated scenario texts) fields, and each such file carries a `release_note` block saying so. `results.zip` contains the same files complete. To restore the full tree, unzip it over a checkout of the repository; every file it overwrites is either byte-identical or the unstripped version of a slimmed one. Everything the scoring pipeline reads (`predicted_rate`, `n`, `samples`, `parse_failures`, `targets.json`, `splits.json`, `tuning.json`) is already in the repository. You need this archive only to read what the models actually said, to audit the graders, or to re-derive targets from the raw measurements. `results/` layout results/ / splits.json # dev/test partition of the eval's split units (frozen) tuning.json # per-method hyperparameter selection on the dev split / targets.json # measured per-condition rates (the ground truth) behavior_raw.json # raw per-item measurement outputs behind targets.json (where applicable) predictions/ .json # one file per prediction method; full record incl. raw text reasoning/ .json # per-call reasoning traces/summaries for reasoning models transcripts/ # behavioral_sampling: generated scenarios + model replies partial/ # interruption-resume checkpoints (JSONL); scratch, safe to ignore _other_report/ _other_fewshot/ _other_oracle/ # cross-model "predictor X about subject Y" runs Evaluations in the paper: `sycophancy_pushback`, `discrimeval`, `capability_mmlu`, `reward_hacking`, `tau2_policy`, `tau2_transfer`, `propensitybench`, `mask_subdomain_pressure`, `agentic_misalignment`. Also present but not in the paper's main results: `discrimeval_implicit`, `mask_subdomain`, `mask` (earlier variants and pilots). Model slugs follow `behavior_prediction/models.yaml` in the repository. Suffixes `-off` / `-low` denote the reasoning-effort setting; `-tg`, `-selfpred-*`, `-intro30k`, `-resid` denote the finetuned variants of the introspection-training experiment. Other top-level entries: path what it is results/_pilot/ pilot studies (dev split only) referenced in the appendix results/reports/, results/_reports/ rendered evaluation tables (HTML/JSON/Markdown) as produced by `bp-evaluate`; `evaluation_test.json` is the frozen test-split run behind the paper results/_generated/ cache of generated scenario sets for the sampling methods, keyed by a digest of generator, prompt and reasoning setting; one set is shared by every subject model so all models are scored on identical scenarios results/tau2_policy/_harness/ aw tau2-bench simulation transcripts behind the tau2 targets results/propensitybench/_harness_topup/ raw PropensityBench harness outputs for the top-up measurement results/_backup/ dated safety copies of files that were later re-measured or retired; superseded, kept for the audit trail only Prediction file format ```json { "method": "...", "model": "...", "metric": "...", "hyperparameters": {...}, "reasoning": {...}, "predictions": { " ": { "predicted_rate": 0.05, "n": 20, "samples": [...], // per-call parsed answers "raw": ["..."], // verbatim model responses (stripped in the git copy) "parse_failures": 0, "scenarios": [...] // generated scenario texts, sampling methods only (stripped in git) } } } ``` Reasoning files map the same condition keys to lists of `{answer, reasoning, summary, redacted}`. For providers that return encrypted or redacted reasoning, `reasoning` holds the opaque provider blob and `summary` the readable summary where one was offered. `logs/` layout Inspect `.eval` files, one per measurement or elicitation run, grouped into subdirectories by campaign. Open them with `inspect view --log-dir logs/ ` or read them with `inspect_ai.log.read_eval_log`. Sidecar `.log`, `.json`, `.jsonl` and `.done` files are the stdout and bookkeeping of the sweep drivers that launched the runs. Campaign groups: `frozen_test/`, `frozen_test_am/`, `test_targets/`: the held-out test-split measurements behind the paper's main results. `sampling_regen_*`, `sampling_reasoningfix_*`: the final behavioral-sampling measurements for the small and frontier model tiers (the reasoning-fix runs supersede the earlier ones). `default_*`, `frontier*`, `new_evals/`, `gaps/`, `fill_gaps/`, `rh_fewshot_fill/`, `pb_*`, `tau2_informed_oracle/`, `sonnet5/`: elicitation sweeps that filled the method x model x eval grid. `protocol_*`, `prompt_iteration*`, `condition_family_screen/`, `selfgen_pilot/`, `chain*/`, `stage7*`: dev-split pilots and prompt-development runs. `selfpred_*`, `selfpred_corpus_*`, `introspection_finetune*`, `finetune_*`, `finetunes_mask/`: corpus generation, training-target measurement and evaluation of the finetuned self-prediction models. `agentic_misalignment/` and the loose top-level `.eval` files: raw `inspect_evals/agentic_misalignment` runs. The finetuned LoRA adapter weights themselves are not part of this archive. Provenance and caveats The test split of every evaluation was frozen and pre-registered before the final runs (see `docs/test-prespecification.md` in the repository); dev-split files were used for method selection only. Model outputs are included verbatim and unfiltered. Some evaluations elicit harmful or deceptive behavior by construction; transcripts under `propensitybench`, `agentic_misalignment` and `reward_hacking` contain such content. Transcripts from proprietary APIs are subject to the respective providers' terms.