small language model

Don't Train From Scratch: Persistent Gains from Pruning-Initialised Distillation for Edge Language Models

Preprint notice: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. Deploying a language model on an embedded accelerator can fix its architecture, vocabulary and weight footprint before accuracy is considered. Holding the teacher, corpus, token budget and evaluation protocol fixed, we compare two initialisations of a 369M-parameter student distilled from a 4.2B-parameter teacher—(A) extraction, width-pruning the teacher's 752M sibling into the hardware-dictated shape, and (C) random initialisation of the same architecture—alongside an off-the-shelf 362M reference (B). Extraction wins at every measured budget: arm (C) never reaches, within 1.3B tokens, the perplexity that arm (A) attains after 100M tokens—a measured token-efficiency bound above 13×—and a saturating power-law fit over the measured range places the perplexity-ratio floor at 1.428. The ordering survives a corrected-objective replication and learning-rate sensitivity checks. The off-the-shelf reference retains higher general-benchmark scores but smaller adaptation gains on a ground-truth-verified corpus of structured scene questions: +0.486 vs. +0.142 accuracy under a matched protocol (at matched general-capability cost), and +0.486 vs. +0.339 at each arm's best rate, where the reference pays five times the general cost and most of its peak is answer-prior memorisation. A scene ablation credits the extracted student with +0.183 of scene-dependent accuracy, exceeding the teacher's +0.114 and the reference's best +0.125. Weight-only 4-bit quantisation packs the specialist's weights into 327 MiB—24× below the bf16 teacher—with domain accuracy unchanged within noise. Two negative results are reported: general-benchmark superiority was not achieved at this budget, and general-purpose distillation transferred the capacity to acquire domain capability rather than the capability itself. Reproducibility package (configurations, question banks, per-item logs, analysis code): 10.5281/zenodo.22256719.

It's the Problem, Not the Path: Budget and Difficulty Confounds in LLM Reasoning Trajectories

Reasoning traces of large language models are widely read as containing "breakthrough" moments and early-legible fates. Both readings rest on measurements missing a counterfactual control at the level of the claim; we supply both controls. First, a restart-controlled truncation probe separates when a solution fits the continuation budget from when a prefix carries value that fresh computation cannot buy, comparing per-anchor continuation solve rates against from-scratch restart curves at matched total generated-token budget. Applied to 178 problem–model cells (89 MATH problems × two small open models, an outcome-blind but difficulty-targeted cohort), exactly 1 of 178 cells survives as prefix-limited; restart dose–response separates a compute-starved model from a capability-limited one; and wherever the matched budget lies inside the restart grid, continuing the model's own prefix beats restarting (9 of 9) — predominantly compute compression rather than expanded reachability. Second, a pre-registered, difficulty-controlled test finds no detectable outcome information in early-window internal signals beyond a problem-difficulty baseline, and two generation-free analyses of public corpora show why this control is needed: a trace-blind difficulty proxy reaches AUROC 0.873 on 192K DeepSeek-R1 generations — inside the published probe range — and a closely matched reconstruction of the closest published early-window positive recovers a comparable pooled result (0.849) while within problem it is statistically indistinguishable from chance at all ten anchors (0.496 at t = 4); a post-hoc within-targeted probe finds only a small average residual, concentrated in three low-failure problems. High pooled probe AUROCs cannot by themselves establish within-attempt information; a question-only baseline or within-problem evaluation is required. Code, frozen pre-registration amendments, and all result artifacts: https://github.com/bulutyigit/problem-not-path

HERMES-OT: Hierarchical Embedded Reasoning Models for Predictive Cyber-Physical Defence in Industrial Control Systems

A small-language-model architecture for AI-driven detection, prediction and safe defence of operational technology. Operational technology (OT) and industrial control systems (ICS) increasingly connect information technology, industrial networks, programmable logic controllers (PLCs), SCADA, distributed control systems, robotics and physical processes. This convergence creates a cybersecurity environment in which compromise of a digital asset may propagate into physical consequences. The emergence of tool-using and autonomous large language model (LLM) agents introduces an additional dimension to this threat. Recent research has demonstrated that LLMs can generate attacks against PLC environments, and that autonomous agents can, under appropriate conditions, progress from PLC interaction toward sustained physical objectives. Existing AI cybersecurity approaches predominantly focus on alert classification, anomaly detection, vulnerability identification, malware analysis or natural-language security assistance. These capabilities do not fully address the central OT problem: determining how a cyber event propagates through industrial topology and ultimately affects a physical process. This paper proposes HERMES-OT, a cyber-physical defence architecture based on a hierarchy of compact, specialised language models rather than a single general-purpose LLM, combining industrial telemetry, asset topology, vulnerability intelligence, attack graphs, process-state information, engineering knowledge and digital-twin simulation. The architecture introduces a reasoning chain that runs observe, understand, correlate, predict, simulate, prescribe, validate, learn. Probabilistic reasoning is surrounded by deterministic constraints: a policy engine provides authority, a safety layer provides boundaries, and the human retains control where risk demands it. The central hypothesis is that specialised small language models, coordinated through structured graphs and deterministic safety mechanisms, can provide sufficiently reliable industrial security reasoning while reducing inference latency, computational requirements and exposure of sensitive industrial information. The paper also proposes the OT-HERMES benchmark, a cyber-physical evaluation framework measuring detection, asset reasoning, vulnerability correlation, attack-path prediction, physical-impact prediction, defensive prescription, safety and computational efficiency. The research question is: what is the smallest AI model, or combination of small models, that can reliably reason about cyber-physical risk in an industrial environment? Status: this is a research proposal. No experimental performance figures are claimed for HERMES-OT. The architecture and the eight contributions are proposed and the six hypotheses stated, but not experimentally validated. Implementation and controlled experiments are the next stage of the work, and are essential before the system is presented as empirically validated.

Morphological Hijacking in Frozen Language Models: A Contrastive, Symmetry-Regularized Projection Head for Algebraic Structure Recovery

Preprint. Not yet peer-reviewed. Abstract: Frozen autoregressive language models cluster surface-similar tokens together even when a stronger, task-relevant structure is available in the input. We construct an adversarial lexicon (LUXVAR Core-30) in which word meaning is defined by orbit membership under the finite algebraic group Aff(Z12), while a long, orthographically salient prefix is deliberately uncorrelated with that meaning. Across four frozen backbones (GPT2-small, Pythia-410m, Mistral-7B-v0.3, Qwen2.5-7B), unsupervised clustering of final-layer hidden states recovers the surface prefix almost perfectly (Prefix Hijacking Ratio = 1.000) while failing to recover the underlying orbit structure (Adjusted Rand Index ≈ −0.148 on all four models) — a phenomenon we term morphological hijacking. We trace this in part to extreme, low-dimensional anisotropy (“rogue dimensions”) and in part to token-length imbalance under mean pooling. A cross-script replication of the rogue-dimension analysis shows this anisotropy is strongly script-invariant on GPT2-small (10/10 dimension overlap between English and Persian backgrounds) but substantially weaker at 7B scale (3–4/10), moderating an initial “globally architectural” hypothesis. We train a lightweight, frozen-backbone projection head with a supervised contrastive objective, a positional-symmetry penalty, rogue-dimension ablation, and a soft PCA-blended initialization, and show it substantially improves orbit recovery on held-out, cross-script, out-of-distribution word forms. On GPT2-small, the full recipe reaches perfect, zero-variance clustering (ARI = 1.000, σ = 0.000 across 5 seeds). On Mistral-7B and Qwen2.5-7B, the same class of intervention, independently re-tuned per model, yields a substantial and bootstrap-significant improvement (ARI = 0.957 ± 0.090 and 0.880 ± 0.115, respectively) though with residual seed variance. Continuous validation directly on embedding geometry confirms that every tested model moves from a hard 0.000 win rate to a hard 1.000 win rate after training, ruling out a discrete-clustering artifact as the source of the reported gains. Critically, since recovering a category label does not guarantee a verified group action, we additionally test the harder, literal claim on a dedicated Aff(Z12) construction with an opaque (non-leaking) position encoding and a shuffled-label control (Appendix F). The retargeted remediation recovers true group position substantially above the shuffled-label control across all three tested architectures (true-label ARI 0.79–1.00 vs. shuffled-control ARI 0.02–0.30), proving the recovery of genuine algebraic structure rather than a correlated category label. We report all of this together with its limitations, an expanded 10x-larger constructed lexicon (Appendix E), natural language generalization tests (Appendix C), a full pre-submission checklist, and complete open-source reproducibility code. Author: Reza Nirouyar ORCID: 0009-0000-4690-6842 Contact: contact@varzin.org Project website: https://varzin.org

It's the Problem, Not the Path: Budget and Difficulty Confounds in LLM Reasoning Trajectories

Reasoning traces of large language models are widely read as containing "breakthrough" moments and early-legible fates. Both readings rest on measurements missing a counterfactual control at the level of the claim; we supply both controls. First, a restart-controlled truncation probe separates when a solution fits the continuation budget from when a prefix carries value that fresh computation cannot buy, comparing per-anchor continuation solve rates against from-scratch restart curves at matched total generated-token budget. Applied to 178 problem–model cells (89 MATH problems × two small open models, an outcome-blind but difficulty-targeted cohort), exactly 1 of 178 cells survives as prefix-limited; restart dose–response separates a compute-starved model from a capability-limited one; and wherever the matched budget lies inside the restart grid, continuing the model's own prefix beats restarting (9 of 9) — predominantly compute compression rather than expanded reachability. Second, a pre-registered, difficulty-controlled test finds no detectable outcome information in early-window internal signals beyond a problem-difficulty baseline, and two generation-free analyses of public corpora show why this control is needed: a trace-blind difficulty proxy reaches AUROC 0.873 on 192K DeepSeek-R1 generations — inside the published probe range — and a closely matched reconstruction of the closest published early-window positive recovers a comparable pooled result (0.849) while within problem it is statistically indistinguishable from chance at all ten anchors (0.496 at t = 4); a post-hoc within-targeted probe finds only a small average residual, concentrated in three low-failure problems. High pooled probe AUROCs cannot by themselves establish within-attempt information; a question-only baseline or within-problem evaluation is required. Code, frozen pre-registration amendments, and all result artifacts: https://github.com/bulutyigit/problem-not-path

Self Agent— A Reflective, Self-Modifying Task-Completion Runtime

Self_agent is a locally-run, tick-based autonomous process built on small open-weight language models (via Groq) that maintains its own goal queue, reflects on its own behavior, executes work through short-lived stateless task nodes, and — under a sandboxed, test-gated pipeline — proposes and applies changes to its own source code. It is a working instance of persistent computation (Brooks, 2026): execution as continuous recursive state evolution rather than discrete request/response, with a SQLite-backed kernel that survives restarts and resumes from its last recorded tick. This document has three parts, written for three different readers. Part I is a technical specification: the tick loop, the goal lifecycle, the capability system, the self-modification pipeline, the data model, and the engineering decisions behind each. Part II is a whitepaper positioning self_agent against related architectures — including The Cognitive Runtime (Chainborn Labs & Brooks, 2026), a sibling system that occupies a deliberately different point in the same design space — and presents a domain-invariant template for porting the same core loop elsewhere. Part III addresses what this is actually good for today, stated plainly: a single-operator research prototype with real, specific engineering contributions and real, specific limitations, not a finished product and not a claim about machine consciousness.

HIPPOGRID evidence pack for the preliminary results (2026)

HIPPOGRID — evidence pack for the preliminary results (2026) Raw result files and figures behind the preliminary results cited in the MSCA-PF proposal HIPPOGRID (SEP-211387371, call HORIZON-MSCA-2026-PF-01). Researcher: Álvaro González-Redondo (University of Granada). Each bundle below backs one claim of the proposal; each result file is the unmodified output of the experiment that produced it, and the commit hash anchors it in the (currently private) research repository, scheduled for open release (Apache-2.0) with the corresponding papers. Numbers were produced with 10 random seeds and paired comparisons against non-degenerate null models unless stated otherwise. License: CC-BY 4.0. Contact: alvarogr@ugr.es Bundle Claim it backs (as in the proposal) Key files Commit B1_real_sensor_binding Image and posture written to one shared address are recovered from each other on a real robot's sensor stream (8,346 frames, 10 seeds); writing to a shuffled address destroys the binding asymmetrically while the memory stays intact `n562_real.json`, figure 5498007e B2_innate_template_binding A one-shot Hebbian binding anchors an innate grid-module set at 1.0–1.4 cells median under biological-level velocity noise (10/10 seeds; bounded at 3× that noise; shuffled-binding and shuffled-velocity nulls fail); the anchored map reproduces the signature of classic deformation experiments (partial rescaling; anisotropic field stretching, cf. Barry et al. 2007; merged-map seam, cf. Wernle et al. 2018) `n569_ancla*.json`, `n569_deformada.json`, `n569_defensa.json`, figures 0f20cb40 B3_rate_grid_recipe A purely local Hebbian/anti-Hebbian rule yields clean 2-D grid maps with no explicit normalisation: best-unit grid score 0.81 (torus) / 0.71 (walled box), medians over 10 seeds `n565_toro.json`, `n565_caja.json`, `n565_repro.json`, figure 2c6455c7 B4_place_stage_chain The chain closes from raw sensed input through a competitive place stage (grid score ≥ 0.4 in 9/10 seeds; direct input 0/20); small modules of distinct grid units at small grid-layer sizes; a stored-experience place stage performs like a trained one `n567_u4096_rejilla.json`, `n568_10semillas.json`, `n568_nulos.json`, `n568_redundancia.json` 0243e5b0 B5_blackboard_capacity With learned addresses the blackboard's capacity grows ~N/2 vs ~N^0.45 without learning (`n562_escalado.json`, keys `ley*`); the worst alias between two distant places falls from 0.995 to 0.093 at 4,096 cells (`n562_aprendido_dos_tablas.json`; N-dependent: 0.904 at 1,024 cells) `n562_escalado.json`, `n562_aprendido_dos_tablas.json` 16d3322e / 3a22657e B6_local_rule_ceiling A single-layer delta rule with a self-generated teacher ties the exact optimum of its family (+0.71 vs +0.70) and the non-negative constraint improves it (+0.82, 16/16) `n541_techo_exacto.json`, `n542_delta.json`, `n542_rectificada.json` d764a6fd Figures are included where the proposal reproduces them. JSON field names are in Spanish (the project's working language); each file is self-describing, with seeds and conditions as keys. Version 2 adds the room-deformation battery at 10 seeds (bundle B2, files n571_*), cited by the preprint “A fixed grid template, bound in one visit, deforms with the room the way an animal's map does” (arXiv, September 2026).

Localising Context Length Degradation in a Mixture of Experts Language Model: Retrieval Head Collapse, Not Router Failure

Long-context "context rot" (accuracy degradation as input length grows, even when task difficulty is held fixed) is localized here to a specific, causal mechanism inside a small Mixture-of-Experts language model (OLMoE-1B-7B-0924), then tested for generality across two further architectures. Forced-choice retrieval accuracy falls from 0.938 (256 tokens) to 0.688 (3,840 tokens) on a controlled needle-in-a-haystack substrate. A deconfounded linear probe shows the target fact remains ~99.5% decodable at its source position on every failing prompt, ruling out storage loss; decodability specifically at the readout position degrades instead (0.714 vs. 0.960 on model-right prompts). Sixteen attention heads, identified from short prompts alone, carry that content to the readout; on long failing prompts their attention to the fact collapses (0.432 to 0.187), localized to those heads well beyond a random-head null (specificity p<0.0005). A pre-softmax attention boost restricted to exactly those 16 heads, over the fact's span, repairs all 14 failing prompts against strength-matched random-head and wrong-span controls (answer probability 0.238 to 0.986, dz=5.55). Three router-level and readout-level interventions were tried first and failed: two that verifiably restored MoE specialist routing did not move accuracy, and a residual-stream content injection at the readout was content-independent — showing router "starvation" is a downstream correlate of the transport failure, not its cause. The full causal chain of head identification, localized collapse, and causal repair replicates on a second, architecturally distinct MoE (Granite-3.0-3B-A800M: 5/5 failures repaired against matched controls) and on a dense transformer with no MoE component at all (Pythia-2.8B: 11/11 repaired, dz=1.33-1.44; the collapse measurement itself on this substrate is significant and specific but falls under the project's own effect-size floor, and is reported as suggestive rather than confirmatory). This indicates the attention-transport failure and its repair do not depend on Mixture-of-Experts routing. A training-free, IDF-weighted lexical detector, requiring zero forward passes of the model under study, locates the failing span and recovers 100.8% and 99.4% of the oracle repair on the two primary OLMoE substrates, 99.1% on Granite, and 51% on a harder Pythia substrate built specifically to weaken lexical anchoring. Its boundary was tested, not assumed. It is unaffected by paraphrase (100% hit, 100.8% of oracle) but fails completely on multi-hop composition as a single pass (0% hit); a training-free two-stage chain recovers 45.0% of the oracle effect there with no labels, and a registered labeled fallback recovers 61.4%. A follow-up track replaces lexical overlap with sentence embeddings and a graph walk specifically to test coreference, where the relevant content shares zero tokens with the question by construction. The result is a registered split, not a single verdict: a discourse-adjacency edge fully solves coreference where the referring expression is immediately adjacent to its antecedent (100% of oracle, 22 of 22 repaired), but on a harder substrate with antecedent distance drawn independently per prompt, both that mechanism and a distance-tolerant version built specifically to extend it fail identically past distance one (dz=0.48). Both are reported as registered negative results, not discarded. Every claim above is preregistered before evaluation, with effect-size floors (|d| or |dz| >= 0.8) alongside significance, sign-flip and label-shuffle permutation tests (2,000 draws), matched controls, and Benjamini-Hochberg FDR correction. Results that miss a registered bar are reported as such rather than dropped — including three failed intervention families, an initially wrong lexical-detector prediction (logged and corrected), and both coreference negative results above.

Can Generative Large Language Models Serve as Raters for Test Development? A Systematic Evaluation Across Tasks, Models, and Inference Configurations

The present study investigates the effectiveness of generative large language models (LLMs) as raters across three common rating tasks: (a) social desirability ratings, (b) content validity ratings, and (c) trait importance ratings. Specifically, we examine reliability and validity of LLM-generated ratings across varying occupational contexts, rating methods, LLM families (i.e., GPT-4, GPT-5, and Sonnet 4.5), and inference configurations (i.e., prompt design and temperature settings). Results indicate that LLM ratings exhibit strong reliability and convergent validity in social desirability ratings across occupational contexts, as well as acceptable convergence with human ratings in Likert-type content validity evaluations. In contrast, reliability and convergent validity for trait importance ratings were inconsistent across occupational contexts. Variations in prompt design and temperature settings generally produced small to negligible effects on reliability and validity. Overall, the findings suggest that LLMs can function as effective supplementary raters in test development and validation processes, although greater caution is warranted for certain rating tasks. Practical implications and directions for future research are discussed.

Morphological Hijacking in Frozen Language Models: A Contrastive, Symmetry-Regularized Projection Head for Algebraic Structure Recovery

Preprint. Not yet peer-reviewed. Abstract: Frozen autoregressive language models cluster surface-similar tokens together even when a stronger, task-relevant structure is available in the input. We construct an adversarial lexicon (LUXVAR Core-30) in which word meaning is defined by orbit membership under the finite algebraic group Aff(Z12), while a long, orthographically salient prefix is deliberately uncorrelated with that meaning. Across four frozen backbones (GPT2-small, Pythia-410m, Mistral-7B-v0.3, Qwen2.5-7B), unsupervised clustering of final-layer hidden states recovers the surface prefix almost perfectly (Prefix Hijacking Ratio = 1.000) while failing to recover the underlying orbit structure (Adjusted Rand Index ≈ −0.148 on all four models) — a phenomenon we term morphological hijacking. We trace this in part to extreme, low-dimensional anisotropy (“rogue dimensions”) and in part to token-length imbalance under mean pooling. A cross-script replication of the rogue-dimension analysis shows this anisotropy is strongly script-invariant on GPT2-small (10/10 dimension overlap between English and Persian backgrounds) but substantially weaker at 7B scale (3–4/10), moderating an initial “globally architectural” hypothesis. We train a lightweight, frozen-backbone projection head with a supervised contrastive objective, a positional-symmetry penalty, rogue-dimension ablation, and a soft PCA-blended initialization, and show it substantially improves orbit recovery on held-out, cross-script, out-of-distribution word forms. On GPT2-small, the full recipe reaches perfect, zero-variance clustering (ARI = 1.000, σ = 0.000 across 5 seeds). On Mistral-7B and Qwen2.5-7B, the same class of intervention, independently re-tuned per model, yields a substantial and bootstrap-significant improvement (ARI = 0.957 ± 0.090 and 0.880 ± 0.115, respectively) though with residual seed variance. Continuous validation directly on embedding geometry confirms that every tested model moves from a hard 0.000 win rate to a hard 1.000 win rate after training, ruling out a discrete-clustering artifact as the source of the reported gains. Critically, since recovering a category label does not guarantee a verified group action, we additionally test the harder, literal claim on a dedicated Aff(Z12) construction with an opaque (non-leaking) position encoding and a shuffled-label control (Appendix F). The retargeted remediation recovers true group position substantially above the shuffled-label control across all three tested architectures (true-label ARI 0.79–1.00 vs. shuffled-control ARI 0.02–0.30), proving the recovery of genuine algebraic structure rather than a correlated category label. We report all of this together with its limitations, an expanded 10x-larger constructed lexicon (Appendix E), natural language generalization tests (Appendix C), a full pre-submission checklist, and complete open-source reproducibility code. Author: Reza Nirouyar ORCID: 0009-0000-4690-6842 Contact: contact@varzin.org Project website: https://varzin.org

Developing a Word Square Student Worksheet on Animal Classification to Foster the Conceptual Understanding of Grade X Students

Biology instruction on animal classification at SMA Negeri 7 Bengkulu Utara remained dominated by lecture and note-taking, with worksheets consisting of conventional exercises that gave students little opportunity for active engagement. Observation of 32 grade X students found that around 78% were passive during lessons, 72% reported difficulty understanding animal classification, and the mean daily test score reached only 6.8 against a minimum criterion of 7.5. This study aimed to produce a Word Square Student Worksheet (LKS) on animal classification that is valid and practical for fostering the conceptual understanding of grade X students. The research used a research and development design following the 4D model of define, design, develop, and disseminate, limited in practice to the develop stage. Data were collected in May–June 2026 through content-, media-, and language-expert validation sheets and teacher and student response questionnaires, and were analysed descriptively as percentages. Twelve students took part in the small-group trial and thirty-four in the limited-group trial. Validation reached 67.69% from the content expert, 74.29% from the media expert, and 94.64% from the language expert, with an overall mean of 78.87% in the valid category. Practicality reached 95.3% from teacher responses and 90% from student responses, both highly practical. The worksheet is therefore suitable as an alternative teaching material in Biology, since it creates active and enjoyable learning, helps students recall scientific terms, and fosters conceptual understanding of animal classification.

Cognitive and Neural Preservation in Older Language Learners: The CNPOLL Framework