small language model

frb-vlm-benchmark: a zero-shot Vision-Language Model benchmark for Fast Radio Burst detection

Code and evaluation artifacts supporting the article "Generalist Vision-Language Models for Fast Radio Burst detection: a zero-shot benchmark against a specialized detector" (Santos, Queiroz, Duarte, de Farias & Batista), submitted to The Astrophysical Journal Supplement Series (manuscript AAS80842; preprint arXiv:2607.07382). The pipeline simulates PSRFITS dynamic spectra containing Fast Radio Bursts (FRB), structured Radio Frequency Interference (RFI) and system noise; renders them as anonymized PNG images; classifies those images with small, open-weight Vision-Language Models (Gemma 4 E2B/E4B) in a zero-shot, prompt-only regime; and evaluates the predictions, including a paired, sample-by-sample comparison against the specialized detector SwinYNet. The same models, prompts and decision threshold are then applied unchanged to the 1600 real FAST observations of the FAST-FREX data set (doi:10.57760/sciencedb.15070). Contents and file formats (169 files) Source code (56 .py files) — Python packages implementing each pipeline stage: simulate_dataset/ (synthetic PSRFITS generation and selection of the balanced binary subset), plot_dataset/ (PSRFITS to anonymized dynamic-spectrum PNG), vlm_classifier/ (zero-shot classification with Gemma 4 via Hugging Face Transformers), evaluation/ (discrete, probabilistic and paired metrics), benchmark_export/ and benchmark_predictions/ (anonymized FITS export and import/comparison of external detector output), prepare_real/ (preprocessing and rendering of real search-mode PSRFITS), plus tests/ (unit tests). Entry points: run_pipeline.py (CLI integrating all stages), fetch_fast_frex.py (selects and downloads the FAST-FREX files), prompt_timing_experiment.py (prompt-latency audit), and the read-only diagnostics check_burst_visibility.py, check_burst_visibility_sim.py, analyze_visibility.py, summarize_real_run.py. Configuration (2 .yaml, 1 .toml, 1 uv.lock) — config.yaml and config_binary.yaml are the exact configurations used for the published runs, pinning the instrumental setup, the global seed and the image-rendering protocol; pyproject.toml and uv.lock pin the Python environment. Data manifests (14 .jsonl files) — line-delimited JSON. dataset/metadata/labels.jsonl and image_manifest.jsonl record, per sample, the ground-truth class, the injected burst or RFI parameters, the source PSRFITS and the anonymized image identifier; dataset_binary/metadata/ holds the same for the balanced 2000-sample binary benchmark; dataset_real_full/metadata/image_manifest.jsonl documents the per-file preprocessing of all 1600 rendered FAST-FREX observations (time window, masked channels, decimation). The predictions.jsonl file in each results directory holds one raw model response per sample. Tabular data (21 .csv files) — the burst-visibility diagnostics that underlie Section 7 of the article (fast_frex/burst_visibility_full.csv and dataset/metadata/burst_visibility_sim.csv: dedispersed peak significance, recovered peak time and catalogued parameters per burst), the FAST-FREX input manifest and burst-parameter tables, and the paired_scores.csv files aligning the continuous scores of each VLM and of SwinYNet sample by sample. Metrics and reports (15 .json, 12 .txt) — the aggregate metrics computed by evaluation/ (accuracy, macro-F1, per-class precision and recall, ROC-AUC, average precision, Brier score, ECE, McNemar tests and bootstrap intervals) and their human-readable summaries, one set per run. Figures (27 .png, 16 .pdf) — the article figures and the diagnostic plots produced by each run; figures_src/ regenerates them from the tracked artifacts. Results directories and their relationship to the article results_vlm2b_0t_binary/, results_vlm_0t_binary/ — Gemma 4 E2B and E4B on the binary FRB vs NON_FRB task (2000 simulated samples): Tables 5-7 and Figures 3-4. results_external_binary/ — SwinYNet predictions imported into the pipeline format, used as the specialized baseline. results_comparison_2b_0t/, results_comparison_4b_0t/ — paired VLM vs SwinYNet comparison: Tables 8-9 (McNemar tests, false positives by contaminant type). results_vlm2b_0t/, results_vlm_0t/ — multiclass FRB/RFI/NOISE runs on 3000 samples: Table 11. results_prompt_timing_2b/, results_prompt_timing_4b/ — prompt-latency audit on a stratified 200-image subsample: Table 10. results_real_binary_full_2b/, results_real_binary_full_4b/ — both models on the 1600 real FAST-FREX observations: Tables 12-15 and Figures 6-7. Software needed to reuse this material Python >= 3.10 with the environment pinned in pyproject.toml/uv.lock (managed with uv). The manifests, diagnostics, predictions and metrics are plain text (JSON Lines, CSV, JSON) and can be read with any standard tool, without installing the pipeline. Reproducing the runs additionally requires: simulateSearch (ascl:2205.025) for the synthetic PSRFITS; the Gemma 4 checkpoints google/gemma-4-E2B-it and google/gemma-4-E4B-it through Hugging Face Transformers, with a CUDA GPU; and, for the baseline, SwinYNet v1.0.0 with the authors' pretrained weights. Astropy, NumPy, scikit-learn, Matplotlib and PyTorch are used throughout. Data not hosted here The simulated PSRFITS files and rendered images (about 60 GB) are regenerated exactly from the pinned configuration and global seed. The real observations (about 412 GB of PSRFITS) are the public FAST-FREX data set, downloaded from the Science Data Bank by fetch_fast_frex.py; the manifests tracked here pin exactly which files were used and how each was preprocessed. Released under the MIT license. See README.md for step-by-step reproduction instructions.

Vibe coding for ophthalmologists

PURPOSE OF REVIEW: Recent advances in large language models (LLMs) have given rise to 'vibe coding', a conversational approach to software development in which users describe desired functionality in natural language and collaborate with an LLM agent to generate functional code. This review examines vibe coding through an ophthalmology-specific lens, identifying where it adds value and how it can be used responsibly. RECENT FINDINGS: Vibe coding platforms now enable nonprogrammers to create fully functional applications, including user authentication and database integration, from natural language prompts alone. In ophthalmology, clinicians may apply this approach to build clinical workflow, research and medical education tools. These applications are most effective when directed at niche, subspecialty-specific problems that are too small or too specialized to attract commercial development. SUMMARY: Vibe coding provides ophthalmologists with a pragmatic pathway to translate unmet needs into executable prototypes. For low-risk applications that do not directly influence clinical decision-making, clinicians may in certain settings be able to deploy vibe-coded tools independently. For higher risk applications, vibe coding does not replace professional software development, but it enables experimentation, clarifies user needs, and makes ideas more tangible. With appropriate guardrails, vibe coding can accelerate clinician-led innovation while preserving safety and accountability.

Posterior Ontology Memory: Bayesian Prediction Across Mutable Semantic Schemas

Posterior Ontology Memory (POM) models uncertainty in the vocabulary used by a symbolic memory, rather than assuming that its semantic schema is fixed. It treats mappings from surface predicates to latent semantic categories as uncertain and carries that uncertainty into prediction. The model combines a partition prior, collapsed Beta–Bernoulli rule models, noisy compiler judgments, and Bayesian model averaging. A temporal version allows schemas to split or merge and rule rates to reset, using exact enumeration for small reference cases and sequential Monte Carlo for approximate inference.In finite grounded-rule experiments, model averaging improves held-out prediction when surface predicates share latent behavior, but the advantage disappears when that assumption breaks down. The temporal study passed seven of eight predeclared gates: delayed split evidence met its criterion, while delayed merge evidence did not. In a separate study of 19 GitHub API migrations, a local language model retrieved 18 correct notices, compared with 7 for token overlap, although direct mutation classification was less reliable. The paper presents a bounded proof of concept for symbolic memory that retains uncertainty over changing semantic schemas.

Subset Heterogeneity in Reward Model Benchmarks: A CONFIRM Validation of 174 RewardBench 2 Models

Background. RewardBench 2 aggregates pairwise preference accuracy across heterogeneous task families (factuality, instruction following, safety, and others). A single leaderboard score can mask systematic subset specialization, yet standard benchmark reporting rarely tests whether accuracy is independent of task category. Methods. We applied CONFIRM, a chi-square test of independence with Cramér's V effect sizing and empirically anchored letter grades, to 174 publicly released reward models. For each model we constructed a 5 × 2 contingency table (subset × correct/incorrect) from published per-prompt scores (n = 1,763 prompts after excluding the non-binary Ties subset). The null hypothesis was that correct/incorrect outcomes are independent of subset category. Grades A–F reflect V magnitude (CONFIRM v2 thresholds); grade I denotes insufficient power. Non-significant results grade F only when power ≥ 0.80 to detect V = 0.10. Results. All 174 models rejected independence at α = 0.05 (all p < 0.02; median V = 0.272, range 0.082–0.467). No model received F or I. Grade distribution: A 36.8% (n = 64), B 50.6% (n = 88), C 11.5% (n = 20), D 1.1% (n = 2). Pooling across models, mean per-subset accuracy was lowest on Precise IF (36.3 per 100) and highest on Safety (75.7 per 100). In 93.1% of models the largest subset gap involved Precise IF as the weakest category; Safety was the strongest endpoint in 70.7% of cases. Conclusions. Published RewardBench 2 reward models exhibit statistically detectable subset heterogeneity at this sample size. Aggregate accuracy therefore understates structured performance imbalance, particularly weakness on precise instruction-following relative to safety-oriented subsets. CONFIRM provides a reproducible heterogeneity diagnostic to be read alongside ranking metrics. Plain-language summary. For all 174 reward models tested, the rate of correct judgments differed across the five task types by more than the prespecified statistical threshold. The direction of the difference was shared: 93.1% of models were weakest on precise instruction-following, and 70.7% were strongest on safety. Each model was tested on 1,763 prompts, enough sensitivity to detect even a small difference had one been present, so a model showing no difference would have been identifiable as such. None did. A single overall leaderboard score does not show this — two models with the same average can differ substantially underneath. This analysis measures whether a model's accuracy is uneven across task types, not which model is best overall, and it identifies a pattern without establishing its cause. Supplementary material. The deposited archive (rewardbench2_validation.zip) contains per-model results, subset breakdowns, contingency cell counts, validation flags, and step-by-step mathematical derivations for all 174 models. Competing interests. The author is affiliated with TraceSeis, Inc., which is developing CONFIRM as a commercial product. This constitutes a competing interest. All results are reproducible from the cited public data and the deposited analysis outputs. AI use disclosure. Generative AI tools were used during preparation of this work, in two distinct roles. For drafting and implementation: Anthropic Claude assisted with manuscript prose; the CONFIRM engine and analysis pipeline were implemented with AI coding tools (Cursor, Anthropic Claude) to the author's specification; and Google Gemini was consulted during writing and analysis runs. For review: Perplexity provided editorial review of a late draft, and xAI Grok was used as a general consistency check. This reflects the author's record of tool use and is not offered as an exhaustive log. Research design, statistical methodology, and interpretation are the author's. Because the analysis software was AI-implemented, every reported statistic was independently recomputed from observed cell counts and checked against pipeline output before reporting; per-model derivations are deposited as confirm_math.html and can be checked by hand. The author takes full responsibility for the contents of this record.

Human versus machine

For teachers to effectively use large-language-model(LLM)-based ratings in the formative or summative assessment of texts, it is essential to ensure that such ratings can assess student writing in a valid and reliable manner. This study investigates whether a validated human text-rating procedure (benchmark rating) can be replicated by an LLM-based rating procedure. We tested the replication with two genres of elementary school students’ text—narrative and instructive—using nine LLMs from three providers (OpenAI, Anthropic, Mistral). Each LLM generated three independent scores per text via structured, benchmark-aligned prompts that were then aggregated into a consensus score. Results showed that intrarater reliability was high to excellent, ICC(3, k) ≈ .68–.97, and alignment with human ratings ranged from moderate to strong, ICC(3, 1) ≈ .47–.85, with larger models consistently outperforming smaller ones. Systematic bias patterns emerged, varying by model and genre, indicating a need for calibration. Increasing output token windows and reducing temperature parameters mitigated truncation and schema-related failures. Although LLM-based benchmark ratings can approximate expert judgments and reduce the need for labor-intensive human triple coding, limitations remain regarding cost (for larger models), genre- and task specificity, and sensitivity to text presentation and student grade level—factors that constrain immediate classroom use, particularly for formative feedback.

Applicability of Infrared Thermography in Occupational Health for Multidisciplinary Clinical Decision-Making: A Systematic Review with Narrative Synthesis

Abstract Objectives To identify and synthesize evidence on the applications of infrared thermography in the assessment of work-related conditions and to examine its contribution as a non-invasive, adjunctive, and complementary tool to support multidisciplinary clinical assessment and decision-making in occupational health. Methods A systematic review with narrative synthesis followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 statement and was registered in the International Prospective Register of Systematic Reviews. Nine bibliographic databases were searched without date or language restrictions. Critical appraisal used Joanna Briggs Institute tools selected according to study design. Results Database searches identified 317 records, and seven additional reports were identified through other methods. Forty-nine studies published from 1975 to 2025 were included. Infrared thermography characterized thermal patterns, monitored occupational exposures and physiological responses, documented changes following interventions or across assessed conditions, and contributed to exploratory screening or predictive models. Applications addressed peripheral vascular responses to cold and vibration, musculoskeletal loading, heat stress, fatigue and drowsiness, respiratory protection, and emerging computational technologies. Observational designs predominated, with heterogeneous protocols and several small or convenience samples. No included study was classified as a diagnostic accuracy study. Conclusions Overall, infrared thermography was used primarily as a non-invasive, adjunctive, and complementary imaging method to document skin-surface temperature distribution and thermal patterns in occupational contexts. When interpreted alongside occupational history and clinical, physiological, ergonomic, and environmental information, thermographic findings may contribute to multidisciplinary clinical assessment and decision-making. Methodological limitations and heterogeneity warrant caution when transferring findings across work-related conditions and occupational settings.

RadTechBERT: a domain-specific language model for radiological technology evaluated using Japanese national examination-derived cloze questions

Modelling the Formation of Reality, Step by Step

Relational and information-based descriptions of reality have been explored in many domains, from relational readings of quantum mechanics to proposals that physical phenomena arise from the organization of information. This paper approaches the same questions from an unexpected side: sustained interaction with a large language model. In that interaction, a distinctive dynamic is visible in sequence. A space of weighted possibilities, shaped by context, realizes one continuation; the realization immediately feeds back and reshapes the conditions for the next. In other domains, the space of possibilities behind a single event can only be reconstructed across many runs but in a language model, it is visible at every step. And because nothing runs in the model between two realizations, each step can also be replayed. The loop can be abstracted into a substrate-neutral description of context fields meeting in a shared configuration. The paper lays four systems side by side under the same steps: a human, a language model, a detector, a particle, each described in the vocabulary of its own domain. Based on this, it proposes an examination: a conditioning signature obtained in controlled settings of a large language model and compared with how recorded distributions in physical domains respond to their own experimental knobs, with the criteria for failure stated in advance. The framework is then placed among its relational and informational neighbours: it shares their direction and differs in what it needs to assume and in the instrument it offers. The shift is small but consequential. Wherever a description calls an event “spontaneous”, this reading asks whether a participant has simply dropped out of the description. It also sets limits on its own use: the analogy cannot decide whether the underlying mechanisms are the same. Those claims have to be tested within the domains concerned.

Cell-type separability predicts annotation accuracy and outweighs algorithm choice: a factorial benchmark across seven scRNA paradigms

Automated cell-type annotation is a prerequisite for most single-cell RNA-sequencing (scRNA-seq) analyses, but the rapid proliferation of methods spanning marker-based, correlation-based, classical machine-learning, deep-learning, semi-supervised, large-language-model (LLM), and transformer foundation-model paradigms has outpaced head-to-head evaluation. Existing benchmarks rely on convenience samples of real datasets in which cell count, class imbalance, cell-type number, and differential-expression strength co-vary uncontrollably, precluding causal attribution of performance to any dataset property. To resolve this, we benchmarked 63 tools across seven paradigms using a Taguchi L9(3 4 ) orthogonal array that varies four dataset properties independently, progressively reconfiguring experimental control across five phases: fully controlled simulation, within-platform and cross-platform real-data validation, database-connected and LLM-based annotation under ontology-aware scoring, and fine-tuned foundation models. Using standardized oracle inputs and Cohen's κ, we found that, within the ranges tested, the major paradigms achieved comparable accuracy. Accuracy was predicted near-linearly by the separability of cell types in a shared expression embedding, measured as k-nearest-neighbor (kNN) purity, a relationship that held across sequencing platforms and in fine-tuned foundation models. We attributed the vast majority of κ variance to dataset structure and only a small share to tool identity. Computational cost traded against workflow accessibility rather than accuracy: accessible correlation-based and LLM-based approaches performed competitively, while foundation models matched them only after fine-tuning. Because our oracle design isolates algorithmic capability from upstream noise, these results reframe how methods should be selected: the field's near-term gains lie in strengthening infrastructure—prioritizing tool accessibility, standardized evaluation, and robustness to pipeline variation.

The Role of Artificial Intelligence in Teaching Ophthalmology Skills: A Systematic Review

Ophthalmology training requires visual interpretation, procedural skill, and supervised clinical reasoning, but trainee volume, faculty availability, and case mix constrain education. AI-enabled tools may support scalable instruction, assessment, and feedback. To evaluate AI-enabled interventions for improving ophthalmology diagnostic, clinical reasoning, and surgical skills, and summarize knowledge acquisition, AI performance metrics, and learner perceptions. This PROSPERO-registered review (CRD420251231199) followed PRISMA guidelines. Embase, Ovid MEDLINE, and Cochrane Library were searched through November 15, 2025. Eligible studies were observational or randomized trials in which trainees or clinicians performed ophthalmic diagnostic, clinical, or surgical tasks using AI-based instruction or assessment. Outcomes included diagnostic accuracy, knowledge, clinical reasoning, surgical skill, usability, satisfaction, and educational value. Risk of bias was assessed using ROBINS-I. Findings were synthesized narratively. Seven studies (200 participants) spanned image-based deep learning for diagnostic training, video-based deep learning for surgical assessment, and large language models for educational simulation. AI-tutored learners showed greater gains than lecture-based instruction in disease recognition and diagnosis (Cohen’s d = 0.82, p = .016); an AI myopia system produced large gains in classification and lesion detection (d = 1.3–2.3) where lecture alone showed none (p = .16–0.63). AI-based patient simulation was rated comparably to human actors (p = .48). AI-derived surgical metrics distinguished attending from resident performance (AUC 0.55–0.998). AI-enabled interventions show promise as adjuncts to ophthalmology training, particularly for diagnostic learning and objective feedback. Evidence remains limited by small samples and heterogeneity, requiring larger, standardized studies to define AI’s role.

Selective Forgetting in LLMs

Executive Summary This work approaches the problem of selectively forgetting knowledge from a large language model (LLM) for the purposes of safety, copyright, security, or otherwise. Also known as machine unlearning, this entails training a model to forget certain elements of the dataset on which it was trained. Unlearning methods must be evaluated both in terms of the extent to which the information has successfully been forgotten, and the performance of the unlearned model on the remaining (retained) data. We build on the work of TOFU (Task of Fictitious Unlearning) [21], which provides a dataset and benchmark for evaluating unlearning techniques. We create a new, TOFU-inspired question–answer dataset for the task of machine unlearning. The new dataset includes 10,500 question–answer pairs relating to over 1,000 distinct, synthetic entities of several types. Each question–answer pair is tagged with the entities it refers to, with the graph representation of our dataset containing over 2,600 edges between different entities. We perform two experiments with our dataset. The first experiment aims to capture whether the difficulty of forgetting a concept from a LLM depends on its granularity. For example, is unlearning more likely to be successful if forgetting a single book, rather than the book's author (as an author is connected to multiple books)? We find that granularity does not have a tangible effect on model performance in our dataset. There may be a small effect from granularity on the difficulty of forgetting, but this is not statistically significant across our results. Our second experiment explores the knock-on effect of forgetting a relationship between two entities. For example, if unlearning has been run on a model to forget only who wrote a book, but not the book or author themselves, is the model worse at responding to other questions about that book or author? We find that the model performance is lower on questions that contain the entities pertained in the relationship, than on those that do not. 1

Cerebral microbleeds on susceptibility-weighted imaging, cognitive domains, and serum neurofilament light chain in adults with hypertension

Background Hypertension is associated with cognitive impairment, and cerebral microbleeds (CMBs) may reflect hemorrhagic small-vessel injury. We examined whether CMB burden and location were associated with global and domain-specific cognition in adults with primary hypertension and whether serum neurofilament light chain (sNFL) provided additional discrimination for identifying mild cognitive impairment (MCI) beyond CMB count. Methods In this single-center cross-sectional study, 173 hospitalized adults aged 31–95 years with a pre-existing diagnosis of primary hypertension underwent brain magnetic resonance imaging (MRI) with susceptibility-weighted imaging, the Montreal Cognitive Assessment (MoCA), and sNFL measurement by enzyme-linked immunosorbent assay (ELISA). Six MoCA-derived cognitive domain z-scores were calculated. CMB burden was categorized as 0, 1–4, or ≥5 lesions. Multivariable logistic and linear regression models were adjusted for age, sex, education, systolic blood pressure, diabetes, and prior stroke. Sensitivity analyses additionally adjusted for other cerebral small vessel disease markers, modeled the CMB count as a continuous variable, and excluded participants with prior stroke. Results CMBs were detected in 82 patients (47.4%) and MCI in 83 patients (48.0%). Greater CMB burden was independently associated with MCI (adjusted odds ratio (OR): 4.53, 95% CI 1.95–10.55). After covariate adjustment and false-discovery-rate correction, greater CMB burden was associated with lower executive function, orientation, attention, and visuospatial function scores, but not with memory or language. Cognitive domain scores did not differ according to CMB location. CMB count and sNFL were inversely correlated with cognitive performance. The areas under the curve for identifying MCI were 0.683 for CMB count, 0.794 for sNFL, and 0.792 for the combined model. The combined model had a larger area under the curve than CMB count alone ( p < 0.001) but did not differ from sNFL alone ( p = 0.768). Conclusion In adults with primary hypertension, greater CMB burden was associated with MCI and selected cognitive domains, whereas cognitive performance did not differ according to CMB location. sNFL had a numerically higher area under the curve than CMB count, and adding CMB count did not improve discrimination beyond sNFL. These findings may help characterize cognitive vulnerability in hypertension; therefore, prospective validation is required before clinical application.