frb-vlm-benchmark: a zero-shot Vision-Language Model benchmark for Fast Radio Burst detection

Abstract

Code and evaluation artifacts supporting the article "Generalist Vision-Language Models for Fast Radio Burst detection: a zero-shot benchmark against a specialized detector" (Santos, Queiroz, Duarte, de Farias & Batista), submitted to The Astrophysical Journal Supplement Series (manuscript AAS80842; preprint arXiv:2607.07382). The pipeline simulates PSRFITS dynamic spectra containing Fast Radio Bursts (FRB), structured Radio Frequency Interference (RFI) and system noise; renders them as anonymized PNG images; classifies those images with small, open-weight Vision-Language Models (Gemma 4 E2B/E4B) in a zero-shot, prompt-only regime; and evaluates the predictions, including a paired, sample-by-sample comparison against the specialized detector SwinYNet. The same models, prompts and decision threshold are then applied unchanged to the 1600 real FAST observations of the FAST-FREX data set (doi:10.57760/sciencedb.15070). Contents and file formats (169 files) Source code (56 .py files) — Python packages implementing each pipeline stage: simulate_dataset/ (synthetic PSRFITS generation and selection of the balanced binary subset), plot_dataset/ (PSRFITS to anonymized dynamic-spectrum PNG), vlm_classifier/ (zero-shot classification with Gemma 4 via Hugging Face Transformers), evaluation/ (discrete, probabilistic and paired metrics), benchmark_export/ and benchmark_predictions/ (anonymized FITS export and import/comparison of external detector output), prepare_real/ (preprocessing and rendering of real search-mode PSRFITS), plus tests/ (unit tests). Entry points: run_pipeline.py (CLI integrating all stages), fetch_fast_frex.py (selects and downloads the FAST-FREX files), prompt_timing_experiment.py (prompt-latency audit), and the read-only diagnostics check_burst_visibility.py, check_burst_visibility_sim.py, analyze_visibility.py, summarize_real_run.py. Configuration (2 .yaml, 1 .toml, 1 uv.lock) — config.yaml and config_binary.yaml are the exact configurations used for the published runs, pinning the instrumental setup, the global seed and the image-rendering protocol; pyproject.toml and uv.lock pin the Python environment. Data manifests (14 .jsonl files) — line-delimited JSON. dataset/metadata/labels.jsonl and image_manifest.jsonl record, per sample, the ground-truth class, the injected burst or RFI parameters, the source PSRFITS and the anonymized image identifier; dataset_binary/metadata/ holds the same for the balanced 2000-sample binary benchmark; dataset_real_full/metadata/image_manifest.jsonl documents the per-file preprocessing of all 1600 rendered FAST-FREX observations (time window, masked channels, decimation). The predictions.jsonl file in each results directory holds one raw model response per sample. Tabular data (21 .csv files) — the burst-visibility diagnostics that underlie Section 7 of the article (fast_frex/burst_visibility_full.csv and dataset/metadata/burst_visibility_sim.csv: dedispersed peak significance, recovered peak time and catalogued parameters per burst), the FAST-FREX input manifest and burst-parameter tables, and the paired_scores.csv files aligning the continuous scores of each VLM and of SwinYNet sample by sample. Metrics and reports (15 .json, 12 .txt) — the aggregate metrics computed by evaluation/ (accuracy, macro-F1, per-class precision and recall, ROC-AUC, average precision, Brier score, ECE, McNemar tests and bootstrap intervals) and their human-readable summaries, one set per run. Figures (27 .png, 16 .pdf) — the article figures and the diagnostic plots produced by each run; figures_src/ regenerates them from the tracked artifacts. Results directories and their relationship to the article results_vlm2b_0t_binary/, results_vlm_0t_binary/ — Gemma 4 E2B and E4B on the binary FRB vs NON_FRB task (2000 simulated samples): Tables 5-7 and Figures 3-4. results_external_binary/ — SwinYNet predictions imported into the pipeline format, used as the specialized baseline. results_comparison_2b_0t/, results_comparison_4b_0t/ — paired VLM vs SwinYNet comparison: Tables 8-9 (McNemar tests, false positives by contaminant type). results_vlm2b_0t/, results_vlm_0t/ — multiclass FRB/RFI/NOISE runs on 3000 samples: Table 11. results_prompt_timing_2b/, results_prompt_timing_4b/ — prompt-latency audit on a stratified 200-image subsample: Table 10. results_real_binary_full_2b/, results_real_binary_full_4b/ — both models on the 1600 real FAST-FREX observations: Tables 12-15 and Figures 6-7. Software needed to reuse this material Python >= 3.10 with the environment pinned in pyproject.toml/uv.lock (managed with uv). The manifests, diagnostics, predictions and metrics are plain text (JSON Lines, CSV, JSON) and can be read with any standard tool, without installing the pipeline. Reproducing the runs additionally requires: simulateSearch (ascl:2205.025) for the synthetic PSRFITS; the Gemma 4 checkpoints google/gemma-4-E2B-it and google/gemma-4-E4B-it through Hugging Face Transformers, with a CUDA GPU; and, for the baseline, SwinYNet v1.0.0 with the authors' pretrained weights. Astropy, NumPy, scikit-learn, Matplotlib and PyTorch are used throughout. Data not hosted here The simulated PSRFITS files and rendered images (about 60 GB) are regenerated exactly from the pinned configuration and global seed. The real observations (about 412 GB of PSRFITS) are the public FAST-FREX data set, downloaded from the Science Data Bank by fetch_fast_frex.py; the manifests tracked here pin exactly which files were used and how each was preprocessed. Released under the MIT license. See README.md for step-by-step reproduction instructions.