Vision-Language Models Evaluation for Telecom Field Safety Auditing

Abstract

Quality assurance of fiber-optic field installations still mostly relies on manual visual inspection, which does not scale with installation volume. This paper investigates whether vision-language models (VLMs) can automate this safety audit task. We benchmark 18 VLMs spanning frontier, consumer-GPU and edge-device tiers on a 14-requirement compliance checklist over a dataset of real and synthetic images. A decomposed scoring methodology separates relevance recall (identifying applicable requirements) from conditional F1 (judging compliance correctly), revealing failures hidden by standard F1 score. Some models are strong graders (conditional F1 of 0.89), but weak relevance filters (silently skip 37% of gradeable criteria), thus suggesting a miscalibration problem instead of a competence ceiling. Finally, some local (consumer-GPU) models can match or even exceed frontier APIs for this task. Local models further show output stability varying sharply across models and that reasoning substantially helps weaker models while offering little benefit to strong ones. Edge-tier models remain too inaccurate without further fine-tuning.