This work evaluates several state-of-the-art VLMs across two datasets and multiple prompting strategies to test whether an explicit distinction between hazard and anomaly changes model behavior, and shows that explicitly separating anomaly from hazard provides a more informative evaluation of VLM safety reasoning and exposes failure modes that binary safety judgments can obscure.
Abstract
Modern safety-critical systems increasingly rely on human-robot interaction to reduce disaster risk and support decision-making during emergencies. Vision-Language Models (VLMs) are promising for these settings because they can interpret complex scenes and communicate safety-relevant information, but they still require careful evaluation to ensure reliable safety reasoning. In particular, current evaluations often frame danger recognition as a binary decision (Safe/Unsafe), making it unclear whether a model is identifying true physical hazards or merely reacting to unusual scene elements. We address this limitation by introducing an explicit distinction between hazard and anomaly, and by separately recognizing hazardous and anomalous states. We evaluate several state-of-the-art VLMs across two datasets and multiple prompting strategies to test whether this distinction changes model behavior. Our results show that VLMs frequently misinterpret anomalousness as hazardousness, revealing an over-reliance on contextual irregularity as a proxy for danger. We further show that explicitly separating anomaly from hazard provides a more informative evaluation of VLM safety reasoning and exposes failure modes that binary safety judgments can obscure. Our public dataset is available on Roboflow https://app.roboflow.com/vlm-in-context-anomaly-and-hazard-detection/camera-ready-roman-ds.
In high-risk environments such as disaster response, situational awareness depends not only on detecting hazards but also on communicating them clearly to human operators. Vision Language Models (VLMs) have shown strong potential for scene understanding in safety-critical settings, yet their value as part of human-facing robotic systems remains underexplored. We present a VR-based Human Robot Interaction framework for studying how VLM-assisted robots can support situational awareness in simulated hazardous environments. In our system, a robot explores a virtual scene and queries a VLM to identify potential hazards and annotate user-facing points of interest. These annotations are presented to a human operator through an immersive VR interface. This framework enables controlled evaluation of both robotic hazard identification and the communication of safety-critical information to users. Results from our study indicate that the annotated VR interface was preferred over the unannotated baseline and that participants reported high clarity, usefulness, and comfort when interacting with the system. These findings suggest that combining VLM-based robotic perception with immersive visualization is a promising approach for supporting situational awareness in hazardous settings.
Mohammad Eskandari, M. Indukuri, Stephanie M. Lukin et al.· 0 citations
FlightLLM, a prior-guided semantic LLM-based approach for interpretable flight safety analysis that achieves competitive classification performance while generating direct and reasonable explanations for event causes is proposed.
Large vision--language models (VLMs) can reason step by step about complex visual scenes, but this open-ended, autoregressive chain-of-thought (CoT) approach is poorly suited to safety-critical, rule-governed settings such as industrial surveillance, where decisions must be bounded, deterministic, and low-latency. Because CoT inference cost scales jointly with reasoning length and the number of concurrent streams, it creates a throughput bottleneck that precludes the real-time, multistream monitoring required for industrial accountability. Here we present MonitorVLM-v2, a deployment-oriented framework that recasts VLM-based safety assessment as probabilistic inference over a finite regulatory decision space, compressing multimodal reasoning into single-step rule-ID predictions and reducing decoding from a variable-length sequence to a single token. We introduce symbolic policy optimization (SymPO), a novel contrastive policy optimization algorithm that sharpens decision boundaries within this finite symbolic space, together with an entropy-driven triage mechanism that routes uncertain predictions to human reviewers for expert confirmation. In a four-month prospective deployment across 10 concurrent camera feeds in an operational underground mining facility, MonitorVLM-v2 achieved a 19.45-fold increase in inference speed and identified 2.78 times as many confirmed violations as the site's routine manual inspection workflow, demonstrating the practical value of compressed symbolic decision-making for real-time, auditable industrial monitoring.
Jiang Wu, Sichao Wu, Yinsong Ma et al.· 0 citations
This study proposes “Vanguard AI,” an end-to-end cloud-edge architecture utilizing an experimentally validated YOLOv11m detection framework, delivering the first unified, production-viable architecture for context-aware compliance auditing and dynamic risk assessment in industrial environments.
Vishrutkumar Patel, Amol R. Madane, Srijit Maiti et al.· SN Computer Science· 0 citations
The overtake car maneuver involves high risk and complex judgement. For autonomous vehicles this is challenging, especially for human-initiated overtake requests. If a user requests the maneuver there must be a rapid safety assessment. Language models have great potential to classify safety with explanations, but they struggle to disentangle critical information from complex vehicular environments. We apply CLEAR (Cue Learning using Evolution for Accurate Recognition) to evolve prompt cues that optimize the ability of language models to correctly predict safety scores for overtaking maneuvers. To achieve this, we create a novel open-source symbolic traffic model EvoDrive designed specifically for EC research, which outputs LLM-readable snapshots. We show that LLMs + EvoDrive with CLEAR can reduce error by more than 20% compared to without CLEAR, with statistically significant results. Analysis shows evolved cues are coherent and have reduced variability in LLM output.
Peter J. Bentley, S. Lim, Fuyuki Ishikawa et al.· Annual Conference on Genetic...· 0 citations