Large Acoustic Models For Visual Scene Narration With Large Visual Embeddings

Abstract

Accurate, context-aware visual scene narration is vital for restoring environmental comprehension to blind and visually impaired (BVI) individuals. However, existing assistive systems are heavily polarized, delivering either shallow, object-naming captions or suffering from massive end-to-end latencies exceeding 300 seconds, which are incompatible with on-device utility. To resolve these challenges, this study introduces a multimodal, low-latency streaming pipeline that leverages Large Visual Embeddings (LVE) with Large Acoustic Models (LAM) to generate expressive, emotionally inflected scene narrations. Spatiotemporal features are extracted using an AnyRes variable-resolution tiling strategy on a LLaVA-NeXT-Video-7B backbone and aligned via a learnable projection matrix directly with a 3B-parameter Orpheus speech-synthesis model. To make on-device deployment feasible, we compile our acoustic decoders into parallelized TensorRT engines and implement speculative decoding, which reduces end-to-end processing latency from 316.50 seconds to a near-instantaneous 3.82 seconds (an $82.8\times $ computational speedup). Furthermore, a randomized, double-blind crossover study conducted with $N = 5$ BVI participants demonstrates that our proposed system yields a 46.8% increase in situational comprehension and cuts NASA-TLX cognitive workload scores in half (from 68.42 to 31.15) compared to standard object-detection baselines. These results demonstrate an edge-optimised real-time assistive scene-to-sound translation tool.