SDAR employs a symbolic reasoning engine to guide agentic decision making, connecting low-level visual cues with structured symbolic representations of events, and enables interpretable reasoning chains that capture causal relationships, contextual dependencies, and event categories.
Abstract
Analyzing complex environmental and behavioral activities requires intelligent systems that not only perceive visual scenes but also reason about underlying events and interactions. Conventional computer vision models often operate at the object-detection level, limiting their ability to generalize across diverse outdoor contexts or provide interpretable explanations for higher-level behaviors such as waterway activity, construction-site analysis, and waste management. This work introduces symbolic-driven agentic reasoning (SDAR), a multimodal framework that bridges perception and cognition for event-level analysis. SDAR employs a symbolic reasoning engine to guide agentic decision making, connecting low-level visual cues with structured symbolic representations of events. Grounding is performed with pre-trained open-vocabulary perception models, without task-specific fine-tuning, while the event logic memory is derived from unlabeled exploration samples rather than manually specified rules. This design enables interpretable reasoning chains that capture causal relationships, contextual dependencies, and event categories. Comprehensive evaluations across 34 real-world field-scene tasks show that SDAR improves average precision by over 10%, achieves 91.7% accuracy in zero-shot open-set detection, and enhances event-level reasoning performance by more than 20%.
Robots operating in human environments need memories that capture not only what objects exist and where, but also how people use them over time and how individual interactions compose into goal-directed activities. Existing 4D scene graphs preserve object and place histories but omit activity structure, whereas activity representations are either not grounded in persistent 3D scenes or rely on externally provided event boundaries and object associations. We present GESTO (Grounded Event and Spatio-Temporal memOry), a spatio-temporal memory that couples a persistent 4D scene graph with a two-level hierarchy of atomic human--object interactions and goal-driven events. From an RGB-D observation stream, GESTO automatically extracts timestamped interactions, grounds them to persistent scene entities, groups them into events, and uses event context to refine uncertain object associations. A relation-aware tool-calling agent queries the resulting memory for activity-centric spatio-temporal reasoning. We evaluate GESTO on the reproducible text, binary, and time categories of an existing benchmark, together with 40 new Space2Event and Event2Space queries. GESTO achieves scores of 0.71, 0.75, and 0.70 on the standard categories, approaching a method supplied with ground-truth event and object grounding, while substantially outperforming the same reasoning framework when these inputs are removed. It further achieves 0.73 and 0.75 on Space2Event and Event2Space queries. Ablations show that hierarchical event structure and context-aware grounding refinement provide complementary benefits, supporting activity-grounded hierarchical memory for retrospective reasoning in dynamic human environments.
Ermanno Bartoli, Buwei He, Dennis Rotondi et al.· 0 citations
High-complexity operational environments require methods that detect and anticipate temporally distributed patterns rather than classify isolated events. This paper introduces TRACTA (Temporal Reasoning and Capability-Trajectory Analysis), a controlled synthetic benchmark for temporal structural reasoning in high-complexity event-driven systems, instantiated through Multi-Domain Operations (MDO)-like scenarios. The benchmark includes three tasks: early_warning, pattern_detection, and run_classification, and compares raw-event neural models, a contract-lite semantic baseline, and a neuro-symbolic configuration operating on semantically grounded trajectories. Results show that raw event-level learning remains informative, but learned temporal modeling over semantic capability and contextual direct-impact trajectories achieves the highest aggregate point estimates, with the largest margins on the temporal tasks. Ablation analysis indicates that capability dynamics, contextual impacts, and temporal structure contribute complementary information. Shortcut diagnostics indicate that the most direct cross-run global-identifier shortcut is controlled in the primary neural input view, while residual shallow signals remain. Overall, the findings support a bounded methodological conclusion: in controlled synthetic settings, semantically grounded trajectories provide an effective representation for temporal structural reasoning, supporting further investigation of semantic interfaces between event data, structured representations, and temporal learning.
Michael Romei de Socio, G. Pozzato, Alessio Merlo· 0 citations
We consider First-Order Logic (FOL)-based semantic communication for neuro-symbolic decision-making in collaborative environments such as autonomous driving networks. Each connected autonomous vehicle (CAV) converts its partial sensor observations into a natural-language scene description and corresponding grounded FOL evidence. Under an uplink budget, a semantic encoder at each car selects the observations most informative for evaluating traffic rules and transmit to a Road Side Unit (RSU). The RSU fuses all received evidence, evaluates collaborative rules, performs logical deduction for vehicle-specific safety and right-of-way information for constrained downlink transmission. Each CAV combines the received deductions with its local description, enabling a local LLM agent to select a high-level driving action. We develop a principled, verifiable semantic communication method using a random-support Dirichlet--Categorical model of inductive logical probability, providing a modern statistical reinterpretation of Carnap's and Hintikka's systems. From this model, we derive a goal-oriented semantic information-bottleneck formulation that prioritizes evidence transmission by its reduction of uncertainty over task goals. Using 152 traffic rules extracted from the California Driver Handbook, we evaluate the framework on MDrive simulator in CARLA. Under identical communication budgets, semantic evidence selection completes every scenario without safety hazards, whereas uniform evidence selection produces collisions, showcasing semantic communication's superiority.
High-level planning for autonomous driving is a knowledge-intensive engineering decision task that requires accurate scene understanding, timely inference, and internally consistent action selection. Vision-language models (VLMs) can make intermediate reasoning explicit, but their use in deployed planners is constrained by costly structured supervision, unnecessary reasoning in routine scenes, and possible inconsistencies between generated rationales and driving actions. We present a cognitive dual-process planning framework that represents planning-relevant scene knowledge in a machine-parsable structured chain-of-thought (S-CoT) schema. An automated data engine integrates perception foundation models, critical-path filtering, and an expert VLM to generate S-CoT supervision without manual annotation of individual rationales. A lightweight visual Arbiter estimates scene complexity from multilevel vision-encoder features before language decoding and routes each input to either fast meta-action prediction or slow structured reasoning. For slow-path outputs, a deterministic rule-based validator checks whether the parsed S-CoT fields are consistent with the final meta-action and provides verifiable rewards for Group Relative Policy Optimization (GRPO). In a 195-scene manual audit, the generated annotations achieve 91.8\% CoT accuracy and a 98.5\% Logical Consistency Score (LCS). On 574 manually verified NAVSIM test samples, the planner achieves 80.14\% planning accuracy and 97.20\% LCS while reducing average latency by 17.39\% relative to applying slow reasoning to every scene. Evaluation on external long-tail subsets further identifies conditions under which routing and planning performance degrade. Together, these results show how explicit scene knowledge can be operationalized through adaptive reasoning and rule-based verification to support high-level VLM planning decisions.
Zhongyao Yang, Haoyu Li, Yuchen Yan et al.· 0 citations
The transition toward fully autonomous Digital Building Permitting (DBP) requires Automated Compliance Checking (ACC) systems to verify Building Information Models (BIM) against natural-language regulations. While Large Language Models (LLMs) offer strong semantic comprehension, integrating them into ACC introduces “Spatial Hallucinations” and “Serialization Bottlenecks” when processing high-dimensional BIM graphs. This research proposes an Agentic Neuro-Symbolic Framework that decouples semantic interpretation from geometric verification. Instead of relying on generative text for spatial reasoning, an Agentic LLM acts as a dynamic logic synthesizer orchestrating a deterministic geometry kernel (IfcOpenShell). The artifact was evaluated against the Australian National Construction Code (NCC 2022) across three stratified tiers: Semantic-Geometric Alignment, Multi-Parametric Dependencies, and Relational Topology Reasoning. Results demonstrate the framework autonomously resolves ontological ambiguity and synthesizes execution logic dynamically. By implementing connectivity graph traversal, the system isolates structural sub-graphs, reducing computational complexity from O(N) to O(K) and bypassing context-window limits. Offloading calculations to a deterministic environment achieves a highly deterministic accuracy rate for spatial queries, yielding immutable BIM Collaboration Format (.bcfzip) audit trails. Ultimately, this research establishes a scalable foundation for autonomous compliance, demonstrating that AI reliability in engineering significantly improves when probabilistic models orchestrate deterministic tools rather than predicting physical realities.
Visual question answering (VQA) in autonomous driving scenarios demands strong spatiotemporal reasoning capabilities, yet existing vision-language models lack explicit modeling of dynamic relationships in complex traffic scenes. We propose STGraphVQA, a framework that represents driving scenes as dynamic spatiotemporal graphs, where nodes denote traffic participants, edges encode spatial and semantic relationships, and the temporal dimension captures their evolution. A hierarchical reasoning architecture progressively processes information through perception, relation, and decision layers, simulating the human driving cognitive process. A logit-level constrained decoding mechanism further ensures that generated answers comply with traffic rules and physical feasibility. Experiments on DriveLM and STRIDE-QA demonstrate that STGraphVQA significantly outperforms state-of-the-art baselines, achieving a Top-1 accuracy of 76.8% and a reasoning chain completeness of 82.3%, providing a promising direction toward interpretable autonomous driving systems.
Dengquan Zhang, Jiayi Wu, Qingshan Wang et al.· International Conference on...· 0 citations