This work establishes a foundation for trustworthy natural language interfaces by enabling AI systems to recognize when generated specifications may not be reliable, and demonstrates improved translation reliability, robustness under the evaluated cross-tier shifts, and effective uncertainty-aware abstention.
Abstract
Translating natural language instructions into machine-interpretable formal specifications enables robots and autonomous systems to plan, reason, and formally verify their behavior. However, existing translation models typically generate a specification for every input, even when the result is unreliable or fails to capture the user's intent, creating risks in safety-critical applications. Inspired by selective conformal prediction, we propose a selective translation framework that not only generates formal specifications but also determines when they can be trusted. Reliability is scored by two complementary black-box signals, the fidelity of the specification back-translated into natural language and the dispersion of repeated translations under exact semantic equivalence, which fail on different errors and jointly separate incorrect translations more sharply than either alone. Conformal risk control calibrates this score into a decision that accepts a specification or abstains, with a distribution-free bound on the rate at which incorrect specifications are accepted for execution, and a conformal anomaly detector on instruction embeddings screens out-of-distribution inputs before any translation is attempted. The proposed framework is general across formal specification languages, with experiments on Signal Temporal Logic (STL), Linear Temporal Logic (LTL), and geometric Spatio-Temporal Logic (SpaTiaL) demonstrating improved translation reliability, robustness under the evaluated cross-tier shifts, and effective uncertainty-aware abstention. This work establishes a foundation for trustworthy natural language interfaces by enabling AI systems to recognize when generated specifications may not be reliable.
Signal Temporal Logic (STL) enables rigorous verification and control of cyber-physical systems, but writing correct specifications requires expertise that most requirement holders lack. Large language models can translate natural-language (NL) requirements into STL, yet stronger translators alone approach an accuracy...
This case study assesses the feasibility of combining LLM-based extraction, metamodel-guided refinement, automated validation, and human correction for requirements formalization and evaluates the approach on the ten published ACAS Xu safety properties.
Sally Ismail, Patrick Tessier· Proceedings of the ACM/IEEE...· 0 citations
GenCTL is proposed, a prompt-based framework for effective and model-aware NL2CTL translation without task-specific fine-tuning that improves the reliability and practical checkability of LLM-generated CTL specifications.
Ran Tao· Poster Volume 0007 The 2026...· 0 citations
RAISE is introduced, which trains policy synthesizers from formal verification in two stages, verified supervised fine-tuning (SFT) followed by a reinforcement learning (RL) stage that learns from verifier signal, and it is found that SFT succeeds largely by letting models express authorization logic they already have.
Ying-Ming Zhou, Adarsh Vatsa, William Eiers· 0 citations
CEDAR is presented, a counterexample-guided framework that grounds instructions as regular languages over environment event traces and represents both skills and specifications as deterministic finite automata, suggesting that regular languages offer a practical verification layer between natural-language instructions...
Le Chen, Alvaro Velasquez, Ashutosh Trivedi· 0 citations
This work investigates whether relational specifications can ground SLM behaviour by mathematical proof rather than statistical regularities alone, and gives practitioners a route to deploy verifiable SLMs in safety-critical settings.
Chia Sabah· Proceedings of the ACM/IEEE...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.