The first systematic study of imperfect specifications is presented and an automated framework to repair them to enhance the quality of resulting Verilog design is proposed, demonstrating the capabilities of specification repair by {VClare} as well as further potential of LLMs in front-end hardware design.
Abstract
Large language models (LLMs) have demonstrated promising capabilities in generating Verilog code from natural language specifications. However, human-written specifications often contain semantic imperfections such as vagueness, contradictions, and incompleteness, which can significantly degrade the quality of hardware design generated by LLMs. In this paper, we present the first systematic study of imperfect specifications and propose an automated framework {VClare} to repair them to enhance the quality of resulting Verilog design. The proposed framework explores two complementary repair paradigms. The \textit{Spec-Level Repair} conducts LLM-driven inconsistency mining directly on the specification texts, while the \textit{Sim-Level Repair} employs simulation-based behavioral clustering with optional test-time inconsistency arbitration. In addition, we propose two new benchmark datasets with systematically injected specification defects. The first benchmark dataset is derived from the VerilogEval-human benchmark targeting single-module tasks, while the other benchmark dataset is derived from the ComplexVDB dataset and contains 53 multi-module tasks that reflect more realistic engineering scenarios. For single-module tasks, the {VClare} framework can repair the imperfections in the specifications effectively and thus enhance the pass rate of the generated Verilog design by 12.7\%, while for the multi-module tasks this enhancement can reach 13.7\%, demonstrating the capabilities of specification repair by {VClare} as well as further potential of LLMs in front-end hardware design.\footnote{The two benchmark datasets are released at https://anonymous.4open.science/r/VClare/.
Large language models can generate register-transfer-level (RTL) designs directly from natural language specifications. Their failures, however, arise mostly from understanding rather than coding \cite{zhang2026understanding, qiu2025towards}. A specification is informal and ambiguous, the model's interpretation stays implicit, and every misreading is committed silently into Verilog, where only simulation can expose it. Intermediate representations make the interpretation partly explicit, yet existing works don't verify the interpretation against the specification, and repair simulation failures at the code level regardless of where the misreading originated. VeriRefine instead treats specification refinement as a verifiable stage of RTL generation. It progressively refines the prose specification into an explicit, schema-constrained account of design intent, expressed as per-signal Abstract Signal Transition Functions (ASTFs) that commit each signal's logic style, clock domain, and reset behavior before any code exists and ground every behavior in a verbatim specification sentence. The refined specification then passes a five-layer audit spanning soundness, completeness, consistency, FSM integrity, and core RTL design rules, so interpretation errors are repaired at the representation level before any Verilog is generated. Once code is generated, each simulation failure is classified as an understanding error or a coding error and routed back to the corresponding stage for targeted repair. Because every signal's hardware class is fixed during refinement, synthesizability becomes a structural property of the pipeline rather than a post-hoc check. With Claude Sonnet 4.6, VeriRefine reaches 94.0\% functional correctness on RTLLM v2.0 and 98.1\% on VerilogEval-Human v2.
Xiangfei Kong, Tasnim Tabassum, Marwan Abdelwahab et al.· 0 citations
Large language models (LLMs) are increasingly used to generate register-transfer-level (RTL) designs from natural-language specifications. However, assessing functional correctness at early stages remains a fundamental challenge. Existing oracle-free approaches rely either on simulation-based agreement, which depends on LLM-generated testbenches that can fail or vary across models, or on LLM-as-a-judge heuristics, which produce inconsistent predictions. We introduce NoTB, an oracle-free triage framework that infers correctness from cross-model formal consensus. NoTB generates RTL implementations from multiple independently trained LLM families and applies Sequential Equivalence Checking (SEC) to identify designs that are provably equivalent. We show that the diversity of model families within an SEC-equivalent cluster induces a calibrated correctness signal, enabling risk-coverage tradeoffs without requiring testbenches. On 78 CVDP RTL-generation tasks, four-family formal consensus achieves 94.7% precision at 27% coverage; three-family consensus achieves 87% precision at 33% coverage. These operating points give designers a tunable accept/defer rule before a trusted testbench or golden RTL is available. Overall, NoTB demonstrates that formal cross-model agreement provides a reliable basis for high-confidence triage without model-dependent oracles
Elisavet Lydia Alvanaki, Je Yang, Biruk B. Seyoum et al.· 0 citations
A reproducible benchmarking platform that evaluates open-source LLMs on Verilog RTL generation across 50 curated tasks consisting of combinational, sequential, finite state machine (FSM), and mixed designs, enabling reproducible evaluation of generative AI for hardware design workflows.
It is shown that task-equivalent implementations can differ substantially in verifiability and that implementation diversity helps find verification-friendly artifacts, and that implementation diversity helps find verification-friendly artifacts.
This work proposes VPID, a multi-agent framework for generating complex Verilog that achieves monotonic functional improvement and introduces an experience-guided refinement strategy that distills historical waveform mismatches into constraints, guiding the targeted debugging for the unverified ports.
Hongguang Wang, Jiaming Guo, Rui Zhang et al.· 0 citations
This paper addresses automated unit test generation with large language models (LLMs). LLM-based test generation has not yet attained a quality level sufficient for practical use in industry. Although LLMs often reproduce API syntax faithfully, they frequently disregard semantic usage constraints and execution-environment dependencies, leading to assertion failures, mock-related errors, and reference/resolution errors. A prior failure analysis of Java unit test generation using GPT-4o classified 2980 trials into eight failure patterns and identified three root-cause mechanisms: external context ignorance, internal context ignorance, and a syntax–semantics gap. Building on that analysis, this paper proposes a prompt design comprising three strategies: (1) making the execution state explicit in the generated test, (2) stating semantic constraints explicitly, and (3) injecting environment constraints prior to generation. In contrast to generic techniques such as few-shot learning or chain-of-thought prompting, each proposed strategy is tied to a specific root-cause mechanism, yielding a systematic design in which each rule is explicitly justified by its correspondence to a specific root-cause mechanism. Experiments on 298 methods with five models (GPT-4o, GPT-5, GPT-5.1-Codex, Claude Sonnet 4.5, and Gemini 2.5 Pro) show improved test execution success rates for every model, with absolute gains ranging from 1.1 to 21.1 percentage points (pp). Mock-related errors were reduced by 61.9%–99.2% relative to the baseline prompt, demonstrating effectiveness against the targeted failure patterns. Finally, conditions under which the strategies transfer to other code-generation tasks are discussed, along with limitations on their scope.