Skip to content

DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis

Sep 2026 · 0 citations · 31 references
Computer Science

TL;DR

The proposed Koan-Safe, which combines a prompt-only intent parser, a replaceable generator, and structural repair with default safety parameters, is proposed, which supports explicit trade protections and execution-based evaluation, while distinguishing declared safety from a general guarantee.

Abstract

A structurally valid DeFi workflow can still authorize a costly trade. We introduce DeFiFlowBench, a benchmark of 207 team-authored prompts for natural-language DeFi workflow synthesis. It measures graph coverage, configuration completeness, and declared safety predicates, then tests supported trade configurations on a local EVM. Direct, constrained, and few-shot prompting produce 14-19 unsafe held-out executions per configuration under a fixed 5% price-impact cap. A slippage bound derived from a quote does not prevent the price impact of the order itself. We propose Koan-Safe, which combines a prompt-only intent parser, a replaceable generator, and structural repair with default safety parameters. On 75 held-out workflow prompts, its hybrid variant scores 0.67 on the static safety proxy, compared with 0.33 for the best baseline. Koan-Safe records no unsafe executions on the saved benchmark outputs. A matched-candidate ablation produces 14-17 unsafe executions when enforcement is disabled. Additional tests expose the limits of default injection: permissive existing thresholds can still authorize unsafe trades. A separately evaluated policy cap addresses this failure on a 36-case diagnostic grid. These results support explicit trade protections and execution-based evaluation, while distinguishing declared safety from a general guarantee.

View source

Similar papers

Open access Oct 2026

CAST: A Compiler-Based Framework for Systematically Testing LLM Compositional Safety

This paper proposes CAST, a systematic testing framework designed to evaluate the compositional safety of LLMs in the domain of malicious code, and demonstrates that CAST systematically exposes severe safety violations in strongly aligned models that resist conventional red-teaming.

Lu Yan, Zhuo Zhang, Xiang-Zhe Xu et al. · 0 citations
#artificial intelligence Preprint Sep 2026

SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics

SemVerBench is introduced, the first benchmark of LLM version-constraint resolution semantics across three ecosystems (npm, PEP 440, Cargo), and six frontier models are evaluated: Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar).

Qi-Bai Chen, Ze-Ming Liu · 2 citations

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

SWE-Flux is introduced, a repository-level benchmark for dynamic execution reasoning containing 480 execution-grounded instances across 12 real Python repositories, with gold answers automatically harvested from instrumented test executions rather than written manually or judged by LLMs.

Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband et al. · 1 citation
Review Aug 2026

PonyEval: Evaluating LLM-Based Program Repair for Capability-Safe and Actor-Oriented Pony Software

This work introduces PonyEval, a SWE-bench-style benchmark of 291 real GitHub issue-pull-request pairs from 15 Pony repositories that defines a matched evaluation with mini-SWE-agent 2.4.6 for GPT-5.6-sol, DeepSeek-V4-Pro, GLM-5.2, MiniMax-M3, and Kimi-K3, followed by strict patch application, compilation, and hidden-t...

Bang Xie, Hao Liu, Zhen-Yu Shi et al. · 0 citations
Book Open access Oct 2026

Formal Runtime Verification for Tool-Using LLM Agents: An Offline Same-Benchmark Study on AgentDojo and STAC

Results show that formal temporal monitoring adds value exactly when the trace exposes trustworthy history, and quantify how far current benchmarks are from that point and propose a twelve-field enforcement-ready trace schema.

Nikolaos Kekatos, Stylianos Basagiannis, Marinelio Chintri et al. · 2 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.