Measuring the quality and efficiency of AI-generated codes for financial markets prediction with LSTM

Abstract

Generative AI coding assistants are increasingly used to write machine-learning code, yet their ability to produce reliable LSTM implementations for financial prediction remains underexplored. This study evaluates the LSTM code generated by seven assistants ChatGPT 4.5, GitHub Copilot, Deepseek 3, Perplexity, Gemini 2.0 Pro, Claude 3.7 Sonnet, and Meta’s Llama from a single standardized prompt, on three indices (Nikkei 225, S&P 500, STOXX Europe 600). Each assistant’s generated script was re-executed over independent runs; accuracy (MAE, MSE, RMSE, R2, execution time) is reported as mean ± standard deviation on the original price scale, complemented by a static code-quality analysis (Pylint, Radon, SonarQube, Pytest, Bandit). The assistants converge on nearly identical LSTM architectures, so performance differences arise mainly from data-handling and code-correctness defects: Meta’s Llama near-zero errors are an artifact of normalized-scale metrics combined with a shuffled train/test split (data leakage), and once corrected its accuracy is among the weakest; Gemini 2.0 Pro, once its predictions are evaluated consistently on the price scale, is among the most accurate assistants. Differences are validated with Diebold–Mariano and Wilcoxon tests. AI-generated forecasting code can be accurate but is not uniformly trustworthy: its generated preprocessing and evaluation code must be audited before use.