Evaluating Deep Learning Architectures for Multivariate Time Series Forecasting: Accuracy, Efficiency, and the Learning Efficiency Score

Abstract

Deep neural network architectures are extensively utilized in the processing of complex multivariate datasets, which may have nonlinear interdependencies, sequential dependencies, and high dimensionalities. Examples of commonly adopted models include MLPs, CNNs, RNNs, LSTMs, GRUs, and Transformer models. Although previous research has mainly focused on the prediction accuracy, which is usually evaluated based on metrics like RMSE and MAE, they often neglect learning efficiency and the volume of data needed for achieving the best possible performance. In this study, we propose the Learning Efficiency Score (LES), a new metric that considers both convergence time and prediction accuracy when assessing architectures. The experiments show that extremely sophisticated architectures do not always lead to a better efficiency score; in some cases, it would be better to adopt an intermediate or simple architecture to strike a balance between accuracy and computational costs.