Predicting Financial Bubbles with Imbalanced Data

Abstract

This paper explores the prediction of financial bubbles within the S&P 500 using a machine learning framework as well as econometric bubble detection tests. The identification of asset bubbles is critical for maintaining global financial stability, as the delayed recognition of market exuberance can lead to severe systemic collapses and prolonged economic downturns. Utilising daily price-dividend ratio data from the State Street SPDR Portfolio S&P 500 High Dividend ETF (SPYD ETF), the study compares the performance of various machine learning models, namely Support Vector Machines (SVM), Random Forest, and XGBoost. By transitioning from traditional low-frequency analysis to a high-frequency approach, this research seeks to capture the rapid, nonlinear dynamics that often precede a price correction. To address the rarity of bubble events, the Synthetic Minority Over-sampling Technique (SMOTE) is employed to correct class imbalance which is often overlooked in literature. The resulting models achieve good predictive accuracy scores though relatively low F1 scores. The results suggest that even with the SMOTE synthetic data, the noise in daily data makes it difficult to generate F1 scores comparable to bubble prediction using lower frequency data. The tradeoff is between faster but noisier bubble detection and slower but more accurate bubble detection.