A Machine Learning Framework for S&P 500 Directional Classification and Kelly-Optimal Position Sizing
TL;DR
The findings suggest that while supervised learning offers a modest predictive signal, the Kelly Criterion is effective primarily as a risk-management tool rather than a return-enhancing strategy.
Abstract
This study investigates whether supervised learning models can predict short-term directional movements in the Standard & Poor's 500 (S&P 500) and whether these predictions can be converted into a viable trading strategy using the Kelly Criterion. Using 26 years of daily Open, High, Low, Close, Volume (OHLCV) data (2000–2026), 18 features — including lagged returns, technical indicators, volatility and volume measures, and calendar effects — were engineered, and three classifiers — Logistic Regression, Random Forest, and eXtreme Gradient Boosting (XGBoost) — were compared. Model outputs were transformed into optimal position sizes via the Kelly Criterion and evaluated on cumulative return, Sharpe ratio, and maximum drawdown, benchmarked against a full-investment variant and Buy-and-Hold. Results show that all models achieved approximately 40% test accuracy — above the 33% random baseline — with Logistic Regression performing best on accuracy (40.69%) and Area Under the Curve (AUC) (0.589). Kelly-based strategies cut maximum drawdown by more than half relative to full investment, to between 10% and 14%, though Buy-and-Hold still achieved a higher Sharpe ratio. These findings suggest that while supervised learning offers a modest predictive signal, the Kelly Criterion is effective primarily as a risk-management tool rather than a return-enhancing strategy.