Slimmed down AI models are best at forecasting market trends

Football, a plain-speaking Scot once said, is a simple game made complicated by people who should know better.

Can the same be said for AI models forecasting stock market returns? Our research suggests it might.

We developed an AI model to predict by how much the returns on S&P 500 equities would exceed the ‘risk-free’ rate for each year since 2020. It out-performed a model  which predicts future outcomes based on a simple historical average using the same data. Our model reduces forecasting error by around 30 per cent. We saw the same improvement in predicting real stock returns.

This forecasting accuracy is even better over the longer term, with errors for five year returns halving. That’s because short-term market ‘noise’ fades – allowing the fundamental metrics embraced by the model to come to the fore.

Savvy readers on the Bayes website might be pleased to hear that the model is in bull mode in its predictions for the next 12 months. If there is an AI bubble, it’s not going to burst this year, the results suggest. The S&P 500, the model predicts, will deliver returns around 6 per cent above the risk-free rate of return.

Of course, readers will also be aware that no model predicts precise outcomes. What matters is whether it gets the direction of travel right – and gets reasonably close in the scale of that movement (up or down) by reflecting market fundamentals, but not random “noise”. Across rolling one-year forecasts, the model has called market direction correctly every time.

The results produced by our relatively simple model challenge the conventional belief that more sophisticated, automated machine learning methods are the most accurate forecasters of market movements. Indeed, the outcome is a small temple to both (relative) simplicity and the enduring value of human oversight. That particularly applies when working with limited data – which is inevitable in long term financial forecasting.

At the dawn of the AI revolution, investors have come to venerate complex machine-learning models. These systems often gorge on a king’s banquet of estimators and variables simultaneously. So-called “black boxes” produce predictions without clearly revealing how those conclusions were formed.

Even their designers cannot easily tell what is driving the final forecast. The machine effectively makes the decision automatically, with only minimal human intervention.

Our model is deliberately transparent. This “glass-house” form of machine learning leaves its bloated rivals in the shade. Our model survived on a low calorie diet of just one estimator and a few economically meaningful and carefully constructed predictors – such as interest rates and valuation ratios. The humans can see exactly which ones it is digesting. Instead of feeding an algorithm every available variable or letting a crowd of competing models vote behind the scenes, it relies on a single, clearly specified forecasting approach.

The real game changer, however, is double-benchmarking. A largely-ignored approach, double benchmarking compares both stock returns and the variables used to predict them against two relevant benchmarks, such as inflation or the ‘risk-free’ rate of return. While it may sound modest, its impact is anything but. Focusing on the right variables matters far more than piling on layers of modelling complexity. Combined with the “glass-house” approach to forecasting, the result is a forecast that cuts cleanly through the noise.

We wanted to provide both institutional investors and independent financial advisers with a simple and effective model to forecast long-term stock returns. We think this model achieves that – which could be good news for anyone with a pension or other long-term savings.