← AI Engineering
Classical Machine Learning · Time Series Forecasting

NASDAQ 100 Forecasting: Statistics vs Machine Learning

Three forecasting philosophies, one 20-month horizon: a classical statistical model, a modern decomposition-based model, and a machine learning ensemble — with an honest look at why two very different models failed in exactly the same way.

View on GitHub Data source (Yahoo Finance)
Python statsmodels Prophet scikit-learn

The problem

Can three fundamentally different forecasting approaches predict the NASDAQ 100 index roughly 20 months into the future? 12.5 years of daily closing prices (2014–2026) were used to train and test ARIMA, Prophet, and a Random Forest fed with time-based features.

Line chart of NASDAQ 100 daily closing price from 2014 to 2026, showing a strong long-term uptrend from around 3,500 to nearly 30,000
NASDAQ 100 daily close, 2014–2026 — a more than eightfold increase, with a clear long-term uptrend.

Key results

5.52%
Prophet's forecast error (MAPE)
3x
Better than the other two models
415
Days forecast ahead
2
Models that failed identically

Why time series break the usual rules

Unlike every other project in this series, row order matters here — shuffling the data would destroy the signal being modeled. The train/test split was drawn at a fixed calendar date (everything before January 2025 for training, everything after for testing) rather than the random splits used elsewhere, and features had to be built by looking strictly backward in time.

An Augmented Dickey-Fuller test confirmed the raw price series was non-stationary (p-value = 0.997) — expected, given the visible long-term uptrend. First-order differencing resolved this (p-value ≈ 0.0000), while also revealing a secondary pattern: volatility clearly increased in the more recent portion of the data, a limitation acknowledged but not corrected for.

Three models, three philosophies

ModelMAERMSEMAPE
Prophet1,2571,6315.52%
Random Forest (recursive)3,7194,56514.24%
ARIMA(5,1,0)3,7954,64214.53%

Prophet won decisively — not marginally, but by roughly a factor of three across every metric.

Prophet's decomposed forecast components: an upward trend with widening uncertainty bands over time, a weekly seasonality pattern, and a yearly seasonality pattern with peaks around mid-year
Prophet's trend component projects growth beyond the training range, with uncertainty widening the further out it forecasts — an honest signal of growing uncertainty, not overconfidence.

Why the two "losers" failed identically

The most interesting finding wasn't that Prophet won — it was how ARIMA and Random Forest failed. Both flattened into a near-constant forecast, hovering near the last training value, while the actual index climbed to new highs throughout the test period.

Line chart comparing ARIMA, Prophet, and Random Forest forecasts against the actual NASDAQ 100 price. ARIMA and Random Forest both flatten into near-constant lines, while Prophet tracks the rising trend and the actual price fluctuates around it
ARIMA and Random Forest both flatten into a near-constant forecast; only Prophet tracks the underlying trend.
A linear statistical model (ARIMA) and a non-linear tree ensemble (Random Forest) — built on entirely different principles — produced the same practical failure, for two different structural reasons.

ARIMA's autoregressive coefficients were small and mostly statistically insignificant, so its multi-step forecast mathematically converged toward a flat line. Random Forest's failure had a different root cause: tree-based models learn decision thresholds from the range of values seen in training, and once the recursive forecast pushed prices beyond the historical maximum, the model had no learned basis for extrapolating further — trees are strong interpolators but structurally poor extrapolators.

Prophet's edge wasn't superior sophistication — it was the only one of the three with an explicit, projectable trend component, capable of extending beyond values seen during training.

Tools used

Python yfinance statsmodels Prophet scikit-learn Matplotlib