7 Common Mistakes When Splitting Data for Time Series Forecasting

A model that looks sharp in validation and then produces useless forecasts the week it ships is a familiar story, and the algorithm is rarely to blame. The split usually is. Time series rows are ordered: each one carries a timestamp that decides what a model is allowed to know at prediction time. Treat them like interchangeable spreadsheet rows and you get metrics that flatter you and forecasts that fail.

Why a Random Split Leaks the Future

Imagine you forecast daily demand for one store and call train_test_split(df, test_size=0.2, shuffle=True). Most of your test rows now sit between training rows. Monday's sales in the test set have Saturday and Tuesday in the training set, so the model does not have to forecast anything. It interpolates. Scores climb because the task quietly changed from extrapolation into smoothing.

The leak is often subtler than that. A test row from next week can land inside a training window of rolling means, price changes and promotions that would never have been available at that point in real time. The validation number measures how well the model fills gaps in history, not how well it predicts a period nobody has seen.

If your test fold can be interpolated from the rows around it, you are measuring smoothing accuracy, not forecasting skill.

The Seven Mistakes

1. Shuffling rows before splitting. Any cut that mixes dates across the boundary is invalid for forecasting. Even shuffle=False with a single split is only a first step: it still has to respect your horizon and any reporting lag.
2. Splitting at the wrong granularity. With many series at once, a row-level split puts the same timestamp in train and test. Cut on dates, not rows. If deployment involves entirely new stores or sensors, hold out whole entities as a separate experiment.
3. Fitting transforms on the full dataset. Scalers, imputers, target encoders, PCA and feature selection all learn from data. A scaler fitted on the whole file has already seen the test period's mean and variance. Fit on the training fold, then apply forward.
4. Mismatching the validation horizon with the deployment horizon. A one-step-ahead backtest says nothing about a 30-day-ahead model. Make the test window at least as long as the horizon you will actually use, and build each prediction from information available at that forecast origin.
5. Letting lag and rolling features see forward. A seven-day rolling mean is safe only when computed per series, after sorting by date, and never with center=True. Backfilling, interpolating across the cut, or resampling before splitting can all push future values into the past.
6. Tuning against a single split. One cut gives you one number, and after enough tweaks that number becomes fiction. Use several folds for model selection and keep the newest block of data as a holdout you touch once.
7. Ignoring reporting delays and gaps. Payments settle in a few days, sensors upload hourly, monthly figures get revised. If your features assume instant availability, the backtest will be optimistic. Insert an embargo between train and test that matches the real lag.

Rolling-Origin Validation, Step by Step

The fix is to simulate the way the model will be used. Pick a forecast origin, train on data up to that point, predict the next h steps, then move the origin forward and repeat. Every fold is a genuine forecast, so the scores mean something.

from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5, test_size=30, gap=7)
for train_idx, test_idx in tscv.split(X):
    model.fit(X.iloc[train_idx], y.iloc[train_idx])
    preds = model.predict(X.iloc[test_idx])

The gap matters. If your features include seven-day lags, the last seven rows before the test window describe the test period itself, and a small gap keeps them out. Writing the loop by hand is just as easy and gives you more control over fold spacing:

def rolling_origin(n, horizon=30, min_train=365, step=None):
    step = step or horizon
    t = min_train
    while t + horizon <= n:
        yield range(t), range(t, t + horizon)
        t += step

Expanding versus sliding windows

An expanding window keeps all history and suits stable series where more data helps the model. A sliding window keeps only the last N periods and suits regimes that shift, such as demand after a pricing change. Run both on a few folds. If the sliding window wins consistently, your model is relying on recent behaviour, and that is worth knowing before you ship it.

Reading the Results

Report every fold, not just the average, and look at the spread. A model with a mean error of 4% and folds ranging from 2% to 11% is a different animal from one that stays between 4% and 5%. Plot error by forecast step as well: one-step error that looks fine while step 30 drifts badly tells you the horizon is doing the damage.

Always compare against a naive baseline. For daily demand that is usually yesterday's value or the same day last week. A model that cannot beat last week's number is not adding much, no matter how good the validation chart looks. Beating the baseline by a small margin on every fold is a stronger signal than a large gain on one.

A Checklist Before You Trust a Backtest

• Sort by time and confirm no timestamp appears in both train and test.
• Cut on dates, not rows, and keep related series together when that matches deployment.
• Fit scalers, imputers, encoders and target transforms inside each fold.
• Build lags and rolling features per series, from past values only.
• Set the test window to at least your forecast horizon, plus an embargo for reporting lag.
• Score several origins; judge the spread as well as the mean.
• Keep the most recent data as a one-time holdout for the final check.

None of this is glamorous. Rolling-origin validation is slower to run and your scores will look worse than the ones a shuffled split produced. That is the point: those low numbers are the honest ones, and they are the only version of reality you can plan capacity, inventory or staffing around.

Photo: RuslanSikunov / Pixabay