Backtesting & Validation

Why a Few Periods Can Drive a Whole Backtest

When the average depends on ten good months.

A strategy that beats the market by 1% per period on average sounds like a steady edge. It might be. Or it might be a mostly ordinary strategy that happened to catch a few enormous wins. The average alone cannot tell you which.

Published October 15, 2026 · Insight 009

The problem

Averages are pulled by extremes

An arithmetic average treats every period equally, so one period with a +40% excess return counts as much as forty periods of +1%. Stock returns have fat tails: big moves happen more often than a bell curve suggests. Small portfolios of three or five stocks have even fatter tails, because one stock can dominate the result.

That means a backtest average can be mostly a statement about a few unusual periods. If those periods were luck, or tied to one market regime, the average will not repeat.

Signal 01

Mean far above median

If the typical period is close to zero but the average is strongly positive, a few large wins are carrying the result.

Signal 02

Gains vanish without the best periods

Removing a small number of top periods should dent a real edge, not erase it.

Signal 03

Best periods cluster in time

If the biggest wins all come from one stretch of years, the edge may belong to that regime, not to the strategy.

Two real examples

Same tests, different stories

Market Data Hunter ran two stock-ranking hypotheses on the S&P 500 from 2015 to 2026, each with 144 holding periods of 20 trading days. Both Top 3 portfolios beat SPY on average before costs. The concentration checks separate them.

Check	Trend Quality	63-day momentum
Average (all 144 periods)	+1.34%	+0.83%
Median period	+1.18%	+0.00%
Beat SPY	60% of periods	50% of periods
Without best 1	+1.11%	+0.60%
Without best 5	+0.53%	−0.08%
Without best 10	−0.05%	−0.71%

Top 3 portfolios, average excess return over SPY per 20-day period, before trading costs. 'Without best N' drops the N strongest periods out of 144.

Momentum's typical period was essentially break-even, and five periods out of 144 held its entire average. Trend Quality was positive in the typical period, but its average still did not survive losing its ten best.

Picture it

Watch the average shrink

Trend Quality Top 3: average excess return as the best periods are removed

All 144 periods

+1.34%

Without best 1

+1.11%

Without best 3

+0.81%

Without best 5

+0.53%

Without best 10

-0.05%

Average return versus SPY per 20-day period, before trading costs.

Removing only the best periods is a deliberately harsh, one-sided test. A gentler check trims both ends equally. Trimming the five best and five worst periods left Trend Quality's Top 3 average at +1.28%, close to the full average, which says the distribution is not purely a tail story. Both views are worth reporting.

Where the wins came from

Concentration in time

For Trend Quality, 29 of the 30 best period results across the Top 3, 5 and 10 portfolios came from 2022 onward. For 63-day momentum, the 2022–2026 periods contributed 127% of the Top 3 total; 2015–2021 was negative on balance.

Time concentration matters as much as size concentration. A strategy that worked in one regime can stop working when the regime changes, and nothing in the full-sample average warns you.

A robust edge should show up in most periods and most years, not just in the total.

Checklist

Concentration checks to run on any backtest

Compare the mean period return with the median period return.

Recompute the average without the best 1, 5 and 10 periods.

Report a symmetric trimmed mean alongside the one-sided test.

Count the share of periods that beat the benchmark.

Split the sample into fixed time blocks chosen in advance.

Check whether the best periods cluster in one stretch of years.

Repeat the checks after trading costs.

Ask whether the result would survive one bad year in the future.

See the full results in the Trend Quality and 63-day momentum investigations.

Keep reading

Why a Few Periods Can Drive a Whole Backtest

Averages are pulled by extremes

Mean far above median

Gains vanish without the best periods

Best periods cluster in time

Same tests, different stories

Watch the average shrink

Concentration in time

Concentration checks to run on any backtest

Related reading

What Does Statistical Significance Mean in a Backtest?

What Is Backtest Overfitting?

63-Day Momentum Ranking

What 155 Years of S&P 500 Returns Say About Holding Periods