Backtesting & Validation

What Does Statistical Significance Mean in a Backtest?

Why a p-value of 0.046 is weaker than it sounds.

A backtest that passes a significance test feels like proof. It is not. A p-value answers one narrow question, and most of the things investors care about, such as whether an edge is large, stable and likely to continue, sit outside that question.

Published October 14, 2026 · Insight 007

The narrow question

What a p-value actually measures

A significance test starts by assuming the idea is worthless. That assumption is called the null hypothesis. For a stock-ranking signal, the null says the signal has no connection to future returns, so any pattern in the backtest is luck.

The p-value is the probability of seeing a result at least as strong as the one observed if that null were true. A p-value of 0.05 means that a worthless signal would produce a result this good about 5% of the time.

That is all it means. It is not the probability that the strategy works, not the probability that it will keep working, and not a measure of how much money it would make.

It is

A surprise score

How unusual the observed result would be if the signal had no real information.

It is not

The chance it works

A p-value of 0.05 does not mean a 95% chance the strategy is real.

It is not

A size or stability measure

A tiny, fragile effect can be statistically significant with enough data.

A real example

Trend Quality passed the test. Here is what that did and did not show.

Market Data Hunter's Trend Quality investigation ranked S&P 500 stocks by how persistently they traded above their 200-day average, then measured whether higher-ranked stocks earned more over the next 20 trading days. The test was fixed before it was run: a one-sided test at the 5% level.

To build the null, the signal was shuffled randomly among stocks on each date 100,000 times, breaking any real link between rank and return. The observed result was then compared with those 100,000 random versions.

Measure	Value	What it tells you
Mean rank correlation	+0.0098	Very small effect
One-sided p-value	0.046	Just under 0.05
Two-sided p-value	0.092	Not under 0.05
2015–2018 correlation	−0.003	No effect
2019–2021 correlation	−0.021	Wrong direction
2022–2026 correlation	+0.043	Where the effect lived

Trend Quality, January 2015 to May 2026, 36,688 stock-date observations on 144 rebalance dates. Rank correlation runs from −1 to +1; zero means no relationship.

The result cleared the bar by a small margin, the effect was tiny, and it was absent or reversed for most of the sample. All three facts are true at once, and the p-value only reports the first.

Five reasons

Why 0.046 is weaker than it sounds

01 · Near the line

0.046 and 0.054 are nearly the same evidence

The 0.05 cutoff is a convention, not a cliff. A result just below it is not meaningfully stronger than one just above it, and small changes in data or method can move it across.

02 · Direction choice

One-sided tests halve the p-value

Testing only for a positive effect is legitimate when the direction is set in advance, as it was here. But the same data gives a two-sided p-value of 0.092, which would not pass.

03 · Effect size

Significant is not the same as large

A mean rank correlation of +0.0098 means the signal barely sorted stocks at all. With tens of thousands of observations, even a very small effect can clear a significance test.

04 · Stability

An average can hide a regime

The positive result came from 2022 onward. The test asks whether the full-sample average is unusual, not whether the effect showed up consistently through time.

05 · Multiple attempts

It was the third idea tested

Trend Quality followed two earlier hypotheses that failed. Each extra idea tested raises the chance that one passes by luck, which a single p-value does not account for.

The forking paths problem

Every extra test is another lottery ticket

If you test one worthless idea at the 5% level, you have a 5% chance of a false positive. Test three unrelated worthless ideas and the chance that at least one passes rises to about 14%. Test twenty and it is about 64%.

This is why the number of things tried matters as much as the result that survived. Market Data Hunter publishes the failed hypotheses, the 100-point score and the 63-day momentum ranking, so readers can see the full search that came before Trend Quality.

Formal corrections exist for this. The Trend Quality portfolio tests used a family adjustment across the three portfolio sizes. No adjustment can account for ideas that were tried and never written down, which is why a public record of failures matters.

A p-value is only as honest as the count of attempts behind it.

The portfolio twist

An extreme p-value can sit on a fragile result

The Trend Quality portfolios produced an even more striking number. Their average excess returns over SPY were larger than in all 100,000 randomized versions, giving the smallest p-value the test could report, about 0.00001.

Yet removing the ten best of the 144 periods turned the average excess return slightly negative for every portfolio size. Both facts are true. Random stock picks rarely produce a run of very large wins, so the test flags the average as unusual. It does not say whether those wins will recur.

Statistical significance tells you a result is unlikely to be pure chance under one specific null. It does not tell you the result is robust, repeatable or worth trading.

Checklist

Questions to ask about any significant backtest

Was the test, threshold and direction chosen before the results were seen?

How large is the effect, not just how significant?

Would the conclusion change with a two-sided test?

Does the effect appear in most time periods or only one?

What happens to the result if the best few periods are removed?

How many other ideas, settings or versions were tried first?

Is the p-value close to the threshold or far from it?

Has the result been checked on data that was not used to develop it?

The bottom line

Significance is a gate, not a verdict.

Passing a pre-registered significance test is a reasonable reason to keep investigating an idea. That is how Trend Quality was treated: its rules were frozen and it moved to prospective tracking, where only new data collected after the freeze can count.

Until that new evidence arrives, the honest summary of a p-value of 0.046 is short: the result is somewhat unlikely to be pure luck, the effect is small, and its stability is unproven.

Keep reading