Backtesting & Validation
What Is In-Sample vs. Out-of-Sample Testing?
Why separating development from evaluation matters.
In-sample data helps develop a strategy. Out-of-sample data asks how the resulting strategy behaves on observations that did not participate in that development.
Published · Insight 003
A backtest becomes easier to fool when the same historical information is used both to design a strategy and to judge whether that strategy works.
Researchers naturally learn from data. They compare signals, adjust parameters, remove weak ideas, and refine rules. Those decisions are part of development. But once the historical record has helped make those decisions, performance on that same record is no longer a fully independent test.
In-sample and out-of-sample testing attempt to separate those jobs. One set of observations can help build the hypothesis. Another can provide a more demanding evaluation.
The basic split
Development on one period. Evaluation on another.
Imagine that a researcher has ten years of historical data. Instead of repeatedly developing and evaluating a strategy on all ten years, the researcher divides the history before the final evaluation.
Step 01
In-sample period
Earlier observations are used to explore the idea, choose rules, study parameters, and develop the strategy.
Step 02
Freeze the version
The strategy that will be evaluated is specified before its out-of-sample performance is examined.
Step 03
Out-of-sample period
The chosen rules are applied to withheld observations that did not help select that final version.
The split does not make a strategy valid. It creates a more informative question: did the relationship survive data that played less of a role in choosing it?
In-sample
Where the strategy learns.
In-sample data is the portion of the historical record used during development. A researcher might use it to compare signals, choose lookback periods, study filters, select portfolio rules, or reject ideas that appear unpromising.
Strong in-sample performance can be useful. It can show that an idea deserves further investigation. But because the data helped shape the strategy, that performance must be interpreted in the context of the development process.
Out-of-sample
Where the chosen strategy is challenged.
Out-of-sample data is withheld from at least part of the development process. The final strategy is then evaluated on those observations without first choosing its rules to fit their outcomes.
If performance survives, the evidence becomes more interesting. If it deteriorates sharply, that may reveal that the original result depended heavily on the development sample.
The important catch
Out-of-sample data can become in-sample data.
Suppose a strategy disappoints during its first out-of-sample test. The researcher studies the result, changes a filter, and tests again on the same withheld period. The new version now contains information learned from that period.
Calling the period "out-of-sample" does not restore its independence. Once its results influence the strategy, those observations have participated in development.
01
Withhold
Reserve observations that will not be used to choose the final strategy.
02
Evaluate
Run the specified strategy on the withheld observations.
03
Learn
The result reveals information about how the strategy behaved outside development.
04
Do not relabel
If that information changes the strategy, the reused period is no longer a pristine independent test.
What the split helps with
A useful defense against overfitting, not a cure.
Separating development and evaluation can make it harder for a strategy to benefit from accidental patterns that existed only in the data used to build it. But a single out-of-sample test does not eliminate every source of research error.
Overfitting
A clean split reduces the opportunity to optimize directly against every observation in the historical record.
Selection bias
The result can still be affected by how strategies, samples, universes, or reported tests were selected.
Market regimes
One withheld period may represent conditions that differ substantially from other periods.
Repeated testing
Trying many ideas against the same out-of-sample period can gradually turn it into another development dataset.
Out-of-sample performance is additional evidence. It is not a guarantee of future performance.
Beyond one split
What is walk-forward testing?
A single development period followed by one withheld period is not the only possible design. Walk-forward methods use a sequence of time-ordered development and evaluation windows.
For example, rules may be estimated using earlier observations and evaluated on the next period. Time then moves forward, the development window is updated according to a predefined procedure, and another later period is evaluated.
Why time order matters
The future should not teach the past.
Financial research has a natural chronology. Information available later should not influence a decision that the backtest claims could have been made earlier.
Whatever split or walk-forward design is used, the research should preserve that ordering and prevent future information from leaking into earlier decisions.
Historical vs. prospective
Out-of-sample does not automatically mean future data.
An important distinction is often lost in casual discussions of backtesting. An out-of-sample period can still be entirely historical. The observations may have existed before the research project began even if they were intentionally withheld from a particular development step.
Historical development
In-sample
Existing data participates directly in developing or selecting the strategy.
Historical evaluation
Out-of-sample
Existing observations are withheld from specified development decisions and used for additional evaluation.
New evidence
Prospective
Rules are frozen before the relevant future observations exist, then those observations are recorded as they arrive.
Historical out-of-sample testing can be valuable. Prospective testing answers a different question because the future outcome was unavailable when the methodology was frozen.
The research discipline
Development is not validation.
There is nothing inherently wrong with developing a strategy using historical data. Development is where ideas are refined, weak assumptions are challenged, and a precise methodology is created.
The problem is treating evidence that helped create the strategy as though it were fully independent confirmation of that strategy.
01
Develop
Use research data to explore the question and define a candidate methodology.
02
Evaluate
Challenge the candidate using evidence separated from specified development decisions.
03
Freeze
Write down the final methodology before beginning the prospective phase.
04
Track
Record qualifying new observations without retrospectively changing the test.
A real Market Data Hunter investigation
Trend Quality: historical evidence before prospective evidence
Market Data Hunter's Trend Quality investigation examined whether persistent strength above a long-term trend contained information about subsequent returns.
36,688
Historical observations
144
Evaluation dates
100,000
Permutations
0
Pristine prospective observations
Historical results justified another test
Historically, the top-ranked portfolios produced positive average 20-session excess returns over SPY:
Top 3
+1.34%
Top 5
+1.21%
Top 10
+0.94%
The relationship was not equally strong across historical periods. The historical investigation therefore remains evidence about the past, not proof that the relationship will persist.
Reading the evidence
Ask what each dataset was allowed to influence.
The words in-sample and out-of-sample are useful shorthand, but the labels alone do not establish research quality. A better evaluation asks how information actually moved through the research process.
What chose the rules?
Identify the observations and results that influenced the final methodology.
What was withheld?
Determine which evidence was genuinely separated from the decisions being evaluated.
Was the test reused?
Check whether disappointing evaluation results led to revisions followed by repeated testing on the same data.
What came later?
Distinguish historical evaluation from observations that did not yet exist when the rules were frozen.
A careful split cannot guarantee that a market relationship is durable. What it can do is make it more difficult for the same observations to both shape the hypothesis and provide all of the evidence supporting it.
Better separation makes hindsight more expensive.
One question worth remembering
What did this data know before the test began?
When you see an out-of-sample result, do not stop at the label. Ask whether those observations had already influenced the strategy, directly or indirectly.
Did this evidence help choose the rules, or did it only test rules that were already specified?
That distinction will not tell you what markets do next. It can tell you something important about how much independence the evidence actually has.
Good validation does not remove uncertainty. It makes the source of the evidence easier to see.