Backtests: why the past can be fitted

Published 27 September 2026

You have seen the chart. A line climbs steadily for twenty years, shrugs off every crash, and somewhere in the small print it says back-tested. Every number on it can be correct, and it can still tell you almost nothing about the next twenty years. This is what a backtest is, why a beautiful one is cheap to make, where you will run into one, and how to recognise it without doing any maths.

What a backtest is

A backtest takes a rule and runs it over old prices. Hold these shares when this happens, and not when that happens: what would that have made? The answer is a simulation. Nobody earned that money.

You meet them more often than you might think. Take an index that a large index provider started calculating on 23 March 2011. Its own documentation says the history goes back to the end of 1998, and says plainly: "All information presented prior to this date is back-tested. Back-tested performance is not actual performance, but is hypothetical."⁵ More than twelve years of that history were worked out afterwards.

That is not a trick. Running a rule over the past is the obvious first test of any idea, and a rule that would have lost money for decades deserves to be dropped. The trouble lies somewhere else, and the American futures regulator names it in the warning it requires trading advisers to print next to such figures: simulated trading programs "are designed with the benefit of hindsight."⁶

So a backtest is a story about the past, told by someone who already knew how it ended. The rest of this piece is about what that does to the story.

Why a beautiful backtest is cheap

Here is an experiment you can check in your head. We built a made-up stock market on a computer, one with no pattern in it at all: each month's return is a fresh draw, so nothing about the past says anything about the next month. Then we let 1,000 strategies loose on it. Each one decides every month, by the toss of a coin, whether to be in the market or out of it.

After ten years we looked for the best one. It had turned €10,000 into about €28,400, while simply holding the market ended at about €16,400 (Figure 2).^* Nothing about the winner was clever. It was a coin that happened to land well. In the ten years after, it did what a coin does: about 7% a year, while the market in that particular run did about 14%.

One run can be luck, so we repeated the whole thing 400 times, each time on a fresh market, and varied how many strategies we tried. With one strategy, the backtest showed about 3% a year. With ten thousand, the best one showed more than 12%. What that best one did in the following ten years barely moved: around 3% a year, whichever pile it came from (Figure 3). The more you try, the better the best one looks. Its future does not care how hard you looked.

Researchers have shown the same with real data. Three prominent names in finance research built a strategy on New York Stock Exchange shares from 1963 to 1988, then checked it on 1989 to 2015 and found "even stronger results". It earned, in their words, "nearly 6% alpha a year": roughly, 6% a year more than its exposure to the market's ups and downs would explain. Then they revealed the rule. It buys shares whose ticker symbol has an S as its third letter, and bets against those with a U. "This strategy might seem too good to be true. And it is."¹ They found it by trying thousands of letter combinations until one fitted.

Researchers who study this have a name for it: backtest overfitting. You try enough versions of a rule on the same stretch of history, and the best version ends up describing the noise in that history rather than anything that will repeat. Man Group's academic advisory board put it in one line: "Overfitting is when a model describes noise rather than signal."⁸ A team of mathematicians found a respectable-looking seasonal trading rule in a series of prices that was pure chance, and drew the conclusion: "it is easy to demonstrate that any random data contains similar patterns."²

The part you never see is the pile of rejected versions. As the same authors write, "Because financial analysts rarely report the number of configurations tried for a given backtest, investors cannot evaluate the degree of overfitting in most investment claims and analysis."² The honest question about a backtest is therefore not is this calculation right? It usually is. The question is: how many brothers did it have?

More knobs, a closer fit, and the test that catches it

There is a second way to fit the past, and it does not need a thousand strategies. It needs one strategy with a lot of knobs. Every extra ingredient in a model (another indicator, another threshold, another exception for that one bad year) lets it follow the old line a little more closely. The closer the fit, the greater the chance that it is following the noise.

We tested that too. We gave a model the job of guessing, each month, whether a made-up market with no pattern would go up or down. An honest score here is 50%: a coin. With one ingredient, the model guessed about 53% of past months right. With sixty ingredients, it got about 67% right: two months in three, in a market where nothing can be predicted (Figure 4).^* On ten new years it was back to a coin.

There is a good defence against this, and it is worth knowing its name: cross-validation. You cut your history into, say, five pieces. You build the model on four of them and test it on the fifth, which it has not seen. Then you do it again with a different piece held back, until every piece has been the test once. A model that only memorised the noise does well on the four and badly on the fifth. In our experiment, cross-validation put the sixty-ingredient model straight back at 50%, where it belonged (Figure 4). Variations with names like walk-forward testing work on the same idea.

So the problem is solved? Not quite. Cross-validation protects one model. It does not protect you from trying many. We scored 1,000 models by cross-validation and kept the best. Its cross-validated score was about 61%. On ten new years, the same model got 50% (Figure 5).^*

The authors of the ticker-letter strategy say the same about real data: "it is possible that a false strategy can work in the cross-validated sample." The reason is simple. All five pieces come from the same single history. For shares, that history is short: they count "about 55 years of high-quality equity data (or less than 700 monthly observations)", and call it a tiny sample.¹ Every piece you hold back is still the past, and if you have looked at it before, it is not really held back. In their words, "the only true out of sample is the live trading experience."¹

And then there is what no test can see: a kind of market that is not in the data at all. A rule tuned on ten calm years has never met a crisis. The same advisory board puts it plainly: "in finance, the investment environment can change and a strategy that has worked in the past can stop working."⁸ Another research house notes that many smart beta backtests "have only 10 to 15 years of history and cover a very limited set of market environments."⁷ Cross-validation shrinks the risk. It does not remove it.

Where you will meet one

One common place is also a respectable one: the index behind an index fund. Many newer funds follow what the industry calls a smart beta index, one that picks shares by a rule (cheapness, momentum, low swings) instead of simply by size. The index is often designed shortly before the fund is launched, and its history before that date is calculated backwards.

Three researchers looked at 238 of these funds listed in the United States between 2000 and 2018. Before each fund existed, over an average of about 13 years, its index ran ahead of the market by 2.77% a year, after allowing for market risk. After the fund listed, over an average of about 6 years, the same index trailed the market by 0.44% a year, and the funds themselves, after their fees, by 0.87% (Figure 6).³ On €10,000, that is the difference between being about €277 a year ahead on paper and about €87 a year behind in the real fund.

The researchers expected indexes combining several factors to leave more room for fitting than indexes built on a single one, and that is where the drop was largest: for those multi-factor indexes it was 4.91 percentage points a year, against 1.76 for plain value and growth indexes.³ Nor is it only American. In 77 such funds that invest outside the United States, 37 of them in European shares, the index beat its regional market by 3.02% a year on paper and trailed it by 0.15% a year after listing.³

In one case, a major index provider had a value index, released in 1997. After years without any excess return over the market, in December 2014 the provider built an "enhanced" version, and a fund tracking it listed shortly after. The researchers: "Ironically, the enhanced performance only occurred in the backtesting period before the ETF listing." Their conclusion is careful and worth quoting: "investors need to be aware that backtesting is not, on average, an effective mechanism to identify the future outperformance of smart beta ETFs."³

The second place is anywhere a trading system is on offer: a course, a newsletter, a signal service, an app. Evidence comes from a platform where users built and tested trading algorithms. Researchers took 888 of them and compared how each did in its backtest with how it did in the months of simulated trading after it was switched on. The standard backtest scores, such as the Sharpe ratio (return per unit of risk), explained almost none of the differences between them afterwards: less than 2.5%. And "the more backtesting a quant has done for a strategy, the larger the discrepancy between backtest and out-of-sample performance."⁴ That is the coin-toss experiment again, in the wild.

Some regulators, in the United States and in the EU, require a prominent warning with simulated results.^{6, 15} Where no one requires it, you will have to find the word yourself. The research house quoted above puts the risk in one sentence: "backtested results are easily data mined."⁷

Why it convinces us anyway

If backtests are this easy to fit, why do they work so well on us? The psychologist Daniel Kahneman spent a career on that kind of question. In the introduction to Thinking, Fast and Slow he summarises one part of the book in two sentences that could stand above every backtested chart: "We are prone to overestimate how much we understand about the world and to underestimate the role of chance in events. Overconfidence is fed by the illusory certainty of hindsight."¹⁰

The pattern is older than the backtest. In 1971 Kahneman and his colleague Amos Tversky described a researcher who "computes all correlations between three indexes of anxiety and three indexes of dependency" and then "will often report and interpret with great confidence the single significant correlation obtained."¹⁶ Nine tries, one report: the same contest, on a smaller scale.

Kahneman called one of these errors the illusion of validity. As a young army psychologist he rated officer candidates, knew from the statistics that his ratings predicted little, and felt sure of each one anyway. In a version of the book he adapted for The New York Times Magazine, he explains why: "Confidence is a feeling, one determined mostly by the coherence of the story and by the ease with which it comes to mind, even when the evidence for the story is sparse and unreliable."¹¹ A smooth line rising for twenty years is about the most coherent story a chart can tell.

He also tells what happened when he looked at the results of some 25 wealth advisers at a firm, over eight consecutive years. Their bonuses depended on those results. He checked whether the same advisers came out on top from one year to the next: "the average of the 28 correlations was .01. In other words, zero." The message to the firm's directors was that the firm "was rewarding luck as if it were skill."¹¹ A backtest that picks the best of many versions does the same, only on paper.

Then there is hindsight. In a classic experiment from 1975, people read about a past event and rated how likely each possible outcome had been. Some were told an outcome; others were not. Those who were told an outcome rated it as more likely, by 10.8 percentage points on average, and they were "largely unaware of the effect that outcome knowledge had on their perceptions."¹⁴ A backtest is outcome knowledge written down as a rule. Whoever designs one today already knows how 2008 ended.

Last, the winner's next year. Kahneman tells of a flight instructor who was sure that praising a good manoeuvre made cadets do worse the next time. It was not the praise. Using coins tossed at a target, Kahneman showed the class that "those who had done best the first time had mostly deteriorated on their second try, and vice versa."¹³ The best of a thousand backtests is the best first try. Its second try is the part you pay for.

What a backtest does tell you

None of this means that every backtest is worthless, or that every strategy is luck. Two researchers who built a method to discount backtests for the number of tries found that the strongest results are discounted only moderately: "The strategies with very high Sharpe ratios are probably true discoveries."⁹ Some patterns are real. The problem is telling them apart from the ones that only look real, and a single chart cannot do that for you.

A backtest is also better at some things than others. In the study of 888 algorithms, how much a strategy swung up and down in its backtest explained 67% of the differences in how much it swung afterwards. Its worst fall in the backtest explained 34% of the differences in its worst fall later. Its return explained close to nothing, and pointed weakly the wrong way (Figure 7).⁴

So read a backtest as a description of character, not as a forecast of results. It can tell you that a rule is bumpy, or how far it once fell. It is poor at telling you how much it will make.

A good reason helps too. A rule with an economic story behind it (why should cheap shares pay more?) is less likely to be pure noise than a rule about ticker letters. But stories can be fitted as well: "Theory is flexible, so theory alone is not enough, plus sometimes theory is also overfitted."⁸ A reason is needed. It is not proof.

How to recognise one

You do not need to run any numbers. Six questions do most of the work.

1. Does it say back-tested, hypothetical or simulated anywhere, and where does the real part begin? Look for a date. Everything before it was calculated afterwards.⁵
2. How long is the real part next to the simulated part? Thirteen years on paper and six in real life is the average for the funds above.³ In the index example at the top, more than twelve years of the history were simulated.⁵
3. Is it too smooth? Researchers looked at 23,740 ten-year stretches of real US funds between 1979 and 2018. A lead of more than 4% a year with every single year a win: "the answer is never. Not once."⁷ The same research house notes that most smart beta backtests ignore trading costs, which real funds cannot.⁷
4. Is it an "enhanced", "improved" or "2.0" version of something that disappointed? Then the improvement was, by definition, made with hindsight.³
5. How many versions were tried? Ask. You will rarely get an answer, and that is an answer too.²
6. Does the chart start at a convenient moment? The authors of the ticker-letter strategy give the example of a model that works if the sample begins in 1970 and not if it begins in 1960; then, they write, the model does not work.¹

And then the question that ties them together, again from Kahneman. He once asked his own team, which was writing a school curriculum, how long they would need. The estimates were "narrowly centered around two years". Then he asked the team's curriculum expert how long similar teams had taken. About 40 percent had never finished, he said, and he could not think of any that finished in less than seven years. The book, Kahneman writes, "was completed eight years later."¹²

Kahneman and Tversky later called the first way of thinking the inside view, and the second the outside view: forget the details of your own case and look at how cases like it turned out. His case for it is modest: "if the reference class is properly chosen, the outside view will give an indication of where the ballpark is."¹² For a backtest, that is one question. Not how good is this one? but how did backtests like this one do once they were real? For the smart beta funds in that study, you have seen the answer.

Sources

1. Rob Arnott, Campbell R. Harvey and Harry Markowitz, "A Backtesting Protocol in the Era of Machine Learning", The Journal of Financial Data Science, Winter 2019, pp. 64–74 (duke.edu). Arnott founded Research Affiliates, where Harvey is a partner (source 7 is by the same firm). Our source for: the ticker-letter strategy, 1963–1988 and 1989–2015, "even stronger results", "nearly 6% alpha a year" (p. 65); cross-validation and "a false strategy can work in the cross-validated sample", "about 55 years of high-quality equity data" (p. 66); "the only true out of sample is the live trading experience" and the 1970 versus 1960 example (p. 70).
2. David H. Bailey, Jonathan M. Borwein, Marcos López de Prado and Qiji Jim Zhu, "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance", Notices of the American Mathematical Society 61(5), 2014, pp. 458–471 (we read the authors' version of 1 April 2014, davidhbailey.com). Our source for: the number of configurations tried is rarely reported (abstract); the seasonal rule on random data and "any random data contains similar patterns" (pp. 24–26). One author was at an investment manager, Guggenheim Partners, at the time.
3. Shiyang Huang, Yang Song and Hong Xiang, "The Smart Beta Mirage", Journal of Financial and Quantitative Analysis 59(6), 2024, pp. 2515–2546, doi:10.1017/S0022109023000674 (open access). Our source for: 238 US equity smart beta funds listed 2000–2018; index returns relative to the market, adjusted for market risk, of 2.77% a year before listing and −0.44% after, and −0.87% for the funds after fees (Table 2, p. 2528); the average periods of about 13 and 6 years (p. 2527); the enhanced value index (Figure 1, p. 2519); the conclusion quoted (p. 2521); multi-factor against value and growth indexes, −4.91 and −1.76 percentage points (Table 8, p. 2536, and p. 2535); 77 funds outside the United States, 37 of them European, 3.02% before and −0.15% after against their regional market (pp. 2541–2544, Table 11).
4. Thomas Wiecki, Andrew Campbell, Justin Lent and Jessica Stauth, "All that glitters is not gold: Comparing backtest and out-of-sample performance on a large cohort of trading algorithms", Quantopian working paper, 2016 (SSRN 2745220). Our source for: 888 algorithms; backtest measures such as the Sharpe ratio explaining less than 2.5% of later performance, and the quote on more backtesting (abstract); the shares explained for volatility (67%), worst fall (34%), Sharpe ratio (2%) and yearly return (1.5%, weakly negative) (pp. 5–6). The authors worked for the platform.
5. S&P Dow Jones Indices, Methodology: Dow Jones FXCM Dollar Index, 2012, Performance Disclosure (p. 4) and Data Availability (p. 3). Our source for: first calculated on 23 March 2011, back-tested to 31 December 1998, and the quoted disclosure. S&P Dow Jones Indices licenses its indexes for a fee.
6. US Commodity Futures Trading Commission, 17 CFR § 4.41(b)(1)(i), the statement required next to simulated or hypothetical performance (ecfr.gov).
7. John West and Alex Pickard, "Plausible Performance: Have Smart Beta Return Claims Jumped the Shark?", Research Affiliates, December 2019 (researchaffiliates.com). Our source for: "10 to 15 years of history"; "backtested results are easily data mined"; most smart beta backtests ignoring costs; 23,740 ten-year periods of US funds, 1979–2018, and "the answer is never. Not once." (text and Table 2). Research Affiliates sells smart beta strategies.
8. Man Group, "Overfitting and Its Impact on the Investor", Man AHL Academic Advisory Board discussion, 31 May 2015 (man.com). Our source for: the definition of overfitting, the changing investment environment, and "Theory is flexible". Man Group runs systematic investment funds.
9. Campbell R. Harvey and Yan Liu, "Backtesting", The Journal of Portfolio Management 42(1), 2015, pp. 12–28 (duke.edu). Our source for: "The strategies with very high Sharpe ratios are probably true discoveries."
10. Daniel Kahneman, Thinking, Fast and Slow, Farrar, Straus and Giroux, 2011, Introduction, as the publisher put it online (us.macmillan.com; archived copy). Our source for: "We are prone to overestimate how much we understand about the world …".
11. Daniel Kahneman, "Don't Blink! The Hazards of Confidence", The New York Times Magazine, 19 October 2011, adapted from Thinking, Fast and Slow (nytimes.com). Our source for: the illusion of validity, "Confidence is a feeling …", the 25 wealth advisers over eight years, the 28 correlations averaging .01, and "rewarding luck as if it were skill".
12. Daniel Kahneman, "Beware the 'inside view'", McKinsey Quarterly, November 2011, an edited excerpt from Thinking, Fast and Slow (chapter 23) (mckinsey.com). Our source for: the curriculum story ("narrowly centered around two years", about 40 percent, seven years, "completed eight years later") and the quote on the outside view.
13. Daniel Kahneman, autobiographical essay, Les Prix Nobel 2002 (nobelprize.org). Our source for: the flight instructor and the coin demonstration.
14. Baruch Fischhoff, "Hindsight ≠ Foresight: The Effect of Outcome Knowledge on Judgment Under Uncertainty", Journal of Experimental Psychology: Human Perception and Performance 1(3), 1975, pp. 288–299. Our source for: the mean increase of 10.8 percentage points (p. 290) and the quote on unawareness (abstract, p. 288).
15. Commission Delegated Regulation (EU) 2017/565, Article 44(5)(c), in the Official Journal of the European Union L 87, 31 March 2017 (eur-lex.europa.eu). Our source for: the prominent warning required when information refers to simulated past performance.
16. Amos Tversky and Daniel Kahneman, "Belief in the Law of Small Numbers", Psychological Bulletin 76(2), 1971, pp. 105–110, doi:10.1037/h0031322 (we read a copy hosted at stats.org.uk). Our source for: the researcher who computes all correlations and reports the single significant one (p. 109).
17. * ProofOfReturns, own simulations, September 2026, on a made-up market with no pattern in it: the coin-toss strategies (Figures 2 and 3) and the prediction models (Figures 4 and 5). They illustrate a mechanism; they are not measurements of any real market. Each ran once with a fixed starting value, and we did not go looking for a nicer run.

How we checked. Verified 26 September 2026. Every figure from sources 1 to 8 and 14 was pulled from its source document twice, independently, and the two readings were compared cell by cell against the document itself; they did not differ on a single number. Sources 9 to 13, 15 and 16 were read once and carry only quotations and the handful of figures named above. We have not read Thinking, Fast and Slow itself: every quotation we attribute to Kahneman comes from the book's own introduction as the publisher put it online, from an excerpt the publisher authorised, or from pieces he wrote himself, and we say which. Four things a careful reader should know. First, in source 3 the period "before listing" starts when the index's history starts, not when the index was first published; on average about three of those thirteen years (the median was five months) were already calculated in real time, only without a fund to buy. The authors call the whole period backtested, and report that using the publication date instead gives sharper results. Second, in source 4 the period after launch was simulated trading on live data, not real money, and it was short: six months to about a year. Third, source 4 gives the share explained by backtest measures as "less than 0.025" in its abstract and "less than 0.25" in its discussion; we use the abstract's figure, and the individual results it reports for the full-period Sharpe ratio, the Sortino ratio and yearly return all lie below it. Fourth, source 7 writes in its text that one outcome "happened twice", while its own table gives a different count; we quote only the line its table supports. Sources 1, 2, 4, 5, 7 and 8 come wholly or partly from people at firms with a stake in this business: a trading platform, an index provider and three investment managers. We name that in each source note. Think we got something wrong? Tell us at info@proofofreturns.com. We correct in public.