# Wansiri Research Note # WRN-001 --- Why a Profitable Backtest Is Not Evidence ### Evidence from 1,182 Cryptocurrency Strategy Evaluations **Publication type:** Historical research **Research period:** January 2022 -- July 2026 **Status:** Completed analysis as described in this research note ------------------------------------------------------------------------ ## Abstract A profitable backtest is frequently interpreted as evidence that a trading strategy possesses a genuine trading edge. Quantitative finance research has consistently challenged this assumption, demonstrating that historical profitability alone provides weak evidence once multiple testing, parameter optimisation and non-stationary markets are considered. This research note examines the same question using empirical evidence generated by the Wansiri research framework. Stage 1 of the framework applies multiple statistical validation procedures before any walk-forward or production testing is performed. Across **1,182 statistically valid strategy evaluations**, only **22.5%** satisfied the robustness criteria required to progress automatically to the next stage of research, while **68.7%** were classified as statistically non-viable. The findings reinforce a central conclusion of the existing literature: historical profitability should be viewed as the beginning of quantitative research rather than its conclusion. ## 1. Research Question Can a profitable backtest, by itself, provide sufficient evidence that a trading strategy possesses a genuine and deployable trading edge? This question sits at the centre of systematic trading research. Every quantitative strategy begins with historical testing, yet the majority may fail to survive exposure to unseen market conditions. The practical challenge is therefore not simply producing profitable backtests, but determining which of those backtests provide credible evidence of a genuine market inefficiency. This research note investigates that question using empirical evidence generated by the Wansiri research framework. ## 2. Existing Literature The statistical limitations of backtesting are well established. White (2000) demonstrated that evaluating large numbers of trading rules substantially increases the probability of selecting apparently profitable strategies through chance alone. Hansen (2005) developed the Superior Predictive Ability (SPA) test, refining the evaluation of competing trading strategies while controlling for data snooping. Bailey, Borwein, López de Prado and Zhu developed the Probability of Backtest Overfitting (PBO) framework, examining how extensive strategy searches increase the likelihood of selecting overfit strategies. Bailey and López de Prado later introduced the Deflated Sharpe Ratio (DSR), which adjusts the interpretation of observed Sharpe ratios for selection effects and non-normal return distributions. Collectively, this literature shifts the role of a backtest from proof of a trading edge to an initial observation requiring further validation. The practical question is how a quantitative research framework should accumulate stronger evidence once a profitable backtest has been identified. ## 3. Wansiri Research Framework The Wansiri research framework treats every profitable backtest as a research hypothesis rather than a deployable trading strategy. The dataset analysed in this study consisted of long-only spot cryptocurrency basket strategies evaluated between **January 2022 and July 2026**. Stage 1 applies multiple validation procedures before a strategy progresses to more demanding forms of evaluation. These procedures include: - Deflated Sharpe Ratio (DSR). - Effective independent-trial estimation. - Robustness analysis. - Parameter sensitivity analysis. - Autocorrelation diagnostics. - Mechanism validation. The objective is not merely to identify profitable strategies. Every strategy entering Stage 1 has already demonstrated historical profitability. The objective is to determine whether that profitability survives increasingly demanding statistical scrutiny. Only strategies satisfying the applicable Stage 1 requirements progress to downstream validation, including walk-forward testing, execution analysis, paper trading and production assessment. Those subsequent stages are outside the scope of this research note. ## 4. Empirical Findings The original research campaign generated **1,296 systematic trading test cases** spanning **nine cryptocurrency basket groups** and **144 market regime classifications**. After data-quality validation, **1,182 statistically valid strategy evaluations** remained. Each evaluation represented a historically profitable trading strategy entering Stage 1 validation. The resulting classifications were: Classification Count Percentage ------------------ ----------- ------------ ROBUST 266 22.5% METHOD_DEPENDENT 104 8.8% NOT_VIABLE 812 68.7% **Total** **1,182** **100.0%** ### Figure 1. Stage 1 Validation Outcomes ![Stage 1 Validation Outcomes](wrn-001-validation-outcomes.png) Several observations emerge. First, nearly **seven out of every ten historically profitable strategy evaluations failed statistical validation** before any out-of-sample testing was performed. Second, fewer than one quarter demonstrated sufficient robustness to progress automatically to the next stage of research. Finally, a subset produced conclusions that depended on the statistical methodology itself. These evaluations were classified as **METHOD_DEPENDENT**, indicating that additional investigation was required before a promotion decision could be justified. The purpose of Stage 1 is therefore not simply to identify winners. Its purpose is to eliminate weak or potentially false discoveries before additional research resources are committed. ## 5. Discussion The empirical observations support the conclusions reached throughout the existing academic literature: historical profitability alone provides limited evidence that a trading strategy possesses a genuine and persistent edge. The Wansiri framework expresses the research problem differently from a conventional optimisation exercise. Rather than asking only whether a strategy produced attractive historical returns, it asks whether the strategy has accumulated sufficient evidence to justify progressing to the next stage of research. This changes the objective of quantitative research. Rather than optimising historical performance, the objective becomes the systematic accumulation of evidence. A high rejection rate should not automatically be interpreted as a weakness of the research framework. It may indicate that the validation process is distinguishing between attractive historical results and results that survive additional scrutiny. Validation is therefore both a statistical exercise and a resource-allocation process: only the strongest research hypotheses should receive further investment. ## 6. Research Implications Three practical implications follow. **First**, historical profitability should be regarded as a hypothesis rather than a conclusion. **Second**, statistical validation should combine multiple sources of evidence rather than relying on a single performance metric. **Third**, confidence should increase progressively as strategies survive increasingly realistic validation environments, including walk-forward testing, execution analysis, paper trading and live production. This transforms quantitative research from an optimisation problem into an evidence-accumulation process. ## 7. Conclusions The purpose of a backtest is not to prove that a trading strategy possesses a genuine edge. Its purpose is to identify hypotheses worthy of further investigation. Across **1,182 statistically valid strategy evaluations**, only **22.5%** demonstrated sufficient robustness to progress automatically beyond Stage 1 statistical validation. The remaining evaluations were either classified as non-viable or required further investigation before promotion could be justified. These findings reinforce the central conclusion reached throughout the quantitative finance literature: historical profitability alone provides limited evidence that a strategy represents a genuine market opportunity. Confidence should increase only as independent sources of evidence accumulate throughout the research process. A profitable backtest is therefore not the conclusion of research. **It is the point at which rigorous research begins.** ## References - White, H. (2000). *A Reality Check for Data Snooping.* - Hansen, P. R. (2005). *A Test for Superior Predictive Ability.* - Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. (2014). *The Probability of Backtest Overfitting.* - Bailey, D. H., & López de Prado, M. (2014). *The Deflated Sharpe Ratio.*