# Wansiri Research Note # WRN-002 --- Temporal Consistency After Statistical Validation ### Testing Whether Statistically Validated Strategy Performance Holds Up Over Time **Publication type:** Historical research **Status:** Completed analysis as described in this research note ------------------------------------------------------------------------ ## Abstract A trading strategy can pass statistical validation while still exhibiting an unstable historical performance profile. WRN-001 examined whether profitable backtests provided sufficient statistical evidence of a genuine trading edge. The Wansiri Stage 1 framework applied the Deflated Sharpe Ratio (DSR) and related statistical procedures to assess whether observed strategy performance was sufficiently robust to progress through the research process. However, DSR evaluates performance using the strategy's return history as a whole. It does not, by itself, determine whether that performance was distributed consistently through time. This research note examines that separate question using the strategy population following statistical validation. Stage 2 divides the historical record of each strategy/regime combination into six equal time periods and evaluates three dimensions of temporal consistency: activity concentration, the frequency of negative periods, and variation in results between periods. Across **1,160 strategy/regime combinations**, **265** had passed the preceding significance test. Of these, **205 (77%)** showed no consistency concerns, **39 (15%)** exhibited one consistency warning, and **21 (8%)** exhibited multiple consistency red flags and were placed on HOLD for manual review. The findings demonstrate that statistical validation and temporal consistency provide different information. A strategy can pass DSR while still displaying a historical performance profile that warrants additional investigation. Stage 2 therefore adds a separate layer of evidence before strategies are permitted to progress toward production deployment. ## 1. Research Question Does a strategy that passes statistical validation demonstrate sufficiently consistent performance across time to justify progression toward production? WRN-001 addressed the first part of this problem using the Deflated Sharpe Ratio (DSR) and related statistical validation procedures. The DSR asks whether observed performance provides sufficient statistical evidence that it is unlikely to be explained by luck and selection effects, given the number of strategy configurations evaluated. However, aggregate historical statistics can conceal how performance was distributed through time. Two strategies may produce similar overall returns and similar aggregate performance statistics while exhibiting very different historical behaviour. One may generate returns relatively consistently throughout its history. Another may generate a large proportion of its activity or performance during one particularly favourable period while producing weak or negative results elsewhere. Both may pass an aggregate statistical test. Stage 2 of the Wansiri research framework was developed to investigate this separate question. ## 2. Relationship to WRN-001 and DSR WRN-001 and WRN-002 are designed to answer different questions. ### WRN-001 --- Statistical validation This provides the first major statistical filter in the research framework. ### WRN-002 --- Temporal consistency Stage 2 asks: These questions should not be conflated. A strategy can pass DSR while still generating a disproportionate amount of its historical result during one favourable period. Conversely, a strategy can exhibit temporal variation without that variation necessarily invalidating its statistical evidence. Stage 2 is therefore not a second DSR test. It is a separate consistency diagnostic. An early implementation attempted to apply DSR-style statistical testing independently to the six historical periods. This approach was rejected because the individual periods did not contain sufficient observations to support the statistical structure required by the DSR calculation. The final methodology therefore uses the six-period decomposition to describe and identify potentially unstable performance patterns, rather than attempting to establish statistical significance from six sub-periods. The distinction is: - **DSR asks whether observed performance is statistically credible.** - **Stage 2 asks whether that performance was reasonably consistent through time.** ## 3. Wansiri Research Framework The Wansiri research framework treats strategy validation as a sequence of increasingly demanding evidence requirements. Following the statistical validation performed in WRN-001, the surviving strategy/regime combinations progress to Stage 2 temporal consistency analysis. For each strategy, the available historical record is divided into **six equal time periods**. Three characteristics are examined: - Activity concentration within a single period. - The number of periods producing negative results. - Variation in performance between periods. The purpose is not to require identical performance across all periods. Financial markets are non-stationary, and genuine strategies may naturally perform differently under different market conditions. Instead, the objective is to identify historical performance profiles that are unusually uneven or unstable and therefore warrant additional scrutiny before production deployment. A strategy is not placed on HOLD because of a single warning signal. The Stage 2 methodology requires multiple consistency red flags before a strategy is recommended for manual review. This reduces the likelihood that normal variation in one metric generates an unnecessary production restriction. ## 4. Empirical Findings The Stage 2 analysis evaluated **1,160 strategy/regime combinations**. The resulting production classifications were: Classification Count Percentage ---------------- ----------- ------------ APPROVE 244 21% HOLD 104 9% REJECT 812 70% **Total** **1,160** **100%** The 812 rejected strategies had failed the preceding statistical validation. Of the 104 HOLD decisions, **21 were attributed to consistency review**. The 21 consistency-review strategies therefore represent a specific subset of strategies that had already passed the preceding significance test. Stage 2 is consequently not simply another broad rejection mechanism. Its primary purpose is to identify statistically credible strategies whose historical performance structure warrants additional review. ## 5. Consistency Results Among Statistically Validated Strategies A total of **265 strategies passed the preceding significance test**. Stage 2 classified these strategies as follows: Consistency classification Count Percentage ---------------------------- --------- ------------ NO CONCERNS 205 77% MINOR WATCH 39 15% REVIEW BEFORE PRODUCTION 21 8% **Total** **265** **100%** The majority of statistically validated strategies, **205 of 265**, exhibited no consistency concerns. A further **39 strategies** exhibited exactly one consistency warning. These were classified as **MINOR WATCH** rather than being automatically blocked from production. The remaining **21 strategies**, representing approximately **8%** of the statistically validated population, exhibited multiple consistency red flags and were placed on HOLD for manual review. This demonstrates that statistical validation substantially reduces the candidate population, but does not eliminate the possibility of historically unstable performance profiles. ## 6. Temporal Performance Profiles The six-period analysis provides a way to examine how strategy behaviour changes throughout the historical sample rather than relying solely on aggregate performance statistics. One dimension examined is the distribution of daily returns across the six periods. The original research compares strategies flagged by the consistency analysis with strategies showing no consistency concerns. Returns are normalised to each strategy's own scale so that the comparison focuses on the shape and stability of return distributions rather than absolute strategy performance. ### Figure 2. Return Volatility by Historical Period --- Flagged vs. Healthy Strategies ![Return Volatility by Historical Period — Flagged vs. Healthy Strategies](wrn-002-period-distribution.png) The flagged examples were reported to exhibit greater variation in the distribution of returns between periods. Some showed periods with wider return distributions, more extreme observations, or marked changes in volatility. The healthy examples generally displayed more stable return distributions across the six periods, although they were not uniformly positive and could also contain individual outliers. The purpose of this comparison is not to establish that a healthy strategy must be profitable in every period. It illustrates the type of temporal instability that the Stage 2 consistency analysis is designed to identify. ## 7. Activity Concentration Across Time A second dimension examines how a strategy's **activity** is distributed across the six historical periods. This is distinct from profit concentration. A strategy may conduct a large proportion of its activity during one period without generating a corresponding proportion of its total profit. Activity concentration is therefore treated as one diagnostic characteristic rather than direct evidence that profitability depends on that period. ### Figure 3. Activity Concentration by Historical Period --- Flagged vs. Healthy Strategies ![Activity Concentration by Historical Period — Flagged vs. Healthy Strategies](wrn-002-pnl-shapes.png) The flagged examples were reported to show more uneven distributions of activity across the six periods. By comparison, the healthy examples generally displayed a more distributed activity profile. However, the analysis also demonstrated that **activity concentration alone was not sufficient to distinguish DSR-passing strategies from DSR-failing strategies**. Both populations could exhibit concentrated activity. A simple rule based only on concentration would therefore generate false positives. For this reason, Stage 2 evaluates concentration together with other consistency measures rather than treating it as an independent rejection criterion. ## 8. Findings from the Consistency Analysis One principal finding was that activity concentration alone did not clearly distinguish DSR-passing strategies from DSR-failing strategies. The strategies identified for manual review more commonly exhibited combinations of warning characteristics. In particular, flagged strategies frequently combined: - Negative results across multiple historical periods. - Substantial variation in performance between periods. - In some cases, substantial concentration within one historical period. This supports the use of a multi-signal consistency framework rather than a single threshold. The result reinforces the distinction between **statistical credibility** and **temporal stability**. A strategy can provide sufficient statistical evidence under an aggregate test while still warranting investigation because its historical performance is unevenly distributed through time. The objective of Stage 2 is not to eliminate all variation. It is to identify performance profiles where multiple characteristics collectively indicate that additional scrutiny is warranted. ## 9. From Research Finding to Production Decision Research findings must ultimately affect the decisions made by the production system. Before Stage 2, downstream configuration and selection outputs were generated from the population of strategies approved by the preceding statistical validation process. This created a potential gap: a strategy could pass DSR, subsequently be identified as having multiple temporal consistency concerns, and nevertheless remain present in downstream production-selection outputs. Stage 2 closes this gap. The consistency analysis produces a per-strategy summary containing: - The three consistency metrics. - The consistency classification. - The resulting production decision. - Supporting diagnostic information. The process then rebuilds the production-eligible strategy universe using Stage 2 decisions. Rankings and dependent selections are recalculated against the reduced population so that downstream selections remain consistent with the approved universe. The subsequent live-account selection uses this Stage 2-approved population. The resulting process is: **Statistical validation → Temporal consistency analysis → Production strategy filtering → Recalculated rankings and controls → Live-account selection** The significance is that the Stage 2 result does not remain isolated within a research report. **The research finding becomes a production constraint.** ## 10. What Stage 2 Does Not Establish Stage 2 evaluates historical performance using data contained within the existing backtests. It therefore does not establish that strategy parameters would have been selected using only information available at the time. If parameters were selected using the full historical sample, dividing that same sample into six periods does not eliminate the possibility that information from later periods influenced the original parameter selection. The latter requires a separate form of validation, such as walk-forward re-optimisation and out-of-sample testing. A clean Stage 2 result should therefore be interpreted as additional supporting evidence, rather than definitive proof of future performance. ## 11. Research Implications Three practical implications follow from the Stage 2 findings. **First**, statistical significance and temporal consistency should be treated as separate dimensions of strategy evidence. Passing an aggregate statistical test does not guarantee that performance is distributed consistently through time. **Second**, temporal consistency is better evaluated through multiple complementary indicators than through a single concentration threshold. The analysis showed that activity concentration alone was not sufficiently discriminating. **Third**, research diagnostics should directly influence production eligibility. Stage 2 changes the population from which production strategies can be selected. This creates a progressive evidence filter in which strategies must satisfy multiple requirements before being permitted to progress toward live deployment. ## 12. Conclusions WRN-001 established that a profitable backtest should not automatically be treated as evidence of a genuine trading edge. The Deflated Sharpe Ratio provides an important statistical filter by addressing whether observed performance is sufficiently unlikely to be explained by luck and selection effects. WRN-002 addresses a different question: **Does that statistically validated performance hold up over time?** Across **1,160 strategy/regime combinations**, **265** passed the preceding significance test. Of those: - **205 (77%)** showed no consistency concerns. - **39 (15%)** showed one consistency warning. - **21 (8%)** showed multiple consistency red flags and were placed on HOLD for manual review. The analysis found that activity concentration alone was not sufficient to distinguish stronger and weaker strategies. Strategies identified for review more commonly exhibited combinations of repeated negative periods and substantial variation in performance across time, with some also showing substantial concentration in individual periods. The findings reinforce an important principle of quantitative strategy research: **Statistical significance and temporal consistency are different forms of evidence.** A strategy should not progress toward production solely because it passes an aggregate statistical test. It should accumulate evidence across multiple dimensions of validation. Stage 2 therefore extends the Wansiri research framework from asking whether observed performance is statistically credible to examining whether that performance is sufficiently stable and distributed across time to justify further progression. **A strategy that passes one test has not finished the validation process. It has earned the right to face the next one.** ## References - Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. (2014). *The Probability of Backtest Overfitting.* - Bailey, D. H., & López de Prado, M. (2014). *The Deflated Sharpe Ratio.* - Hansen, P. R. (2005). *A Test for Superior Predictive Ability.* - White, H. (2000). *A Reality Check for Data Snooping.*