Methodology
Validation-First Qualification
A diagnostic protocol for determining whether a decision candidate has accumulated sufficient evidence to warrant resource allocation.
Objective
Determine whether a strategy candidate has accumulated sufficient evidence — across historical validation, forward testing, and regime analysis — to qualify for live capital deployment. The methodology produces a qualification decision with an explicit evidence trail, not a score.
Scope
Covers
- Historical backtest validation (in-sample and out-of-sample)
- Forward testing via paper trading simulation
- Backtest-to-paper divergence detection
- Regime-conditional performance assessment
- Multi-dimensional qualification criteria
Does Not Cover
- Strategy generation or parameter selection
- Capital sizing or portfolio construction
- Exchange connectivity or order routing
- Post-deployment monitoring or exit mechanics
- Market regime classification (handled by separate methodology)
Preconditions
These inputs and assumptions must be satisfied before the methodology can produce valid results.
| Input | Requirement |
|---|---|
| Strategy specification | Valid StrategySpec with indicators, entry/exit conditions, and risk parameters |
| Historical price data | Minimum 90 days of OHLCV candle data at the strategy's timeframe |
| Regime catalog | Classified regime periods covering the backtest window |
| Paper trading infrastructure | Tick-based simulation with isolated capital and position tracking |
| Qualification criteria | Defined thresholds for each qualification dimension |
Procedure
Pre-validation
Validate the strategy specification against the canonical StrategySpec contract. Check indicator support, parameter ranges, risk limits, and structural validity. Reject malformed or unsupported configurations before consuming compute.
Decision Criteria
If spec validation fails → terminate with 'invalid_spec'. No partial credit.
Historical Validation (In-Sample)
Execute the strategy against the in-sample portion of historical data (default: first 70% of the window). Record all trades, entry/exit conditions, and position outcomes. Compute composite scoring: return (30%), win rate (20%), Sharpe ratio (20%), max drawdown (15%), trade count (15%).
Decision Criteria
If zero trades generated → terminate with 'zero_trades'. If composite score < minimum threshold → terminate with 'failed_backtest'.
Out-of-Sample Validation
Execute the same strategy against the held-out portion (last 30%). Compare IS and OOS performance. Compute confidence-adjusted score with fee-efficiency multiplier (0.70-1.00 based on average trade return vs 0.3% round-trip fees).
Decision Criteria
If OOS performance degrades beyond coherence threshold → flag as 'failed_oos'. Strategy may still proceed to paper trading at reduced confidence.
Paper Trading Admission
Admit the validated strategy to paper trading with isolated capital. The strategy runs against live market data at 1-minute tick cadence. No capital risk. Minimum 7 days of forward observation required before qualification assessment.
Decision Criteria
If portfolio diversity constraints are violated (asset, family, timeframe caps) → queue for admission when slot opens. If bot capacity exhausted → reject with 'capacity_full'.
Forward Performance Observation
Monitor paper trading performance over the observation period. The 6-hourly evaluation batch computes divergence metrics, behavioral signatures, and regime-conditional performance. Do not intervene in trading — observe only.
Decision Criteria
If paper ROI diverges from backtest ROI beyond threshold (cohort divergence p < 0.05) → flag for divergence review. If fee-bleeder pattern detected (rolling 20-trade avg return < 0.3%) → log advisory alert.
Qualification Assessment
Apply multi-dimensional qualification criteria. All dimensions must pass — there is no weighted average. Dimensions: (a) minimum paper trading duration, (b) minimum trade count, (c) backtest-paper divergence within tolerance, (d) regime proof (positive performance in at least one classified regime), (e) no active disqualifying conditions.
Decision Criteria
All dimensions pass → 'qualified'. Any dimension fails → remains 'unqualified' with specific failure reason. Qualification is re-assessed every 6 hours — a strategy that fails today may qualify tomorrow as evidence accumulates.
Expected Outputs
Qualification Status
One of: qualified, conditionally_qualified, globally_qualified_regime_unproven, unqualified, disqualified
Evidence Trail
Per-dimension pass/fail with specific metrics: divergence p-value, regime proof details, trade count, duration
Regime Proof
Which regimes the strategy has demonstrated positive performance in, with trade count and average return per regime
Divergence Assessment
Backtest-to-paper performance comparison with statistical significance
Interpretation Guide
How to read the outputs this methodology produces.
Qualified
All dimensions pass. Strategy has demonstrated forward performance consistent with historical validation across at least one regime.
Action: Eligible for live deployment. User may promote with capital allocation.
Conditionally Qualified
Core dimensions pass but regime proof is limited to current regime only.
Action: Eligible for live deployment in current regime. May be paused on regime transition.
Globally Qualified, Regime Unproven
Strong overall performance but has not yet traded in the current regime.
Action: Eligible for deployment but flagged for regime monitoring.
Unqualified
One or more dimensions have not yet passed. May qualify as evidence accumulates.
Action: Continue paper trading. Re-assess on next 6-hour cycle.
Disqualified
Evidence indicates systematic failure. Unlikely to recover.
Action: Consider retirement. Do not promote.
Worked Example
Scenario
A momentum strategy on ETH/USDT with 4h timeframe, RSI + MACD entry conditions, ATR-based trailing stop.
Pre-validation passes: valid indicators, supported timeframe, risk parameters within bounds
→ Spec validated
IS backtest over 63-day window produces 12 trades, composite score 67.3
→ IS validation passes
OOS backtest over 27-day window produces 4 trades, composite score 58.1
→ OOS coherent (within 15% of IS)
Admitted to paper trading with $1,000 isolated capital
→ Paper account created, first tick evaluates
After 18 days: 6 paper trades, 3.2% paper ROI vs 4.1% backtest ROI (over comparable period)
→ Divergence within tolerance (p = 0.34)
Qualification assessment: duration ✓, trades ✓, divergence ✓, regime proof ✓ (trending_up: 4 trades, +2.8% avg)
→ Status: qualified
Outcome
Strategy qualifies for live deployment. Evidence trail records: 6 paper trades over 18 days, divergence p=0.34, regime proof in trending_up. User may promote with chosen capital allocation.
Failure Conditions
How to recognize when the methodology is not producing valid results.
Zero trades in backtest
Entry conditions are too restrictive for the given data window
Reject. Do not proceed to paper trading — will not generate trades.
IS/OOS divergence > 40%
Strategy may be overfit to in-sample period
Flag as failed_oos. May still paper trade but at reduced confidence.
Paper ROI diverges > 3σ from backtest
Forward conditions differ materially from historical assumptions
Log divergence. Continue observation — may resolve with more data.
Fee-bleeder detected (avg return < 0.3%)
Strategy generates trades but profits are consumed by fees
Advisory alert. Not auto-paused but flagged for review.
No trades after 14+ days of paper trading
Strategy is inactive in current market conditions
Not a failure — may be regime-appropriate. Continue observation.
Quality Checks
| Check | Passes When | Fails When |
|---|---|---|
| Cohort divergence uses correct statistical test | p-value computed from matched-period comparison, not all-time comparison | Comparing full backtest period to shorter paper period (apples to oranges) |
| Regime proof uses entry_regime, not exit_regime | Trades attributed to the regime present when position was opened | Trades attributed to regime at close (may have changed mid-position) |
| Fee efficiency reflects actual venue costs | 0.3% round-trip cost assumption matches Hyperliquid taker fees | Fee assumption doesn't match deployment venue |
| Qualification is re-assessed, not cached | Each 6h batch recomputes from current evidence | Qualification status persisted without re-evaluation |
Handoff
Where the outputs of this methodology go next in the protocol chain.
Live Execution Engine
Qualified Strategies Intelligence View
Learning Layer
Termination State
Produces
A qualification decision (qualified/unqualified/disqualified) with a complete evidence trail linking every dimension to specific metrics.
Confidence
Qualification confidence is a function of observation duration and trade count. Strategies with 7 days and 3+ trades have minimum viable confidence. Strategies with 30+ days and 10+ trades have high confidence.
Next Step
Qualified strategies are eligible for user-initiated promotion to live trading. Unqualified strategies continue paper trading with periodic re-assessment.
Replication Notes
What another team would need to reproduce this methodology.
- Backtest engine must support IS/OOS split with configurable ratio (default 70/30)
- Paper trading simulation requires 1-minute tick cadence against live market data
- Regime classification must be available for the full backtest + paper window
- Divergence detection requires matched-period comparison (not full-period)
- Qualification dimensions are conjunctive — all must pass, no weighted average
Knowledge Lineage
Introduced In
Investigated In
Depends On
Influences
Related Research
Research Areas
See this methodology implemented in Orqis