Methodology
Evidence Loops
A methodology for transforming execution outcomes into institutional knowledge through continuous cycles of observation, attribution, aggregation, and feedback injection.
Objective
Transform execution outcomes — every trade, every exit, every regime transition — into institutional knowledge that measurably improves future decision generation and qualification. The methodology produces a closed feedback loop where each cycle's outcomes directly influence the next cycle's inputs, compounding decision quality over time.
Scope
Covers
- Exit audit data capture (MAE/MFE, drift, mechanism attribution)
- Outcome attribution at the parameter level (bucketed decision fingerprints)
- Aggregation by family, timeframe, regime, and asset
- Weekly learning summary computation
- Feedback injection into generation prompts and template parameters
- Fee-bleeder detection and generation filtering
- Regime transition evidence accumulation
Does Not Cover
- Individual attribution algorithms (e.g., how MAE is calculated — that is instrumentation detail)
- Strategy generation mechanics (covered by the Decision Lifecycle methodology)
- Parameter optimization trials (covered by the Continuous Optimization methodology)
- Qualification assessment logic (covered by the Validation-First Qualification methodology)
- Market regime classification (an input, not produced here)
Preconditions
These inputs and assumptions must be satisfied before the methodology can produce valid results.
| Input | Requirement |
|---|---|
| Exit instrumentation | position_exit_audit table capturing MAE, MFE, post-exit drift, close reason, and mechanism for every closed position |
| Outcome attribution pipeline | Parameter-level bucketing (indicator, period, operator → outcome) with sufficient closed positions (minimum 20 per bucket for statistical significance) |
| Learning summary infrastructure | Weekly aggregation job that combines backtest patterns, paper trading outcomes, exit attribution, and regime data |
| Generation pipeline with injection points | Template engine and AI generation prompts that accept learning summary data as input |
| Regime catalog | Classified regime periods for attributing outcomes to market conditions |
| Closed position history | Sufficient closed positions across multiple families, timeframes, and regimes to detect patterns |
Procedure
Capture Exit Audit Data
On every position close (paper or live), record exit quality metrics in position_exit_audit. Capture: close reason (exit_signal, stop_loss, take_profit, trailing_stop, time_decay, manual), MAE (maximum adverse excursion during position lifetime), MFE (maximum favorable excursion), post-exit drift (price movement in the candles after close), entry regime label, and exit mechanism attribution. This is the raw observation layer.
Decision Criteria
Every closed position must produce an exit audit record. Missing records break the evidence chain. Post-close hooks (qualification refresh + micro-backtest divergence) fire asynchronously and do not block the close.
Compute Outcome Attribution
Aggregate exit data into parameter-level outcome attribution. Bucket each trade's decision fingerprint: which indicators were used, what operator/threshold combinations, which family, timeframe, and regime context. For each bucket, compute win rate, average return, trade count, and confidence interval. Weekly attribution aggregate job computes preferred/avoid/neutral recommendations per parameter bucket.
Decision Criteria
Minimum 10 trades per bucket for any recommendation. Minimum 20 trades for high-confidence recommendations. Buckets with fewer trades remain 'neutral' (insufficient data). Wilson-score Bayesian smoothing via M4 indicator ranker provides per-regime rankings.
Aggregate by Family, Timeframe, and Regime
Roll up parameter-level attribution into higher-order patterns. Compute: (a) per-family success rates across timeframes, (b) per-timeframe success rates across families, (c) per-regime performance by family, (d) fee-bleeder rates (% of strategies with rolling 20-trade avg return < 0.3%) by family x timeframe, (e) asset family scores blending backtest pass rates with regime-aware paper trading PnL.
Decision Criteria
Fee-bleeder rate > 40% for a family x timeframe combo → flag for hard-gating in generation. Rate 20-40% → advisory log. Asset family scores: 'preferred' (paper confirms backtest), 'neutral' (insufficient data), 'caution' (negative overall but has regime edge), 'avoid' (negative in all regimes or catastrophic — 3+ trades at avg < -1%).
Compute Weekly Learning Summary
The weekly learning summary job aggregates all evidence layers into a single document that the generation layer consumes. Inputs: backtest patterns (IS/OOS pass rates by family, common rejection reasons), paper trading outcomes (qualified/retired/active by family and regime, top 10 paper winners with specific indicator patterns), exit attribution (per-family exit profile effectiveness, stop/TP/trailing hit rates), regime transition matrix (which transitions are toxic, which are survivable), and fee-bleeder rates per family x timeframe.
Decision Criteria
Learning summary must be internally consistent — fee-bleeder rates should match the paper trading outcomes that generated them. If learning data is sparse (new system or regime), summary includes confidence flags. Paper winners section includes per-trade divergence and proven regimes to distinguish luck from skill.
Inject Feedback into Generation
The learning summary is consumed at generation time by both the template engine and AI generation prompts. Template engine: fee-bleeder combos are hard-gated (candidates skipped with audit log), M4 indicator advisor adjusts periods for top/weak indicators, exit-profile-advisor adjusts rotation order and multipliers, regime-aware direction resolution uses transition data. AI generation: learning summary is injected into the LLM prompt (market context, preferred families, avoid combos, indicator guidance, top performer patterns).
Decision Criteria
Generation diversity must be preserved — learning feedback narrows the search space but must not collapse it. M4 circuit breaker: when >50% of ranked indicators are suppressed for current regime, skip all period adjustments. Preferred-asset concentration cap: max 1 use per preferred asset per batch. 25% exploration budget in foundry sweeps generates unconstrained candidates.
Validate Improvement via Next-Cycle Metrics
After feedback injection, measure whether the next generation cycle produces better candidates. Track: (a) backtest pass rate (should increase if fee-bleeder combos are filtered), (b) paper qualification rate (should increase if learning-informed candidates are higher quality), (c) generation diversity (should remain healthy despite narrower search space), (d) exploration budget utilization (should produce occasional breakthrough candidates outside the learned distribution).
Decision Criteria
If backtest pass rate declines after learning injection → learning summary may contain incorrect signals. Investigate specific filters. If diversity collapses → learning is over-constraining. Increase exploration budget. If qualification rate improves → loop is working. Track per-cycle metrics to detect when learning plateaus.
Expected Outputs
Learning Summary
Weekly aggregation of all evidence layers: backtest patterns, paper outcomes, exit attribution, regime transitions, fee-bleeder rates, top performers — consumed by generation pipeline
Fee-Bleeder Generation Filters
Hard gates that prevent generation of family x timeframe combos where >40% of historical strategies are fee bleeders
Parameter Attribution Records
Per-bucket (indicator, period, operator, regime) win rates and recommendations (preferred/avoid/neutral)
Asset Family Scores
Four-tier classification (preferred/neutral/caution/avoid) blending backtest pass rates with regime-aware paper PnL
Indicator Guidance
M4 Wilson-score rankings per regime, consumed by template engine (period adjustments) and AI prompt (indicator suggestions)
Interpretation Guide
How to read the outputs this methodology produces.
Fee-bleeder rate > 40%
This family x timeframe combination consistently produces strategies whose returns are consumed by fees
Action: Hard-gate in generation. Do not produce candidates in this combo. Log to audit trail.
Fee-bleeder rate 20-40%
Elevated but not conclusive. Some strategies in this combo may be viable.
Action: Advisory log. Continue generation but monitor next cycle's paper outcomes.
Asset family score: preferred
Paper trading confirms backtest — this asset x family combo produces positive outcomes in current conditions
Action: Prioritize in generation. Cap at 1 use per preferred asset per batch to prevent concentration.
Asset family score: avoid
Negative in all regimes or catastrophic (3+ trades at avg < -1% with no regime edge)
Action: Deprioritize in generation (moved to end of ranking). Not hard-blocked — may recover in future regimes.
M4 indicator: top-ranked (>55% win rate, 10+ trades)
This indicator has demonstrated above-average predictive value in the current regime
Action: Tighter periods in template engine. Featured in AI prompt. Subject to concentration cap.
M4 circuit breaker active
More than half of ranked indicators are suppressed — M4 signal is noisy
Action: Skip all M4 period adjustments. Fall back to default template parameters.
Qualification rate improving cycle-over-cycle
Learning loop is producing measurably better candidates
Action: Continue current learning configuration. Monitor for plateau.
Worked Example
Scenario
The evidence loop detects that scalping strategies on 15-minute timeframes are overwhelmingly fee bleeders, and injects this knowledge into the next generation cycle.
Exit audit records accumulated: 47 closed positions from scalping x 15m strategies over 6 weeks. Average return per trade: 0.18% (below 0.3% fee threshold).
→ Fee-bleeder flag raised on 34/47 positions (72.3% fee-bleeder rate for scalping x 15m)
Outcome attribution buckets scalping x 15m: win rate 61%, but avg winning trade +0.42%, avg losing trade -0.38%. Net after fees: -0.06% per trade.
→ Parameter-level attribution: scalping x 15m → 'avoid' recommendation
Aggregation: scalping x 15m fee-bleeder rate = 72.7% (> 40% threshold). Scalping x 1h fee-bleeder rate = 23% (elevated but below threshold).
→ Hard-gate: scalping x 15m. Advisory: scalping x 1h.
Weekly learning summary includes: 'fee_bleeder_rates: { scalping_15m: 0.727 }' and 'asset_family_scores' with scalping downgraded.
→ Learning summary document updated with new fee-bleeder data
Next generation cycle: template engine encounters scalping x 15m candidate → skipped with audit log 'fee_bleeder_hard_gate'. AI prompt includes fee-bleeder warning.
→ Zero scalping x 15m candidates produced. Scalping x 1h candidates still generated with advisory note.
After 2 more cycles: overall backtest pass rate improved from 34% to 41%. Paper qualification rate improved from 12% to 16%. Diversity maintained (no single family > 55%).
→ Loop validated: removing fee-bleeder combos improved downstream quality without collapsing diversity.
Outcome
The evidence loop detected a systematic fee-bleeder pattern (scalping x 15m), propagated the evidence through attribution and aggregation, injected it into generation as a hard gate, and measured a 7% improvement in backtest pass rate and 4% improvement in qualification rate. Total loop cycle time: 2 weeks (1 week to accumulate evidence, 1 week to measure impact).
Failure Conditions
How to recognize when the methodology is not producing valid results.
Exit attribution coverage < 90%
Closed positions exist without exit audit records — evidence chain is broken
Fix instrumentation. Missing attribution means the learning summary is built on incomplete data.
Learning summary produces empty sections
Insufficient closed positions in some family/timeframe/regime buckets
Expected in early system lifecycle. Flag sparse sections with confidence warnings. Do not generate recommendations from insufficient data.
Diversity collapses after feedback injection
Learning constraints are over-constraining — too many combos gated or deprioritized
Increase exploration budget (default 25%). Review gate thresholds (40% fee-bleeder rate may be too aggressive for sparse data).
Backtest pass rate declines after learning injection
Learning summary contains incorrect or stale signals
Audit specific filters that were added. Check if fee-bleeder rates were computed from too few trades. Temporarily disable suspect filters.
Qualification rate plateaus despite active learning
The loop has extracted most available value from current evidence — diminishing returns
Normal. Shift focus to exploration budget and structural diversity (new indicator combinations, new asset classes).
Same parameter buckets oscillate between preferred and avoid
Evidence is noisy or regime-dependent — bucket classification is unstable
Increase minimum trade count for recommendations. Add regime conditioning to attribution buckets.
Quality Checks
| Check | Passes When | Fails When |
|---|---|---|
| Exit attribution covers all closed positions | Every closed position (paper and live) has a corresponding exit audit record | Positions exist in closed status without exit audit data |
| Fee-bleeder rates match paper trading data | Computed rates are consistent with observable paper trading outcomes | Rates are stale, computed from wrong time window, or use different fee assumptions |
| Learning summary is internally consistent | Fee-bleeder filters, asset family scores, and M4 rankings align with the underlying attribution data | Summary sections contradict each other (e.g., a family is 'preferred' but its fee-bleeder rate exceeds 40%) |
| Generation diversity survives feedback injection | No single family exceeds 60% of generated candidates; exploration budget is utilized | One or two families dominate generation after learning filters remove too many alternatives |
| Cycle-over-cycle metrics are tracked | Backtest pass rate, qualification rate, and diversity metrics are compared across consecutive learning cycles | No longitudinal tracking — impossible to tell if the loop is improving or degrading quality |
Handoff
Where the outputs of this methodology go next in the protocol chain.
Strategy Generation Pipeline
Foundry Research Pipeline
Strategy Fit Scores
Allocation Intelligence
Termination State
Produces
No terminal state — the evidence loop is continuous. Each weekly cycle produces an updated learning summary that reflects the latest evidence. The methodology 'terminates' only when the system is shut down.
Confidence
Confidence in learning signals increases with trade count and regime diversity. A learning summary from 1,000+ closed positions across 3+ regimes is high-confidence. A summary from 50 positions in a single regime is low-confidence with appropriate warnings.
Next Step
Each learning summary is consumed by the next generation cycle. Evidence quality is validated by tracking cycle-over-cycle improvement metrics. If the loop plateaus, the response is increased exploration, not abandonment of learning.
Replication Notes
What another team would need to reproduce this methodology.
- Exit attribution must be captured at close time (not backfilled) to preserve temporal accuracy of post-exit drift
- Parameter-level bucketing requires a consistent indicator vocabulary — changing indicator names breaks bucket continuity
- Fee-bleeder detection uses a 0.3% round-trip cost assumption calibrated to Hyperliquid taker fees — recalibrate for different venues
- The learning summary aggregation job must run weekly with consistent timing (Monday 4am UTC) to maintain cycle regularity
- M4 indicator rankings use Wilson-score Bayesian smoothing — raw win rates without smoothing produce noisy rankings from small samples
- Exploration budget (20% of foundry candidates) is essential to prevent the loop from collapsing into a local optimum
- The loop requires longitudinal tracking infrastructure to measure cycle-over-cycle improvement — without it, you cannot validate that learning is actually helping
Knowledge Lineage
Introduced In
Expanded In
Modeled By
Investigated In
Influences
Related Research
Research Areas
Frameworks
See this methodology implemented in Orqis