Decision Quality

01

The most common error in evaluating decisions is judging them by their outcomes. A trader who bets their portfolio on a single coin and doubles their money is celebrated. A risk manager who diversifies across uncorrelated assets and underperforms that single bet is questioned. The outcome is visible; the process is invisible.

This conflation is not merely an analytical error — it is a structural one. When systems learn from outcomes alone, they absorb survivorship bias, regime effects, and small-sample luck into their models. A strategy that happened to be long during a bull market looks brilliant. The same strategy in a ranging market looks broken. Neither judgment is reliable because neither distinguishes between the quality of the decision and the environment in which it executed.

The problem intensifies in automated systems. When an algorithm evaluates its own strategies based purely on P&L, it cannot distinguish between a strategy that lost money because the market moved against it and a strategy that lost money because it is fundamentally broken. The first deserves patience. The second deserves retirement. Outcome-only evaluation treats them identically.

02

Measuring decision quality requires decomposing the concept into concrete, independently assessable dimensions. A single “quality score” would reproduce the very conflation problem it aims to solve. Instead, the system implements six distinct layers, each measuring a different aspect of decision quality.

These layers are ordered from most deterministic to most probabilistic. The first layer is binary — a strategy either meets the criteria or it does not. The sixth layer is statistical — it measures whether the overall selection process produces better portfolios over time. Each layer adds nuance, but none depends on the layers above it. A strategy can fail at Layer 1 (qualification) regardless of what Layer 6 (capital intelligence) might say about it.

Layer 2: Process Quality Scoring. Robustness is measured as the transfer ratio between in-sample and out-of-sample performance. A strategy that returns 40% in-sample but -5% out-of-sample has poor process quality regardless of its absolute performance. Behavioral consistency checks whether the strategy trades in a way that matches its declared thesis — a mean-reversion strategy that only enters trend-following trades has low process quality even if it profits.

Layer 3: Confidence-Adjusted Assessment. Raw metrics are discounted by sample size. A strategy with 3 trades and 80% win rate is not assessed the same as one with 50 trades and 80% win rate. The system applies confidence penalties that increase the uncertainty band around small-sample results, preventing statistical flukes from being treated as reliable signals.

Layer 4: Context-Dependent Evaluation. Performance is measured relative to market regime. A strategy that returns 5% during a trending market is evaluated differently from one that returns 5% during a choppy market, because the difficulty of generating returns differs across regimes. Regime-specific evaluation prevents strategies from being penalized for environmental headwinds or rewarded for environmental tailwinds.

Layer 5: Outcome Attribution. When a trade closes, the system fingerprints the decision that produced it — which entry conditions fired, which exit mechanism triggered, what the market regime was at entry and exit. These fingerprints accumulate into parameter-level learning: not “did RSI work?” but “did RSI > 70 as an entry condition for short positions in trending-down regimes produce positive outcomes across 20+ trades?”

Layer 6: Capital Intelligence KPIs. The meta-layer that measures whether the selection process itself produces value. Qualification spread (do qualified strategies outperform unqualified ones?), regime coverage (does the portfolio have strategies ready for each market condition?), and deployment efficiency (what fraction of qualified strategies are actually deployed?) are system-level indicators that the decision quality framework is functioning.

03

The most dangerous artifact in automated decision systems is the small-sample outlier. A strategy that takes 3 trades and wins all 3 shows a 100% win rate. A strategy that takes 50 trades and wins 35 shows a 70% win rate. Naive evaluation prefers the first. Calibrated evaluation recognizes that 3 trades tell you almost nothing.

Confidence discounting addresses this by adjusting quality assessments based on sample size. The system applies larger uncertainty margins to strategies with fewer observations, effectively widening the confidence band around their metrics. A strategy needs to demonstrate consistent behavior over a statistically meaningful number of trades before its metrics are treated as reliable.

This mechanism interacts with the qualification gates at Layer 1. The minimum trade count requirement (5 closed trades for global, 3 for regime-specific) establishes a floor. Confidence discounting adds granularity above that floor — a strategy with 6 trades is qualified but less trusted than one with 30 trades. The qualification gate is binary; confidence adjustment is continuous.

04

A strategy declares a thesis — mean reversion, momentum, breakout. Its actual trading behavior should be consistent with that thesis. A mean-reversion strategy that only enters trades in the direction of the trend is exhibiting a thesis-behavior mismatch. It may still be profitable, but its profitability is coming from a source other than its declared edge, which makes it unpredictable.

Behavioral intelligence detects these mismatches through pattern analysis of actual trades. The system examines entry timing relative to market structure, holding periods relative to declared timeframe, and exit mechanisms relative to risk parameters. A 4-hour momentum strategy that holds positions for 15 minutes is behaving like a scalper. A breakout strategy that enters on pullbacks is behaving like a mean-reversion strategy.

These mismatches are not always negative — a strategy might be accidentally discovering a better edge than the one it was designed to capture. But they are always informative. When behavior diverges from thesis, the system flags the divergence as an intelligence signal rather than treating it as noise. This signal feeds back into the learning loop: the template generator learns which thesis-behavior combinations produce stable edges and which produce unreliable ones.

05

The decision quality framework is not a theoretical proposal — it is a production system with measurable outputs. The evidence presented here comes from the same operational infrastructure described in the preceding sections, running on live market data with real strategy candidates.

The key question is not whether the six layers produce different scores — they are designed to, by definition. The key question is whether process-oriented measurement identifies failures that outcome-only measurement misses, and preserves strategies that outcome-only measurement would incorrectly discard.