Framework
Qualification Framework
A reusable framework for building qualification gates that determine when ideas earn the right to execute.
Overview
Most systems treat the boundary between idea and action as a single gate — a threshold, a score, an approval. This works when ideas are scarce and expensive. When ideas are abundant and cheap, a single gate cannot distinguish between ideas that happen to look good and ideas that have genuinely earned execution through accumulated evidence.
The Qualification Framework provides a reusable architecture for building multi-dimensional qualification gates across any domain where decisions must earn execution. It is not specific to trading, finance, or any single application. The framework defines how evidence accumulates, how gates are structured, how confidence decays, and how qualification status evolves over time.
The framework is conjunctive by design — all gates must pass simultaneously for qualification to hold. This is a deliberate architectural choice. Disjunctive gates (any gate passes) allow weak dimensions to hide behind strong ones. Conjunctive gates force every dimension to independently justify itself, producing higher-integrity qualification decisions at the cost of higher false-negative rates.
Design Principles
These are governing constraints, not guidelines. Every architectural decision in the framework must be traceable to one of these principles.
Conjunctive Gates
All qualification dimensions must pass simultaneously. No single strong dimension compensates for a weak one.
A strategy with excellent returns but no regime proof is not qualified — it is untested in a critical dimension. Conjunctive evaluation prevents the halo effect where one impressive metric masks fundamental weaknesses.
Continuous Re-Qualification
Qualification is not a credential. It is a continuously evaluated state that must be re-earned as evidence accumulates and conditions change.
A strategy qualified six months ago under different market conditions may no longer deserve that status. Without continuous re-qualification, the system accumulates stale trust that does not reflect current reality.
Evidence Over Opinion
Qualification gates accept structured evidence, never confidence, narrative, or subjective assessment.
Opinions are cheap and abundant — especially from AI systems that can generate compelling narratives for any position. Evidence is costly and scarce. The qualification boundary must be crossed with evidence, which is the only currency that resists inflation.
Fail-Closed Default
When evidence is insufficient, ambiguous, or missing, the default state is unqualified. Qualification must be actively earned, never assumed.
In domains where the cost of unqualified execution exceeds the cost of delayed execution, the safe default is to withhold trust. This principle prevents the system from drifting toward permissiveness as volume increases.
System Map
Four layers from raw observation to managed qualification status. Evidence flows upward from the observation layer. Assessment consumes evidence and produces gate evaluations. Gate evaluation determines status. Status management handles lifecycle transitions, re-qualification, and confidence decay.
Layers & Components
Each layer has distinct responsibilities, inputs, outputs, and ownership. No component spans multiple layers.
Evidence Accumulation
Gathers raw observations from validation, simulation, and execution into structured evidence that qualification gates can consume.
Validation Evidence
Captures historical performance evidence from backtesting
Simulation Evidence
Captures forward-looking behavioral evidence from paper trading
Environmental Evidence
Captures regime context and environmental conditions relevant to qualification
Multi-Dimensional Assessment
Evaluates evidence across independent dimensions. Each dimension produces a pass/fail judgment with supporting metrics.
Cohort Analysis
Compares backtest and paper performance to detect divergence
Regime Proof
Evaluates whether a strategy has demonstrated performance in its claimed regime
Duration and Volume Gates
Ensures sufficient observation before qualification decisions
Gate Evaluation
Applies conjunctive logic across all assessment dimensions. Produces qualification status with full evidence trail.
Conjunctive Gate Engine
Evaluates all dimensions simultaneously — all must pass for qualification
Conditional Qualification
Handles partial qualification when some but not all dimensions pass
Status Management
Manages the lifecycle of qualification status — transitions, re-qualification, confidence decay, and disqualification.
Status Lifecycle Manager
Governs transitions between qualification states based on evidence changes
Confidence Decay
Reduces qualification confidence when fresh evidence is not accumulating
States & Transitions
A decision moves through these states as it accumulates evidence. Each transition is gated — there are no free promotions.
Unqualified
Default state. Insufficient evidence to justify execution. No gates have been evaluated or all have failed.
Entry: Initial state for all candidates, or re-entry after disqualification
Exits: Evidence accumulates, some gates pass -> Conditionally Qualified
Conditionally Qualified
Some qualification dimensions pass but not all. Evidence is accumulating in a positive direction.
Entry: Core performance gates pass, but regime proof or duration gates remain open
Exits: All gates pass -> Qualified · Core gates fail -> Disqualified · Evidence stales -> Unqualified
Qualified
All conjunctive gates pass. Strategy has earned the right to execute with real resources.
Entry: All dimensions pass simultaneously in a single evaluation
Exits: Evidence degrades -> Conditionally Qualified · Severe failure -> Disqualified
Disqualified
Active determination that the strategy should not execute. Distinct from unqualified — disqualification implies evidence against, not just absence of evidence for.
Entry: Severe divergence, catastrophic paper performance, or behavioral anomaly detected
Exits: Significant new evidence after cooling period -> Unqualified (restart)
| From | To | Trigger | Guard |
|---|---|---|---|
| Unqualified | Conditionally Qualified | Partial gate passage | At least core performance and duration gates pass |
| Conditionally Qualified | Qualified | 6h evaluation batch | All conjunctive gates pass simultaneously |
| Qualified | Conditionally Qualified | Re-qualification evaluation | One or more non-critical gates fail, core gates still pass |
| Qualified | Disqualified | Severe evidence failure | Catastrophic divergence, negative regime proof, or behavioral anomaly |
| Conditionally Qualified | Disqualified | Core gate failure | Performance or divergence gates fail |
| Disqualified | Unqualified | Cooling period expires with new evidence | Minimum 14-day cooling period, new validation results |
Boundary Rules
Boundaries define what crosses between layers and what does not. Every boundary has an explicit failure mode.
Evidence Accumulation -> Assessment
Structured evidence records with timestamps, regime labels, and provenance
Raw market data, intermediate calculations, execution engine internals
Missing evidence -> assessment dimension returns 'insufficient data', which fails the conjunctive gate.
Assessment -> Gate Evaluation
Per-dimension pass/fail with supporting metrics and confidence levels
Assessment methodology details, threshold tuning rationale, intermediate scores
Ambiguous assessment -> treated as fail (fail-closed default).
Gate Evaluation -> Status Management
Qualification decision (qualified/conditional/disqualified) with full evidence trail
Gate engine internals, evaluation history for other strategies
Gate evaluation failure -> status unchanged (conservative). No silent status changes.
Status Management -> External Systems (Allocation, UI)
Current qualification status, conditions for full qualification, evidence summary
Internal status transition history, decay computation details, gate evaluation logs
Status query failure -> external systems treat as unqualified (fail-closed propagation).
Operational Semantics
How this framework operates in practice — cadence, consumers, and staleness rules.
| Dimension | Value | Rationale |
|---|---|---|
| Qualification assessment cadence | Every 6 hours (batch) | Balances evidence freshness against compute cost. Qualification is not latency-sensitive — strategies trade on minute cadence, but qualification evolves over days. |
| Confidence decay half-life | 14 days without new trades | A strategy that stops trading loses qualification confidence. After 14 days of inactivity, confidence is materially degraded. |
| Minimum observation period | 7 days of paper trading | Shorter periods produce unreliable behavioral characterization. 7 days provides a minimum window for behavioral signal accumulation; confidence increases with longer observation. |
| Minimum trade count | Domain-specific (trading: 3+ trades for regime proof) | Statistical significance requires volume. Three trades is the minimum for any per-regime assessment. |
| Re-qualification trigger | Regime transition, 6h batch, or significant outcome event | Qualification must respond to environmental changes, not just scheduled evaluation. |
| Disqualification cooling period | 14 days minimum | Prevents oscillation between qualified and disqualified states. Forces meaningful new evidence before re-entry. |
Knowledge Lineage
Depends On
Introduced In
Expanded In
Operationalized By
Investigated In
Influences
Related Research
Research Areas
Methodologies
See this framework implemented in Orqis