Research Area
Qualification Systems
Qualification systems determine whether an idea, candidate, or artifact has accumulated sufficient evidence to justify real-world execution. They sit between generation and deployment — the gatekeepers of Decision Infrastructure. Every domain that converts ideas into actions implicitly or explicitly operates a qualification system, whether it is a clinical trial protocol, a manufacturing IQ/OQ/PQ sequence, or a CI/CD pipeline. The common thread is staged confidence accumulation: raw candidates enter, structured evidence is gathered, and only those meeting defined thresholds proceed to execution.
Research Thesis
Qualification systems determine whether an idea, candidate, or artifact has accumulated sufficient evidence to justify real-world execution. They sit between generation and deployment — the gatekeepers of Decision Infrastructure. Every domain that converts ideas into actions implicitly or explicitly operates a qualification system, whether it is a clinical trial protocol, a manufacturing IQ/OQ/PQ sequence, or a CI/CD pipeline. The common thread is staged confidence accumulation: raw candidates enter, structured evidence is gathered, and only those meeting defined thresholds proceed to execution.
Qualification has historically been a human-managed bottleneck. Domain experts review candidates, committees vote on readiness, and institutional knowledge determines what 'good enough' looks like. This worked when candidate volumes were low and the cost of evaluation was proportional to the cost of generation. AI has shattered that proportionality. When a system can generate hundreds of strategy candidates per day, the qualification layer — not the generation layer — becomes the binding constraint. Without systematic qualification, organizations drown in unvetted optionality.
The research question is whether qualification can be formalized as reusable infrastructure rather than ad-hoc domain expertise. If the core pattern — staged evaluation, conditional trust, evidence decay, regime-conditional validity — generalizes across domains, then qualification systems become a design discipline rather than a collection of domain-specific checklists. Orqis is the first implementation of this thesis in capital allocation: a 14-criterion, regime-conditional, staged qualification evaluator that runs as a pure function on assembled evidence.
Evolution of the Discipline
Understanding how this problem has been approached across eras reveals both recurring patterns and persistent gaps.
Pre-Modern Guilds
~1200-1700No formal methodology for assessing competence at scale
Master-apprentice observation over years; guild examinations; masterwork demonstrations
Qualification was inseparable from the qualifier — subjective, slow, non-transferable between domains
Scientific Revolution
1600-1800No systematic method for distinguishing true claims from plausible ones
Hypothesis testing, reproducibility requirements, peer review, controlled experiments
Applied primarily to knowledge claims, not to operational readiness — knowing something works is different from qualifying it for deployment
Industrial Quality Assurance
1900-1970Manufacturing scale outpaced human inspection capacity
Statistical process control (Shewhart), IQ/OQ/PQ protocols, acceptance sampling, Six Sigma
Optimized for physical goods with stable specifications — struggled with adaptive systems where the environment changes the definition of 'qualified'
Software Testing and CI/CD
1970-2015Software changes too fast for manual qualification cycles
Automated test suites, staged deployment (dev/staging/production), canary releases, feature flags
Qualification criteria are author-defined — tests verify what developers think matters, not necessarily what matters in production. Goodhart's Law applies: teams optimize for test passage, not deployment readiness.
AI Evaluation Benchmarks
2015-presentAI systems generate candidates faster than humans can evaluate them
Benchmark suites, A/B testing, shadow deployment, red-teaming, RLHF alignment evaluation
Benchmarks measure capability, not deployment readiness. A model that scores well on benchmarks can still fail catastrophically in production. The gap between evaluation and qualification remains open.
Decision Infrastructure
2024-presentAI-generated candidate abundance makes qualification the binding constraint
Staged qualification with conditional trust, regime-aware validity, evidence decay, pure-function evaluators operating on assembled evidence snapshots
Early stage — limited cross-domain validation. Risk of over-fitting qualification criteria to the first implementation domain.
Landscape Review
How different domains approach this problem today — their assumptions, strengths, weaknesses, and open questions.
Medicine / Clinical Trials
Candidate therapies must prove safety and efficacy through phased trials before deployment to patients
Most rigorous qualification system in any domain; staged phases (I-IV) with clear criteria; regulatory enforcement; post-market surveillance
Extremely slow (10-15 years typical); expensive ($1B+ per drug); binary pass/fail ignores conditional or regime-specific efficacy; Phase III failures destroy value accumulated in Phase I-II
Open question: Can adaptive trial designs and real-world evidence reduce qualification time without compromising safety?
Manufacturing / Quality Assurance
Equipment and processes must demonstrate consistent performance within specified tolerances before production use
IQ/OQ/PQ provides a clear staged sequence; statistical process control enables continuous monitoring post-qualification; well-understood failure modes
Designed for stable environments — requalification triggers are poorly defined when specifications change; manual documentation overhead is high
Open question: How should qualification adapt when the product specification itself evolves continuously (as in software or AI-generated artifacts)?
Software / CI-CD Pipelines
Code changes that pass automated tests and staged rollouts are safe for production deployment
Fast feedback loops; automated execution; staged rollout (canary, blue-green); rollback capability; feature flags for conditional qualification
Test coverage is self-referential — tests verify what developers anticipated, not what they missed. Staging environments rarely replicate production conditions. Qualification criteria drift from actual risk factors over time.
Open question: Can production telemetry be used to automatically derive and update qualification criteria rather than relying on developer-authored tests?
Finance / Risk Committees
Trading strategies and investment decisions require human committee approval before capital deployment
Multi-stakeholder review catches individual blind spots; institutional memory of past failures informs current decisions
Committee-based qualification does not scale with AI-generated candidate volume; anchoring bias and groupthink; approval latency mismatched to market speed; criteria are implicit in committee members' heads, not codified
Open question: Can structured qualification systems replace or augment committee-based approval without losing the institutional knowledge that committees carry?
AI / Evaluation Benchmarks
Model performance on standardized benchmarks predicts real-world deployment readiness
Enables cross-model comparison; reproducible; community-maintained benchmark suites create shared evaluation standards
Benchmark saturation (models optimize for benchmarks, not tasks); benchmarks measure capability, not safety or reliability; no regime-conditional evaluation (a model qualified on English benchmarks may fail on low-resource languages)
Open question: How do you qualify an AI system for deployment when the deployment environment is open-ended and cannot be fully represented by any benchmark suite?
Core Mental Models
Reusable frameworks for thinking about this research area.
The Qualification Funnel
Candidates enter wide and exit narrow. Each stage adds evidence requirements, and each stage filters more aggressively. The funnel shape is not incidental — it reflects the economics of evaluation. Cheap screens run first (syntax, basic thresholds), expensive screens run last (paper trading, regime-conditional proof). Inverting the funnel wastes evaluation budget on candidates that would have been caught by cheap screens.
The Evidence Gradient
Qualification confidence is not binary. It exists on a gradient from 'no evidence' through 'suggestive' to 'proven under specific conditions' to 'proven across conditions.' Systems that treat qualification as binary (pass/fail) lose information. Staged qualification (unproven, globally qualified but regime-unproven, conditionally qualified) preserves the gradient and enables regime-conditional trust.
The Execution Threshold
The point on the evidence gradient where confidence justifies real-world execution. This threshold is not fixed — it depends on the cost of failure, the reversibility of the action, and the opportunity cost of waiting for more evidence. Low-cost reversible actions warrant lower thresholds; irreversible high-stakes actions warrant higher thresholds. The threshold itself is a design parameter, not a discovered constant.
Confidence Accumulation and Decay
Qualification confidence accumulates through evidence and decays without it. A strategy qualified six months ago with no new trades provides less confidence than one qualified yesterday with recent data. Staleness detection (Orqis uses a 12-hour window) operationalizes decay. The open question is the functional form of decay — linear, exponential, or step-function at domain-specific boundaries.
False Negative Cost Asymmetry
In most qualification systems, the cost of false positives (qualifying something that should not have been qualified) exceeds the cost of false negatives (rejecting something that would have succeeded). This asymmetry drives conservative thresholds. But excessive conservatism creates its own cost: missed opportunity, wasted evaluation resources on re-qualification, and selection bias in the surviving population. The optimal threshold minimizes total cost across both error types.
Qualification Debt
Analogous to technical debt. When qualification criteria are loosened for speed, skipped for convenience, or not updated as the environment changes, qualification debt accumulates. The debt manifests as unqualified candidates reaching execution, qualified candidates losing their edge without detection, or criteria that no longer measure what they were designed to measure. Interest compounds: each unqualified execution that succeeds makes it harder to justify tightening criteria later.
Canonical Questions
The research questions that define this area. These are not rhetorical — they represent genuine uncertainties that guide investigation.
What is the minimum evidence set required to qualify a decision candidate for real-world execution, and how does this set vary by domain and risk profile?
How should qualification criteria adapt when the environment changes — should thresholds be fixed, learned from outcomes, or dynamically adjusted by regime?
What is the optimal balance between false positive rate (qualifying bad candidates) and false negative rate (rejecting good candidates) across different cost structures?
Can qualification systems be learned from historical qualification-outcome pairs, or do they require expert-designed criteria?
How should qualification confidence decay over time in the absence of new evidence, and what should trigger requalification?
Does regime-conditional qualification (qualified in environment A but not B) generalize beyond financial markets to other domains with changing operating conditions?
What is the relationship between generation quality and qualification stringency — does better generation reduce the need for qualification, or does it shift qualification to different criteria?
How should qualification systems handle the exploration-exploitation tradeoff — reserving capacity for unqualified candidates that might prove valuable if given execution access?
Working Hypotheses
Not conclusions — working hypotheses. Each includes our current confidence level and the evidence or counterarguments we are aware of.
Qualification quality predicts long-term execution outcomes better than generation quality.
Orqis data shows that strategies passing all 14 qualification criteria significantly outperform strategies with strong backtest scores but incomplete paper trading evidence. Generation quality (backtest composite) is necessary but not sufficient.
Counterargument: Survivorship bias — qualification criteria were designed to select for the metrics being measured. Need forward-return data anchored to qualification timestamp to fully validate.
Regime-conditional qualification produces better outcomes than unconditional qualification because it prevents deployment in environments where the candidate has no evidence.
Strategies qualified in ranging markets that are deployed during volatile regimes show measurably worse performance. Per-regime qualification gates catch this; unconditional gates do not.
Counterargument: Regime classification itself may be noisy, causing regime-conditional qualification to reject candidates that would have performed adequately. The cure could be worse than the disease if regime labels are unreliable.
Pure-function qualification evaluators (no side effects, deterministic on inputs) produce more reliable and auditable outcomes than stateful or interactive qualification processes.
Orqis qualification evaluator is a pure function with 100+ unit tests covering every criterion and state transition. This architecture enables full audit trails, reproducible verdicts, and zero dependency on evaluation-time side effects.
Counterargument: Pure functions require all evidence to be assembled by the caller, which shifts complexity to the orchestration layer. If the caller assembles stale or incomplete evidence, the pure evaluator produces a confidently wrong verdict.
Qualification criteria should be asymmetric between entry and exit — harder to qualify than to disqualify — to prevent oscillation at the threshold boundary.
Orqis implements anti-oscillation guards between auto-pause and auto-resume. Asymmetric thresholds reduce cycling. However, excessive asymmetry means strategies that temporarily degrade are slow to recover even when performance rebounds.
Counterargument: Symmetric thresholds are simpler to reason about and audit. Asymmetry introduces implicit state (how the candidate arrived at its current status) which complicates the qualification evaluator.
Evidence & References
External References
W. Edwards Deming (1986). Out of the Crisis. MIT Press.
Foundational work on statistical process control and the idea that quality is built into processes, not inspected into products — directly applicable to qualification system design.
FDA (2011). Process Validation: General Principles and Practices. FDA Guidance for Industry.
Canonical IQ/OQ/PQ framework for pharmaceutical manufacturing qualification — the most rigorous staged qualification system in any industry.
Marcos Lopez de Prado (2018). Advances in Financial Machine Learning. Wiley.
Addresses the problem of strategy overfitting and the need for combinatorial purged cross-validation — a form of qualification criteria for quantitative strategies.
David Manheim & Scott Garrabrant (2019). Categorizing Variants of Goodhart's Law. arXiv.
Taxonomy of how optimization targets diverge from true objectives — directly relevant to the risk that qualification criteria become gamed or stale.
Comparative Models
How different disciplines approach the same underlying problem. These are structural analogues — not proof of equivalence.
Clinical Trials (Phase I-IV)
Mechanism: Staged evidence accumulation: safety (Phase I) then efficacy (Phase II) then comparative effectiveness (Phase III) then post-market surveillance (Phase IV)
Failure contained: Small cohorts in early phases limit harm; Data Safety Monitoring Boards can halt trials; regulatory rejection prevents market access
Transfers: Staged qualification states (unproven, globally qualified, conditionally qualified) mirror clinical trial phases. Post-deployment monitoring mirrors Phase IV.
Software CI/CD Pipelines
Mechanism: Automated test gates (unit, integration, e2e) plus staged rollout (canary, blue-green) with automated rollback on failure metrics
Failure contained: Canary releases limit blast radius; feature flags enable instant rollback; automated monitoring triggers alerts
Transfers: Paper trading as staging environment; live deployment as production; regime gate as canary metric; auto-pause as automated rollback.
Aviation Certification (DO-178C)
Mechanism: Design assurance levels (A-E) with increasing rigor for more critical systems; traceability from requirements through verification
Failure contained: Multiple redundant systems; certification level matched to failure consequence; post-certification airworthiness directives
Transfers: Risk-proportional qualification thresholds — higher-stakes deployments (larger capital, leveraged positions) could require stricter criteria.
Pharmaceutical IQ/OQ/PQ
Mechanism: Installation Qualification (correct setup), Operational Qualification (performs as designed), Performance Qualification (produces acceptable results under real conditions)
Failure contained: Each phase gates the next; failed OQ prevents PQ; deviation reports trigger investigation before proceeding
Transfers: Generation validation (IQ: spec is structurally valid), backtest validation (OQ: performs in historical data), paper trading (PQ: performs in live market conditions).
Hiring and Credentialing
Mechanism: Resume screen, technical assessment, behavioral interviews, reference checks, probationary period with performance review
Failure contained: Probation period limits commitment; progressive responsibility; performance improvement plans before termination
Transfers: Progressive trust: paper trading as probation, qualification as passing performance review, live deployment as full employment, degradation as performance improvement plan.
Open Problems
Unsolved questions that define the frontier of this research area.
Threshold calibration from outcomes: Current qualification thresholds (45% win rate, 7-day minimum, etc.) are expert-set. Can historical qualification-outcome pairs be used to learn optimal thresholds, and would learned thresholds generalize across market regimes?
Qualification under distribution shift: A strategy qualified in one market regime may encounter a regime never seen during qualification. How should qualification systems handle genuinely novel environments where no historical evidence exists — default to caution, allow limited exploration, or attempt online adaptation?
Multi-candidate portfolio qualification: Current systems qualify candidates independently. But a portfolio of five independently-qualified strategies may have correlated failure modes that individual qualification missed. How should qualification account for portfolio-level interactions without requiring combinatorial evaluation of all possible portfolios?
Evidence decay functional form: Qualification confidence should decay without new evidence, but the optimal decay rate is unknown. Should it be linear (simple but arbitrary), exponential (matches Bayesian updating), or step-function (reflecting domain-specific 'freshness' boundaries)? Different domains likely require different decay profiles.
Adversarial robustness of qualification criteria: If qualification criteria are known (as they are in Orqis), can generation systems learn to produce candidates that satisfy criteria without genuinely earning qualification? This is Goodhart's Law applied to qualification — the criteria become the target rather than the proxy they were designed to be.
Cross-domain qualification transfer: Can a strategy qualified in one asset class or market (e.g., BTC perpetuals) receive partial credit toward qualification in a related asset class (e.g., ETH perpetuals)? Transfer qualification would accelerate evidence accumulation but risks false confidence from superficial similarity.
Implications
Researchers
Qualification systems are an under-theorized area at the intersection of decision science, quality engineering, and AI safety. The core research questions — optimal thresholds, evidence decay, regime-conditional validity, adversarial robustness — have general solutions that would advance multiple fields. The formalization of qualification as reusable infrastructure (rather than domain-specific checklists) is itself a research contribution.
Engineers
Building qualification as a pure-function evaluator with assembled evidence snapshots produces systems that are testable, auditable, and reproducible. The pattern — separate evidence assembly from evaluation logic, persist evaluation snapshots, implement staged states rather than binary pass/fail — transfers directly to any system that gates deployment on evidence. This is the CI/CD pattern applied to decision quality.
Organizations
As AI tools increase the volume of candidates (strategies, product ideas, hiring shortlists, research directions), organizations that lack systematic qualification infrastructure will face a growing gap between generation capacity and deployment confidence. The cost of qualification debt compounds: each unqualified deployment that succeeds by luck makes it harder to justify systematic evaluation later.
AI Companies
AI systems that generate candidates (code, strategies, content, designs) need qualification layers between generation and deployment. Current practice delegates qualification to human review, which does not scale with generation volume. Companies that build qualification infrastructure into their AI systems — not just evaluation benchmarks, but staged, conditional, evidence-based deployment gates — will have a structural advantage in trust and reliability.
Related Research
Methodologies
See these ideas implemented in Orqis