AI Intelligence

Model performance on controlled practical tasks.

1,224 task results
3 model families
17 historical configurations

How AIQ is scored

Compare task performance, read the uncertainty, and trace each score to its evidence.

72 controlled tasks compare model and reasoning-effort configurations. Results describe this task set; they are not a population IQ score.

Ability · 0–100: Calibrated performance on the committed task set. Higher is better.
Strict pass · %: Fully correct outcomes as a share of tasks with a semantic score.
Intervals · conditional 95%: Ability uncertainty with the item bank fixed. Point order alone does not establish a winner.

Current Core definition. Historical records retain their original task and scoring versions.

What is measured

Fixed task scope, calibrated ability, and evidence commitments

Benchmarkaiq-core@1.1.0

Scoring1.0.8

Task scorer1.0.6

Frozen Core task catalogue

Observable outcomes on the committed set

Estimate a one-dimensional latent ability on the committed AIQ Core fixed-fixture set. Do not claim general intelligence or a population IQ norm.
For non-synthetic Official evidence, jointly estimate model locations and one Rasch task difficulty per task from the complete 17-configuration matrix with weak priors on a centered item scale. The published theta interval is conditional on that released item bank.
Publish 100 × logistic(theta) as the calibrated average-task ability score. Keep the raw equal-domain fixed-fixture mean as a separate quality diagnostic.
Strict pass rate is strict successes divided by every attributable task with a semantic task score. Partial scores are non-passes in the denominator; missing, infrastructure-invalid, and unscored tasks are excluded. Wilson bounds use the same denominator.
Keep hidden fixture payloads behind the commitment boundary. Publish versions, counts, outcomes, and provenance.
Do not fabricate or substitute work when a required capability is unavailable.

Domain coverage

72 tasks · 10 equally weighted domains

Exact fixed-fixture domain task counts and macro-average weights.
Domain	Tasks design	Weight %
coding	8	10%
data processing	8	10%
debugging	8	10%
documentation communication	7	10%
instruction following	6	10%
planning execution	7	10%
reliability recovery	7	10%
repository understanding	7	10%
retrieval verification	7	10%
tool use	7	10%
Total	72	100%

Completeness and validity

Missing results, attempted failures, and Official publication requirements

Official, complete synthetic, provisional, or coverage-only

Attempted failure: Only completed semantic correct, partial, and incorrect outcomes contribute a task score. Runtime failures have no semantic score and remain visible as runtime issues. Invalid infrastructure attempts are audited and rerun. A whole configuration can be N/A only when preflight proves it unavailable.
Missing fixture or result: Missing and invalid results block Official and remain in completion accounting. A complete synthetic fixture is descriptive, never Official, and not ranked. A Provisional point estimate averages valid observed tasks within each domain; fixed-fixture completion bounds retain every planned task and assign unobserved tasks zero or one. Provisional requires at least 60 results and at least 4 in every domain, and is not ranked.
Hidden fixture boundary: Hidden payloads stay sealed behind the published fixture-set commitment. Version, task counts, outcome states, and provenance remain public.

Task-level score

Weighted checks, partial credit, hard gates, and runtime failures

Committed weighted checks with explicit hard gates

Each task uses at most 16 binary checks from a content-addressed private evaluator configuration. Check names and weights remain hidden with the task payload, but their exact identities are committed and the verifier replays them.

Valid evaluator resulthard gate or structural failure ? 0 : Σ(weight × passed) ÷ Σ(weight)

Partial credit: A passing check contributes its committed nonnegative integer weight. A valid positive denominator is required. The evaluator does not round before exact replay.
Hard gates: An explicit hard gate or workspace-policy check can force zero. A positive-weight hard gate also participates in the weighted fraction when every gate passes. Only a hard gate may have zero weight.
Outcome labels: 1 is Correct; a score between 0 and 1 is Partial; 0 is Incorrect.
Execution evidence: Runtime failures carry no semantic score and remain visible with an explicit runtime status. Evaluator or infrastructure-invalid evidence also stays unscored and blocks Official publication; neither state is converted into semantic Incorrect.

Uncertainty and task mix

Ability intervals, raw quality sensitivity, and interpretation limits

Two intervals with different meanings

Official calibrated ability uses a conditional 95% interval derived from latent standard error while the released item bank stays fixed. The raw quality diagnostic has a separate task-mix sensitivity interval. Neither interval includes every source of benchmark or deployment uncertainty.

The canonical fixed-fixture sensitivity interval draws committed clusters with replacement inside each domain, includes all task scores in each drawn cluster, and recomputes the raw equal-domain diagnostic. It expands each raw 95-percentile deviation from the observed fixture score by the versioned 1.3 correction and clamps the endpoints to 0 through 100. It is not a universal confidence interval for model capability. Latent standard error is conditional on the frozen calibration bank.

Official calibrated ability100 × logistic(theta); raw domain mean is qualityScore

Time and cost

Observed adapter time and estimated API-equivalent USD

Observed Codex adapter elapsed time and estimated token cost stay distinct

Time is observed Codex adapter elapsed time. Cost is a versioned estimate from covered token aggregate usage and the official OpenAI API pricing documentation, accessed 2026-08-02. It is not actual Codex subscription billing or necessarily an exact API invoice.

Source / as of / estimator version: Official OpenAI API pricing · USD · standard processing tier · 2026-08-02 · aiq.standard-api-equivalent-usd.v1
Formula: Uncached input = total input − cache-read input − cache-write input. Multiply uncached input, cache-read input, cache-write input, and output by their versioned standard per-token rates, then sum them. Reasoning tokens are nested within output and are not added twice.
Coverage: Raw token counters are provider-reported. The verifier recomputes aggregates and the exact cost estimate from those counters when token coverage is complete. Prompts above 272,000 input tokens use 2× input and 1.5× output rates. Aggregate counters cannot identify which requests crossed that threshold, so AIQ displays a published-rate range: all-standard rates as the lower bound and the full long-context uplift as the upper bound. The range is not an exact invoice. Missing, invalid, or JCS-overflowed aggregate usage remains unavailable, never zero.
Observed Codex adapter elapsed time: Sum, median, and p95 of Codex adapter elapsed time: model plus allowed tools. It excludes workspace setup, artifact sealing, and evaluator replay. Runtime-issue tasks consume time; missing or non-invoked cells do not. Full-matrix timings are operational resource-profile evidence under the recorded node, execution order, and concurrency shown with each run. Model, tool, network, and local contention vary. This is not pure task latency or an isolated API-frontier latency test. Cell durations can overlap under concurrency. Signed matrix-stage start and finish times provide the full batch wall-clock, which is counted once across the 17 configurations. TTFT and TPS are unavailable and are not inferred.
Interpretation: AIQ, observed adapter elapsed time, and estimated API-equivalent USD remain separate. Scatter and Pareto context do not create a combined ranking.