Archive extraction containmentPrevent an archive member from writing outside its extraction directory.
Bug repairGo
Medium
Cache adapter separationExtract cache storage behind a small adapter and retain existing callers.
Refactor and migrationTypeScript
Basic
Cache expiration boundaryExpire cache entries at the exact deadline and remove stale reads.
Bug repairTypeScript
Basic
Catalog cursor stabilityStop cursor pagination from dropping records with equal timestamps.
Bug repairTypeScript
Medium
Catalog filter parityAdd identical status and minimum-price filters to the API and CLI.
Feature workTypeScript
Medium
Configuration precedenceFix defaults, file, and environment precedence without treating zero as missing.
Bug repairGo
Basic
Configuration schema upgradeRead legacy flat configuration and write a canonical versioned format.
Refactor and migrationGo
Medium
Dataset import transactionImplement an atomic import and prove that replay creates no extra rows.
Local workflowPython
Medium
Dataset import validationValidate CSV imports and report bad rows without writing to the database.
Feature workPython
Basic
Export field selectionAdd ordered column selection to the existing CSV exporter.
Feature workTypeScript
Medium
Export resume and deduplicationResume an interrupted export without omitting or duplicating records.
Local workflowTypeScript
Hard
Incremental job executionAdd resumable processing with durable job identity and conflict detection.
Feature workGo
Hard
Job recovery ledgerRepair a supplied job ledger and execute only unfinished local jobs.
Local workflowGo
Medium
Ledger reconciliationReconcile three local data sources into a stable ledger report.
Local workflowPython
Basic
Ledger refund accountingCorrect partial-refund totals without counting duplicate events twice.
Bug repairPython
Medium
Log event normalizationMove text and JSON log readers onto one canonical event representation.
Refactor and migrationGo
Hard
Log rotation aggregationAggregate rotated logs with deterministic deduplication and error reporting.
Local workflowGo
Medium
Manifest path containmentKeep manifest scanning inside the chosen root and stop symlink loops.
Bug repairPython
Hard
Manifest serialization migrationRead the legacy manifest and write a deterministic versioned schema.
Refactor and migrationPython
Medium
Report date filteringAdd a UTC date interval filter without changing report aggregation.
Feature workPython
Medium
Report pipeline extractionSeparate loading, calculation, and rendering while preserving both entry points.
Refactor and migrationPython
Hard
Settings merge provenanceMerge three settings layers and retain the origin of every final value.
Local workflowTypeScript
Hard
Settings schema migrationUpgrade flat settings while keeping old input and CLI behavior compatible.
Refactor and migrationTypeScript
Medium
Task design and grading
Basic · 6 tasks
One local behavior or interface, usually one or two relevant modules, with explicit examples and limited state.
Medium · 12 tasks
Two to four relevant modules or interacting rules; requires boundary checks, compatibility, or consistency across outputs.
Hard · 6 tasks
Three to six relevant modules with coupled state, recovery, or migration invariants. The work remains bounded and uses no obscure domain knowledge.
Acceptance
Each specification has exactly three binary requirement gates. Each gate passes only when all its frozen assertions pass. Primary success requires all three gates, normal completion, and all common limits. Report passed gates / 3 as a separate diagnostic, averaged equally per task; do not rank by the number of assertions or blend partial credit, cost, and success.
Difficulty
Difficulty is an author estimate of coordination demand, not a measured pass rate or a promised completion time. All levels share the same 20-minute and USD 2 attempt ceilings. Use separate development tasks to review difficulty after a later execution decision. Never select formal tasks from the observed Codex/Hermes/Pi winner or widen a score gap after seeing results.
Package contract
Each package will include the task prompt, a clean workspace snapshot, visible examples, independent hidden checks, a reference patch or procedure, and a manifest with hashes. Keep references and hidden checks outside agent access. Preinstall dependencies. Use fixed clocks, deterministic ordering, integer minor currency units, and local services where specified. No web account, browser, live service, GPU, or multi-agent dependency is needed.
Same-model cohort
Model and route
gpt-6-sol · medium · OpenAI direct API · Responses / HTTP + SSE · Standard
Planned coverage
24 task specifications · 12 planned workspaces · 3 repeats · 216 planned attempts. Task specifications are authored; workspace packages and graders are not built or evaluated.
Shared attempt ceilings
20 minutes · USD 2 reference inference cost · 128,000 input and 16,000 output tokens / request. Compression and retries consume the same budget.
Use a provider-pinned revision if available. The reviewed model page lists gpt-6-sol without a dated snapshot. Preserve requested and returned model IDs and any provider revision metadata; do not invent a revision. Without an immutable binding, label results as a time-bounded matched-route cohort, not proof of identical weights.
This is a controlled common-capability profile, not a test of every product default. A custom-provider path or reduced context ceiling can change native behavior; conclusions apply to the frozen profiles only.
Model × harness
Which model–harness combination completes these tasks within the recorded limits?
Deferred in V1; no execution or budget is allocated. Future cohorts compare complete, explicitly pinned model–harness combinations. A difference belongs to the complete combination. It cannot be attributed to the harness alone.
Same model
What changes when the same model uses a different harness?
Same model, provider route, tasks, and limits. Native harness instructions and context handling remain part of the comparison. Evidence is conditional on this model and task cohort. It does not establish a model-independent harness winner.
Task families in scope
Task family
Tasksplanned
Acceptancerequired evidence
Details
Bug repair
6
Fix the stated defect and pass all three task gates, including boundary and regression cases. A plausible patch or completion claim is insufficient.
Feature work
6
Implement the stated interface and behavior; pass all three gates for requirements, errors, and compatibility. Partial implementation is diagnostic only.
Local workflow
6
Produce the specified files or local database state. Pass all three gates for selection, consistency, and safe repeat or recovery behavior.
Refactor and migration
6
Change the stated boundary or format, update its consumers, and pass all three gates for compatibility, preservation, and the specified new contract.
Task selection
Cohort structure
Use 12 bounded project workspaces, four per stack: Go, Python, and TypeScript. Each workspace has two independent starting snapshots for two different tasks; each stack has two tasks per family. The 12 entries below are authored project specifications, not selected public repositories or implemented fixtures. Reuse three stack images. Keep six development tasks in separate workspace families and outside the score.
Exclusions
Exclude Core/Mem evidence, public benchmark answer packs, private archives, external accounts, live web facts, paid services, GPU workloads, subjective visual judging, and specialist puzzles. Do not remove a formal task because a target harness performs poorly or the observed score gap is small.
Independent validation
Future package review must match every scored requirement to the task prompt, accept the reference implementation, reject the unmodified baseline and a plausible incomplete implementation, and repeat deterministic grader checks five times. Test behavior rather than an arbitrary implementation. Freeze prompts, fixtures, graders, and analysis before any main attempt. These checks have not run.
Source requirements
Build small multi-file workspaces from realistic CLI, library, data, and local-service workflows. Keep the relevant behavior in roughly two to six modules. If public code is used, record its license, URL, immutable commit, and overlap with public issues/benchmarks. New task authorship is not proof of no training exposure. No existing external repository or task package is claimed by these specifications.
Qualification and controls
Same-model qualification · not run
Codex
Use an isolated CLI profile with model=gpt-6-sol, model_reasoning_effort=medium, a named provider, and wire_api=responses. Qualify HTTP/SSE, the model catalog, effective limits, native instructions, and disabled optional services for the pinned release.
Hermes
Pin model.default, model.base_url, api_mode=codex_responses, and agent.reasoning_effort. Audit compression separately. Disable optional auxiliary slots and all fallback chains; reject an omitted or remapped reasoning field. Exact release behavior remains unqualified.
Pi
Use an isolated Pi profile and explicit model ID, baseUrl, and api=openai-responses. Audit effective reasoning mappings, context/output limits, native tools, package loading, and compaction calls for the pinned release.
All three pinned releases must pass the same four checks before the three-way cohort starts. A client that cannot meet the common wire, limits, or accounting contract is ineligible; do not substitute Chat Completions, OAuth/subscription routing, another model, or an unrecorded adapter.
Cancellation and failure
On a disposable fixture, capture controlled provider failure, bounded retry, cancellation, process-tree termination, budget reservation, and no model/effort fallback. Count all upstream calls.
Continuation and compression
Exercise a multi-turn replay and required native compaction. Check that tool IDs, opaque reasoning items, model/effort, usage, and final completion survive; no auxiliary route escapes the ledger.
Identity and parameters
Capture actual requests and returned model/tier metadata through the final route. Verify task/instruction loading, configured context and output ceilings, sampling omission, and effective medium reasoning.
Tool loop
Complete a read/edit/shell/function-result/final-answer loop on a disposable fixture. A /models listing or a text-only response is insufficient.
Shared baseline
Auxiliary inference
Disable optional title generation, background review, delegation, memory learning, and skill lookup. Required in-session compression must use the same model, endpoint, effort, and budget. No hidden model or parameter fallback.
Fresh state
Use a fresh checkout, HOME/profile, session, and writable temporary directory per attempt. No user configuration, prior session, cross-task memory, third-party skills/plugins, or resumed failed task. Record all bundled tools and disabled features.
Instructions and tools
Give each harness the same task prompt, repository instructions, files, and local capabilities: read/search/edit/shell. Retain and hash native prompts and tool schemas. Record automatic instruction loading. No harness-specific task hints, user intervention, browser, MCP, hosted tools, or subagents.
Model identity
Use the proposed exact model string, one upstream API project and processing tier, and one recording route. Verify actual main and auxiliary request fields, responses, and provider metadata. A label, model list, or successful text reply is insufficient.
Network and grading
Agent-side egress reaches only the study inference route. Offline dependencies and common Linux x86_64 image are prepared before the timer. Hidden tests, reference patches, grader secrets, host files, and provider credentials are outside agent access. Grade the frozen final state after termination.
Task and environment
Use identical commits, dependency/image digests, 4 vCPU, 8 GiB RAM, 10 GiB writable disk, and resource ceilings. Run one evaluated attempt at a time on one reserved worker. Keep setup and independent grading outside the agent timer and report them separately.
Proposed budget
Budget proposal · no spending authority
USD 534 inference + USD 50 compute = USD 584 maximum cash before tax. 16 reviewer hours are a separate labor cap.
Allocation
Workloadplanned
CeilingUSD proposed
Details
Compatibility qualification
12 fixture checks
12
Development tasks
18 unscored attempts
36
Infrastructure replacement
15 attempts in at most five matched replacement blocks
30
Main cohort
216 scored attempts
432
Reconciliation reserve
1 accounting buffer; no additional experiments
24
Budget admission
Before dispatch, reserve a validated upper bound for input and maximum possible output in the shared ledger. Include cached/replayed input, opaque continuation state, reasoning output, and any paid endpoint. Never guess token counts from character ratios. If a safe bound or final usage cannot be established, hold the full reservation and pause; an unenforceable budget is a qualification failure.
Reference cost
Budget reference cost is (all input tokens * 2.50 + all output tokens * 10) / 1,000,000 USD. Apply the same conservative input rate to cached and uncached input so prior cache warmth does not buy extra attempts. Do not add reasoning output twice. Record provider-billed and reference costs separately; this reference is not an invoice.
Scope and labor
Human fixture/task authoring, independent grader construction/review, and engineering labor are outside the cash ceiling and must be tracked separately. The 16-hour review cap covers result/evidence review, not task creation. No paid grader model or subscription inference is included. Local reserved compute adds no cloud charge.
Stop rules
Ceilings are proposals, not forecasts or authorization. Revalidate prices, tier, taxes, and billing behavior before any paid phase. The 24 USD reserve covers unresolved accounting only; it does not authorize more tasks, models, repeats, or reruns. If the API/compute cap, review-time cap, replacement limit, or seven-day cohort window cannot be met, stop and publish the incomplete state.
Worker time bound
249 capped main/development/replacement attempts imply at most 83 agent-worker hours plus 20.75 independent-grading hours; allow up to one further hour for the 12 qualification checks. These are planning maxima, not observations.
Measures and denominators
CostUSD / attempt; USD total
Publish reference cost, token-based billed estimate, and reconciled provider spend as distinct quantities. Include main, compression, retry, excluded, replacement, and cancelled calls. Report estimate/settled/unavailable status and reconciliation scope; missing usage is never zero.
Elapsed timeseconds / attempt
Measure from launch with the complete task to final termination, including model/tool waits and retries. Report median and nearest-rank p95 for all valid attempts and a separate successful-attempt slice. Report timeouts as cap-limited/censored, and setup/grading separately.
Failures and overheadcounts with denominators
Publish failure taxonomy, invalid and missing attempts, tool/model calls, transport retries, compactions, input/cache/reasoning/output tokens, and maximum context. Agent iteration counts differ across clients and are diagnostics only.
Requirements fulfilledpassed gates / 3 per task; task-balanced percent
Diagnostic only: each gate passes when all its checks pass. Average three equally weighted gates per task, then tasks equally; retain gate IDs and failures. More test assertions do not increase a task's weight. This is not the primary ranking score.
Primary independent acceptance under the common limits: all three requirement gates and normal completion. Show the 24-task, 12-workspace specification, actual coverage, raw counts, and separate outcomes for all four families. Difficulty is estimated, not measured.
Resample matched workspace-family clusters within stack, not individual calls, assertions, or repeated attempts. Publish three pairwise differences and the practical margin. These authored projects do not represent every real codebase or model revision.
Repeats and uncertainty
Repeats, failures, and uncertainty
Three fresh attempts per task and harness; use all outcomes, never best-of-three. The main plan contains 24 tasks x 3 harnesses x 3 repeats = 216 attempts, not 216 independent tasks.
Freeze a seeded schedule with 72 matched task/repeat triplets. Within each of the 12 stack-by-family cells, use each of the six harness-order permutations once; each permutation appears 12 times overall. Spread each task's repeats across the schedule and execute each triplet close in time on one reserved worker.
Require normal completion within all limits and all three independent binary requirement gates. Average the three repeats per task, two tasks per workspace, then 12 workspaces equally. Complete balanced coverage equals accepted/72 valid attempts per harness. Report all 24 tasks and each family separately.
Wrong work, malformed tool arguments, agent/harness crashes, unsupported requests caused by the harness, and exhausted task limits are unsuccessful valid attempts. Only independently evidenced provider/worker outages or a reference-grader failure are infrastructure-invalid. A protocol-identity violation stops the cohort; it is not silently repaired or scored as a semantic zero.
If one attempt is infrastructure-invalid, retain and exclude its whole matched triplet from the primary paired analysis, then repeat all three profiles from fresh state. Permit one replacement per original block and at most five replacement blocks total. If a replacement also fails or a sixth block needs replacement, stop; do not rank incomplete coverage. Preserve original and replacement links and every billed call.
Use 10,000 paired bootstrap draws over workspace-family IDs, stratified by stack (four workspaces per stack). Keep each sampled workspace's two tasks, repeats, and harnesses together. Publish marginal 95% score intervals and the three matched differences with confidence exactly 1 - 0.05/3 (display 98.333%). These 12 authored workspace families support a limited screening result, not population coverage. If a statistic has no bootstrap variation, mark its interval unavailable rather than certain.
Rank only a complete qualified cohort by the primary point estimate; equal recorded scores share rank and ties display alphabetically. Do not blend cost, time, and success into one score. A practical advantage requires a paired lower bound above the predeclared five-percentage-point margin. No model-independent winner or pooled Model x harness ranking is inferred.
No LLM judge or subjective answer rubric enters the score. All 24 tasks use three executable requirement gates. A reviewer audits discrepancies between the prompt and grader before unblinding harness labels where possible; an unresolved grader defect invalidates the affected matched block under the common replacement rule. Never silently change a grader or task after observing results.
Documentation and evidence
Documentation review and evidence contract
Documentation review
Protocol reviewed on 2026-10-21. Configuration documentation and task-design research inform the draft; they do not verify a working pairing or task difficulty.
Custom provider configuration exposes a model ID, base URL, Responses wire API, and reasoning effort.
Use an isolated CLI profile with model=gpt-6-sol, model_reasoning_effort=medium, a named provider, and wire_api=responses. Qualify HTTP/SSE, the model catalog, effective limits, native instructions, and disabled optional services for the pinned release.
Provider configuration exposes baseUrl and openai-responses; model settings include context, output limits, and reasoning mappings.
Use an isolated Pi profile and explicit model ID, baseUrl, and api=openai-responses. Audit effective reasoning mappings, context/output limits, native tools, package loading, and compaction calls for the pinned release.
Required experiment evidence
No experiment originals exist for this draft. Future records must retain the following evidence.
Accounting and review
Per-request token/usage evidence, actual requested and returned model/tier, conservative reservations, reference cost, private billing reconciliation, original/replacement lineage, and all initial and final grades.
Attempt record
Cohort/block/task/repeat/profile IDs, start/end times, fresh-state and environment digests, stop reason, native visible session/tool artifacts, final answer, final repository patch, and independent grade. Provider-hidden reasoning is not required or reconstructed.
Configuration identity
Exact harness release/source/binary digest, model/route/tier/effort and capability limits, sanitized resolved config, native instructions/tool schemas and hashes, bundled features, auxiliary routes, and qualification report.
Task identity
Task ID, workspace family/immutable snapshot, stack, estimated difficulty, provenance/license, prompt/instructions, dependency/image digests, three gate IDs and grader hashes, reference commitments, and review history.
Select and record exact Codex, Hermes, and Pi release/source/binary digests; documentation alone does not qualify them.
Build the 12 workspace fixtures and all 24 task packages from the authored specifications. Supply visible examples, reference solutions, independent graders, hashes, licenses, and six separate development tasks. None is built or validated yet.
Review the estimated difficulty on separate development tasks after a later execution decision; formal-task discriminating power has not been measured. Freeze the formal set before results.
Freeze provider access, model revision binding or explicit alias limitation, request limits, and sanitized profiles; build and qualify accounting and disposable worker/grader images.
A later execution decision must approve the proposed cash/time envelope before qualification or provider calls. Model x harness stays deferred with no budget.