Harness Rank

Task outcomes across model–harness combinations.

3 harnesses in scope
24 task specifications
No measurements
Draft 0.3.0

Data Leaderboard Method Records

Study files

Task definitions are available. Trial data has not been collected.

Task set and protocol24 tasks · 72 acceptance gatesJSON ↓

Browse tasks →

Read prompts, deliverables, and acceptance criteria.

Evidence requirements10 file types · for future runs

Required evidence files

No attempt artifacts have been collected. Each future trial must retain these files.

File purpose required file	Contents required evidence	Details
Acceptance results`grade.json`	Three binary requirement-gate results with assertion evidence, overall acceptance, and grader/environment hashes. Keep later corrections as appended records.
Code changes`repository.diff`	Final repository state, forbidden-path changes, and patch hash, captured even on failure.
Cost and usage`accounting.json`	All request IDs, usage, cost statuses, reservation release, billing scope, exclusions, and replacement linkage.
File checksums`manifest.json`	Artifact paths, SHA-256, byte counts, schema versions, provenance, and public-copy derivation/redaction references.
Final answer`answer.md`	The retained final visible answer, including unsuccessful or interrupted output.
Harness configuration`config.json`	Sanitized effective profile, provider/model/effort, limits, native prompt/tool hashes, and release/source identity.
Model requests`transport.jsonl`	Visible request/response and completion/error/usage events through the recording route; no Authorization headers or provider secrets. Preserve opaque items without decrypting hidden reasoning.
Original session`originals/`	Native session files and tool outputs in original formats. Keep protected originals separate from explicitly derived public copies; never normalize over an original.
Run record`attempt.json`	Attempt identity, frozen inputs, schedule block, wall time, resource observations, exit/stop reason, and grade reference.
Task definition`task.json`	Task specification ID, family, stack, estimated difficulty, workspace snapshot/image/input hashes, and the three gate commitments.

Evidence preservation and publication

Record the input/configuration commitments before launch; append attempts and later corrections without overwriting originals. Check artifact hashes, request/usage reconciliation, grading links, and complete matched coverage before publication. Publish redacted, licensed reproductions after the cohort; label derived copies and retain protected originals under explicit custody. Do not publish provider receipts, private profiles, hidden reasoning, or the ELF private archive. A hash proves file identity, not semantic correctness.

Configuration documentation

Research and model references · 7