doc2harness

What is doc2harness

Doc2Harness is a benchmark framework for evaluating large language models on long-horizon coding agent tasks, providing official implementation code.

Key features

• Evaluates large language models' ability to build coding agents through two phases: the construction phase and the evaluation phase
• Uses OpenCode in an isolated container to let the model implement a complete agent framework based on development documentation and interface skeletons
• Performs end-to-end evaluation using a frozen harness and scripted mock model
• Scores via four observable channels (model requests, disk files, exit codes, command output) and positive controls
• Provides 94 public scenarios (including 53 scoring test cases and 41 positive controls)

Who it is for

Suitable for researchers, developers or teams evaluating the performance of large language models on long-horizon coding agent construction tasks.

How to get started

1. Visit the GitHub repository https://github.com/YuyaoGe/doc2harness
2. Ensure Python 3.11+ and Docker are installed locally
3. Run Stage 2 to evaluate an existing harness following the Quick start section in the README

Compiled from public descriptions on the tool's official website / README / launch page. Pricing and user numbers have not been verified; please check the official website. Some sections were drafted with AI assistance. Written in Chinese and English; other languages are machine-translated.