Benchmarking real repositories with AnyBench.

A coding benchmark is only useful if its tasks are grounded in real code and its checks distinguish a working change from a plausible patch. AnyBench starts with a repository's Git history, turns selected changes into coding problems, verifies each problem against the before and after revisions in Docker, and then measures candidate models on the verified set.

I built AnyBench to make that workflow usable with repositories you choose at run time. It does not ship a fixed test set or assume one programming language. The first public example runs it against two mature Python libraries: Humanize and Boltons.

From recent commits to replayable tasks

On September 25, 2026, I sampled the five then-recent commits in each repository. GLM-5.3 Flash drafted six problems from those ten commits. I reviewed the problem statements and test commands, then kept the cases whose checks could be tied to behavior changed by the historical commit.

Verification is the gate. Each selected check ran against both snapshots in an isolated Docker environment: it had to fail on the parent revision and pass after the reference change. Five cases passed that check. One translation workflow was skipped because the environment lacked the required gettext tooling.

This before-and-after test is what makes a commit-derived task useful. A test that simply passes on the fixed version may be unrelated, too broad, or accidentally dependent on behavior that already existed.

What happened in the candidate run

Qwen3.8-27B was run once on each of the five verified cases, with one attempt at a time and a 20-step limit. Three attempts completed and passed their checks. Two ran into the token budget before completion. GLM-5.3 Flash judged the completed attempts at a mean score of 0.90.

These numbers describe a small diagnostic run, not a model ranking. The sample is tiny, the task set was reviewed and adjusted by a person, and two attempts did not finish. The judge score adds context, but the locally verified tests remain the main signal.

Review is part of the benchmark

The builder can propose a command that refers to a test introduced by the target commit, contains invalid syntax, or assumes a tool that is not installed. I inspected every generated case and replaced unsuitable commands with focused checks that work against both snapshots. The translation task stayed visible in the run data with its skip reason rather than disappearing from the totals.

That human pass is not a loophole around automation. It is part of the dataset creation process, and it is recorded so someone replaying the example can see what was selected and why. The checked-in CSVs include the reviewed cases, evaluator commands, and historical reference diffs; the JSON results record the commits, changed files, validation outcomes, and candidate attempts.

Containers keep the environment honest

The default sandbox did not include all the dependencies these libraries needed. I built a shared Python image with pytest, freezegun, and Humanize's generated version module, then used that image for validation and candidate execution. Verification and benchmark containers run without network access; image construction can still download public dependencies.

That boundary matters for reproducibility. A failed setup should be reported as a setup problem, not turned into a model failure. Likewise, a candidate whose patch passes one check but does not finish its attempt is still incomplete; AnyBench records that outcome rather than counting it as solved.

Replay the example

The repository includes pinned cases and model configuration files without credentials. After installing AnyBench and Docker, the examples README shows how to build the image, validate the Humanize or Boltons cases, run a candidate, evaluate completed attempts, and generate an HTML report. Validation itself does not call a model; replaying candidate and judge calls uses your configured API endpoint and can incur costs.

docker build -f examples/Dockerfile.python-libs \
  -t anybench-python-libs:example .

anybench validate examples/humanize/cases.csv --check-tests \
  --image anybench-python-libs:example \
  --verified-output .anybench/humanize-verified.csv

Start with the example data and replay guide for full commands, configuration, and the limits of this run.

What the run changed in AnyBench

Trying the workflow on real repositories exposed practical gaps: the builder needed bounded source inspection, test commands needed clearer guidance, and the runner needed better handling for common line ranges, newline preservation, and cache files. I also tightened container Git ownership and added an end-to-end deadline for API responses.

That feedback loop is the point of the project: use ordinary repositories to exercise the whole path from commit selection to a report, then improve the tool where the workflow breaks. AnyBench makes it easier to build a small benchmark for a codebase; it does not remove the need to inspect the cases or interpret the results carefully.