How round-one benchmarking works, and what it does not claim

Every benchmark page tempts the author to print a big number. The first MythosCode round prints none that measures capability, and this post explains why.

What round one is

The real CLI runs a fixed suite of four tasks, five trials each, against an offline mock gateway with a scripted model. A hidden verifier decides each trial from the resulting workspace. The model never sees it. Token counts are the usage the CLI itself recorded.

One of the four tasks is a negative control that must fail on every trial. If the verifier passed everything, a column of perfect scores would mean nothing. The control is how we know the verifier discriminates.

What it checks

Because the model is scripted, the pass rates measure the pipeline: the CLI starts, runs its tools, accounts for tokens, the verifier separates pass from fail, and the statistics are computed the way the live suites will compute them. Of 20 trials, 15 solved. The 5 that did not are the negative control, failing as it must.

Those numbers are not model quality and not harness quality. Do not quote them as either.

Integrity rules

• The suite, the scripted turns and the verifiers live in the repository, and nothing is tuned after a run.
• The results page is rendered from the recorded trials, so a hand-edited number fails a check.
• Cost columns use illustrative rates. They check the accounting and are not spend.
• A lead of less than three percentage points is a tie unless the intervals separate.

What is pending

None of these has run, and none publishes a number until it does: Terminal-Bench, SWE-bench Pro, an edit-reliability suite, runtime efficiency and a robustness suite. Each needs a gateway key and a budget that has not been set. The plan comes before any run.

Where to read it

The full record is the benchmark file in the repository. This site quotes it and nothing else.