Why we publish the method before the number

Most claims about making AI agents cheaper arrive as a single number. “40% fewer tokens.” Nobody can check it, so nobody should believe it.

We would rather be slow and checkable than fast and quoted.

The rule

A savings figure does not appear on this site without five things beside it:

1. The label. Is it calculated or measured? We say which, every time.
2. The method. What was counted or run, on what, with which settings, and what counted as the same work.
3. The baseline. What it was compared against, handled the same way.
4. The inputs. For a measurement, every run, including the ones that made us look bad. For a calculation, the numbers it was built from.
5. The version. The exact release the figure describes.

If a reader cannot check it, it is not a result. It is a story.

What this means right now

VBW 1 has calculated figures that follow this rule, labelled as calculations, not benchmarks. VBW 2’s figures follow it too: each one says whether it was calculated or measured, and the method sits beside it.

“Same work” is the hard part

Cheaper is easy: do less. The claim that matters is cheaper and no worse. That means defining acceptance up front, with tests that fail first and pass for the right reason, and counting a run as a saving only if the work still passes. Tokens saved on work that no longer passes are not savings.

Hold us to it

Everything here is open source. If you find a claim on this site without its method, open an issue and we will fix the claim or remove it.