The Almedra Token-Efficiency Benchmark, Updated For GLM 5.2

The benchmark is deliberately ordinary.

Almedra Timepieces is a one-page small-business website built with React and Vite. It is the kind of codebase people actually hand to coding agents: real layout, real data, real styling, and a feature request that requires state, UI, and restraint.

The task was to add a two-watch comparison feature to the collection section. Every agent received the same prompt, the same clean repo, and the same GLM 5.2 model via OpenRouter routed through StreamLake.

The Result

We ran each benchmark three times, threw out the worst result, and averaged the remaining two.

While the field compressed, the spread is still obvious.

Lucena posted the lowest billed cost and the smallest token footprint in the Almedra token-efficiency benchmark task using GLM 5.2.

Pi remained cheap, but stopped at build errors in all three runs.

The heavier harnesses all spent materially more tokens, even with aggressive caching. One cache miss, and those costs climb fast.

Lucena finished at $0.011665 and used 34,102 total tokens across 8 model calls, the lowest billed cost and the lowest token count in the GLM 5.2 field shown here.

The Prompt

The prompt asked each agent to add the same practical website feature:

Add a two-watch comparison feature to the collection.

Requirements:

- Each watch in the collection has a compare control.
- Users can select up to two watches.
- If a third watch is selected, replace the oldest selected watch.
- When two watches are selected, show a comparison panel at the bottom of the collection section.
- The panel compares movement, case size, power reserve, water resistance, and price.
- The panel includes both selected watch names.
- Include a `Clear comparison` control.
- Use the existing `timepieces` data.
- Keep the existing editorial visual direction without adding image cards.
- Do not change unrelated sections of the page.

What We Verified

Correctness came first. Cost only mattered after the run actually worked.

• Lucena, OpenCode, Kilo Code, and Continue all produced a buildable result in the GLM 5.2 field we captured.
• Pi stopped at build errors on the GLM 5.2 reruns.
• The verifier selected Azahar 1874 and Turia Moonphase, then selected Serra GMT.
• The third selection had to replace the oldest selected watch.
• The panel had to include movement, case size, power reserve, water resistance, price, both watch names, and a clear control.

The Ledger

The usage table below comes from OpenRouter logs for GLM 5.2 runs recorded for this update.

What This Test Tells Us

GLM 5.2 tightened the field, but it did not erase the working-set differences between harnesses.

Lucena led this update on both cost and total tokens. That is the headline result.

Pi is still the obvious pressure point because it stayed relatively cheap, but the build did not clear. OpenCode, Continue, and Kilo Code all shipped usable results in the capture we used here, but they did it with much larger working sets.

Cache helps. Sending less unnecessary context helps more.

Time to Completion

The time column comes from the OpenRouter session log view we used for this update. In that view, session windows and request timestamps are minute-granular, so the times shown here are approximate rather than second-precise.

That still gives every agent the same provider-side clock. Local CLIs can add startup time, terminal rendering, package resolution, and human paste flow around the run; OpenRouter records the part every agent had to buy from the provider.

Run The Harness

The benchmark repo contains the clean Almedra fixture and the exact prompt used for this run. Open the fixture, run the prompt with another coding agent, then compare the OpenRouter usage.