The Almedra Token-Efficiency Benchmark, Updated For GLM 5.2

The benchmark is deliberately ordinary.

Almedra Timepieces is a one-page small-business website built with React and Vite. It is the kind of codebase people actually hand to coding agents: real layout, real data, real styling, and a feature request that requires state, UI, and restraint.

The task was to add a two-watch comparison feature to the collection section. Every agent received the same prompt, the same clean repo, and the same GLM 5.2 model via OpenRouter routed through StreamLake.

The Result

We ran each benchmark three times, threw out the worst result, and averaged the remaining two. Command Code was measured later on the same fixture, prompt, and GLM 5.2 OpenRouter route.

While the field compressed, the spread is still obvious.

Lucena posted the smallest token footprint in the Almedra token-efficiency benchmark task using GLM 5.2.

Pi used fewer tokens than the heavier harnesses, but stopped at build errors in all three runs.

The heavier harnesses all spent materially more tokens, even with aggressive caching. Cache helps; a smaller working set helps more.

Lucena used 61,506 total tokens across 5 model calls, the lowest token count and fewest calls in the GLM 5.2 field shown here. Input was 54.9k (46.2k cached). Output was 6.6k.

The Prompt

The prompt asked each agent to add the same practical website feature:

Add a two-watch comparison feature to the collection.

Requirements:

- Each watch in the collection has a compare control.
- Users can select up to two watches.
- If a third watch is selected, replace the oldest selected watch.
- When two watches are selected, show a comparison panel at the bottom of the collection section.
- The panel compares movement, case size, power reserve, water resistance, and price.
- The panel includes both selected watch names.
- Include a `Clear comparison` control.
- Use the existing `timepieces` data.
- Keep the existing editorial visual direction without adding image cards.
- Do not change unrelated sections of the page.

What We Verified

Correctness came first. Token count only mattered after the run actually worked.

• Lucena, OpenCode, Command Code, Kilo Code, and Continue all produced a buildable result in the GLM 5.2 field we captured.
• Pi stopped at build errors on the GLM 5.2 reruns.
• The verifier selected Azahar 1874 and Turia Moonphase, then selected Serra GMT.
• The third selection had to replace the oldest selected watch.
• The panel had to include movement, case size, power reserve, water resistance, price, both watch names, and a clear control.

The Ledger

The token counts below come from OpenRouter logs for GLM 5.2 runs recorded for this update.

What This Test Tells Us

GLM 5.2 tightened the field, but it did not erase the working-set differences between harnesses.

Lucena led this update on total tokens and model calls. That is the headline result.

Pi is still the obvious pressure point because it used fewer tokens than the heavier field, but the build did not clear. OpenCode, Command Code, Continue, and Kilo Code all shipped usable results in the capture we used here, but they did it with much larger working sets.

Cache helps. Sending less unnecessary context helps more.

Time to Completion

The time column comes from the OpenRouter session log view we used for this update. In that view, session windows and request timestamps are minute-granular, so the times shown here are approximate rather than second-precise.

That still gives every agent the same provider-side clock. Local CLIs can add startup time, terminal rendering, package resolution, and human paste flow around the run; OpenRouter records the part every agent had to buy from the provider.

Run The Harness

The benchmark repo contains the clean Almedra fixture and the exact prompt used for this run. Open the fixture, run the prompt with another coding agent, then compare the OpenRouter usage.