Changelog

← board · RSS · every entry = the board changed (new model, new measurement, or a methodology change)

2026-07-14

Scoring made bit-reproducible; two upstream IFEval items disclosed

The official IFEval checkers are not reproducible out of the box: langdetect is unseeded, results are hash-order sensitive, and upstream items 1122/1129 carry non-alphabet "letters" ('#', '!') that the official checker replaces with an unseeded random letter every run (~±0.2pp jitter for anyone using the official scorer).
All three sources are now pinned; every number on the board regenerates bit-identically from the published raw/ outputs. Board values shifted ≤0.3pp one final time in this change (well inside every CI). Details: methodology → reproducibility.
Also fixed: stale subset-era numbers on the methodology page (the FM "weak on IFEval" claim was corrected by the full battery — kept as a worked example of small-n noise).

2026-07-13

8 entries (5 open-model vendors + Apple's built-in Foundation Model), full 596-item battery (IFEval official + MMLU-Pro stratified + MATH-500), 95% CIs on everything.
7 of 8 rows device-measured on iPhone 17 Pro (PipelinedBench, greedy token-parity gated).
Finding: the top is a five-way statistical tie — the built-in FM (76%) does not clearly beat the best open 1–2B ports (LFM2.5-1.2B, Youtu-2B, Qwen3.5-2B, Gemma 4 E2B).
Finding: Google's official QAT int4 (Gemma 4 E2B) measures at parity with its bf16 checkpoint on MMLU-Pro/MATH; IFEval ~90%.
Finding: quantization loss is not monotone with size — most ports hold 90–98% on MMLU-Pro; the 4B reasoning model drops to 78%.
Results dataset (CC-BY-4.0): devicemark/results — the ranked board is the default Viewer table.

2026-07-13

Every row now has an embeddable badge at /badge/<slug>.svg (e.g. lfm2.5-1.2b) — score + device tok/s, regenerated on every deploy.
Model authors: embed it with [![DeviceMark](https://devicemark.github.io/badge/<slug>.svg)](https://devicemark.github.io/). Slugs are the artifact_id prefix in board.json.