Changelog

← board · RSS · every entry = the board changed (new model, new measurement, or a methodology change)

2026-07-22

New track: Anchored ops (preview) — device chores, checked deterministically

What it is. 48 single-turn device chores across 7 skill clusters — cross-time-zone reminders, bill splitting, photo-set selection, ownership-filtered action items, note merging, slot copying, batch multiplicity — compiled from the agent task suite's task definitions (one source; the same JSON runs multi-turn in the agent harness). Scored by deterministic checkers: exact amounts, ISO-instant equality, exact item sets; an unparsable reply is a fail. No judge model. Methodology → "Anchored-ops track".
Board placement. A new Ops◦ column (MACRO over the 7 cluster pass-rates) plus a panel with the per-cluster heatmap and two tasks shown verbatim with every model's actual returned arguments. Preview: Mac engine, greedy, cap 4096, not part of the intelligence composite; iPhone runs pending.
Results (7 rows). Youtu-LLM-2B 50% · Granite-4.0-H-1B 25% · Qwen3.5-2B 24% · LFM2.5-1.2B 20% · Nanbeige4.1-3B 10% · Nemotron-3-Nano-4B 5% · Qwen3.5-0.8B 4%. The ranking decouples from the intelligence composite in both directions: Nemotron (2nd on intelligence) writes its reasoning as prose instead of the required JSON call (parse 4%), while Youtu converts long reasoning into computed arguments (bill math 4/7, batch 5/5) at the cost of 9/48 thinking past the cap. Raw outputs ship with the results dataset.

2026-07-22

Gemma on its native LiteRT-LM runtime; a cap-unit bug fixed; editorial framing removed

Gemma 4 E2B now measured on its native LiteRT-LM runtime (Mac, greedy), not the Core AI path (which balloons memory via the PLE-gather graph). Quality is runtime-neutral and the runtime is disclosed; retention (③) is N/A for this row until an fp16 baseline exists on the same runtime. iPhone decode speed + peak memory are now device-measured on an iPhone 17 Pro (LiteRT-LM/Metal): ~57 tok/s, 488 MB — vs the Core AI path's 21.7 tok/s / 4450 MB, i.e. ~2.6× faster and ~9× lighter (Core AI's PLE-gather graph balloons memory, which is why quality is taken on LiteRT). Decode is single-run: the x2-warm protocol can't complete on this runtime (a callback-pool stall), disclosed.
Fixed a measurement bug that unfairly penalized Gemma. The LiteRT-LM harness capped output in characters (4096) while every other row capped in tokens (4096) — a ~4× tighter budget. Re-measured at a token-matched budget: MMLU-Pro no-answer dropped 69→5 and composite rose 48.1% → 52.9%. All rows now share the same 4096-token budget; four rows whose labels still read "cap 1024" were corrected to the 4096 they actually ran at.
Editorial "findings" removed. The board presents results and methodology only — no claims about who wins. Per-row diagnostics added (answered %, median generated tokens, and acc-when-answered on hover) so a low score reads as "delivered fewer answers within budget," not "less capable."
Runtime-validity check: on this runtime Gemma scores 85/100 on GSM8K (matching the LiteRT int8 reference; fp16-activation runtimes score 48/100), confirming the activation path is correct — its lower composite is delivery-rate under the cap and answer-format compliance, not degraded numerics.

2026-07-17

Two new vendors: NVIDIA + Nanbeige (9 vendors, 12 rows)

Nemotron-3-Nano-4B (NVIDIA, Mamba2 hybrid) and Nanbeige4.1-3B (32-layer GQA) join the board — full 596-item battery, float retention, and an iPhone 17 Pro token-exact gate (PB nat 24/24 · oracle 24/24, iOS 24A5380h). Composite 64.9% and 63.1%; decode 14.7 and 16.9 tok/s on iPhone.
Both are heavy reasoners (cap 4096). Nanbeige is the most extreme "won't stop thinking" model on the board: highest completed-only MMLU/MATH (95.7% / 92.3%) yet IFEval ≈ 1% — it almost never emits a final answer within budget (295/300 IFEval items are all thinking). The smartest-when-it-finishes model is also the least instructable — exactly what the completed-only + IFEval split is built to show.
Held back: MiniCPM5-1B — the Core AI int8 export degenerates (repeats \boxed{} hundreds of times); the official GGUF of the same weights scores cleanly, so the bug is in the export, not the model. It ships only once that's fixed.

2026-07-15

Built-in FM decode speed — published as an honest estimate (▵)

Apple's FoundationModels API exposes no token counts, so the FM row's speed cell was n/a. Now measured on both surfaces (streamed chunk timing, prefill excluded; n=8 each, TTFC 0.42 s median): 311.4 chars/s sustained on iPhone 17 Pro (on-device, PipelinedBench PB_FM_SPEED) and 220.3 chars/s on M4 Max.
Board shows ▵ ~70–86 tok/s (est., iPhone) — a band, not a point: tok/s is derived through the empirical 3.61–4.43 chars/token band of the 7 open ports (~4,700 full596 generations). The FM tokenizer is unknown; the spread is the systematic uncertainty, disclosed. Mac (~50–61) kept in the hover + methodology.
Disclosed curiosities: the phone measures faster than the Mac on this protocol, and iPhone per-run rates drift 345→275 across 8 runs (thermal — visible in the raw timings). Raw: raw/fm_speed_mac.jsonl + raw/fm_speed_iphone.jsonl. Protocol: methodology → Speed.

2026-07-14

Cloud sea-level lines added — how far is the pocket from the data center

Two cloud APIs (Gemini Flash + Pro) run the identical 596-item battery through the same scorers. Both land ~93% composite (a tie).
The gap is now visible in one glance: best pocket port ~72% → Apple's built-in FM 76% → cloud ceiling ~93%. Drawn as reference lines (no device axis).
Raw cloud outputs published in the dataset raw/. See methodology → cloud sea-level.

2026-07-14

Scoring made bit-reproducible; two upstream IFEval items disclosed

The official IFEval checkers are not reproducible out of the box: langdetect is unseeded, results are hash-order sensitive, and upstream items 1122/1129 carry non-alphabet "letters" ('#', '!') that the official checker replaces with an unseeded random letter every run (~±0.2pp jitter for anyone using the official scorer).
All three sources are now pinned; every number on the board regenerates bit-identically from the published raw/ outputs. Board values shifted ≤0.3pp one final time in this change (well inside every CI). Details: methodology → reproducibility.
Also fixed: stale subset-era numbers on the methodology page (the FM "weak on IFEval" claim was corrected by the full battery — kept as a worked example of small-n noise).

2026-07-13

v0 launch — 8 entries, full battery, device-measured

8 entries (5 open-model vendors + Apple's built-in Foundation Model), full 596-item battery (IFEval official + MMLU-Pro stratified + MATH-500), 95% CIs on everything.
7 of 8 rows device-measured on iPhone 17 Pro (PipelinedBench, greedy token-parity gated).
Finding: the top is a five-way statistical tie — the built-in FM (76%) does not clearly beat the best open 1–2B ports (LFM2.5-1.2B, Youtu-2B, Qwen3.5-2B, Gemma 4 E2B).
Finding: Google's official QAT int4 (Gemma 4 E2B) measures at parity with its bf16 checkpoint on MMLU-Pro/MATH; IFEval ~90%.
Finding: quantization loss is not monotone with size — most ports hold 90–98% on MMLU-Pro; the 4B reasoning model drops to 78%.
Results dataset (CC-BY-4.0): devicemark/results — the ranked board is the default Viewer table.

2026-07-13

Per-model badges

Every row now has an embeddable badge at /badge/<slug>.svg (e.g. lfm2.5-1.2b) — score + device tok/s, regenerated on every deploy.
Model authors: embed it with [![DeviceMark](https://devicemark.github.io/badge/<slug>.svg)](https://devicemark.github.io/). Slugs are the artifact_id prefix in board.json.