← back to the leaderboard

Methodology

On-Device LLM Leaderboard · v0, iPhone tier

Proof strength is disclosed honestly: where a column rests on a proxy, it says so.

What the leaderboard measures

"Of the models that actually run on this phone, which is the smartest that stays usable?" — the intelligence × decode-speed Pareto front under real device limits. Intelligence is measured on Mac (fast) and transferred to the device by the zoo parity gate (device ≡ Mac ≡ HF greedy token-exact); only tok/s, memory, and power are measured on the device itself. Measuring smarts at ~200 tok/s on a Mac instead of dying at ~20 tok/s on the phone is only honest because the parity gate proves the phone emits the same tokens.

The three columns per entry

Quantization variants (int8 vs int4) are separate entries, never mixed. ① is always measured with each model's shipped recipe default (the quant actually in the ModelStore bundle), not a softer fallback.

② baseline — implementation decision, and its proof strength

The v0 family is Qwen3.5 (0.8B / 2B). Its checkpoint is a vision-language model with a hybrid linear-attention (gated-delta / SSM) + sparse full-attention text backbone (model_type: qwen3_5, mostly linear_attention with full_attention every 4th layer, MTP head, mrope). Three baselines were considered:

• (a) third-party HF transformers (bf16) — the most authoritative, but not available: transformers 4.57.6 has no qwen3_5 model_type (only qwen3, qwen3_moe, qwen3_next, qwen3_vl*). It existed only in the unreleased 4.57.0.dev0 the checkpoint was saved with, and the repo ships no custom modeling code. Chasing a fork for a VL+SSM hybrid is out of scope for v0.
• (b) coreai_models eager reference (float) — chosen. Qwen3_5StatefulForCausalLM is coreai_models' own PyTorch implementation of this exact architecture — the same definition that is quantized+exported for ①. Run unquantized (fp16), greedy, it is a genuine pre-quantization reference. Harness = ref_eval.py.
• (c) fp16-on-engine proxy — the fp16 Core AI bundle. Fast but still engine-compiled, so it isolates only int8-vs-fp16 weights on the same engine. Kept as a secondary cross-check, and validated to agree with (b) eager and, for LFM2.5, with (a) transformers — so it is used as the uniform fleet baseline with eager/ transformers anchors.

Two-tier cost design. The eager reference decodes at ~11 tok/s on the Mac GPU — ~18× slower than the engine — so it is not run on the full battery. Column ① runs the full battery; column ② runs a stratified subset, and retention ③ is computed on those shared keys. Subset size is disclosed per entry.

Evaluation battery v0

Greedy (deterministic), cap = 1024 tokens (4096 for reasoning-heavy models), 0-shot, chat template with thinking OFF (verified: on-device ChatSession runs thinking-off). Prompts instruct step-by-step reasoning in the visible answer.

GPQA-Diamond is excluded from the iPhone tier: at ≤1B it sits at/below chance (13.3% vs 25% chance, 60% never converge within cap) = floor effect, and it is gated (ToS). It moves to the Mac / v1 tier. Benchmark contamination is a real risk; the battery is versioned (battery_version) and rotated.

The cap lever + the retention confound (documented, not hidden)

max-tokens is an explicit design lever. Verbose small models spend the whole budget reasoning: qwen3.5-0.8B int8 at cap-1024 on MMLU-Pro caps 69/196 and produces no boxed answer in 65/196 — those score as wrong (a fair "couldn't deliver a usable answer on-device in-budget" signal). Larger / less-verbose models cap far less.

Finding: ① and ② cap at different points (greedy paths diverge — int8 vs float weights, engine vs eager), so their no-answer rates differ and the retention ratio ③ can be dominated by cap-timing, not quantization quality — it can even exceed 100% at small n. On shared answered problems the two emit identical boxed answers. Two mitigations, both reported:

1. Two accuracies per bench: acc (no-answer = wrong; the on-device "usable" number) and acc_completed (only problems that produced an answer; cap-independent quality). Retention ③ is reported on acc_completed so it reflects quantization damage, not verbosity; acc + no-answer rate carry the usability story. When retention is within noise (≥0.97 or CI-consistent) the board shows "≈100% ▵" and keeps the raw value in the data.
2. Larger n + a per-tier cap: raise the cap when no-answer is the dominant term.

Confidence intervals & ranking

Reproducibility. Every number regenerates bit-identically from the raw per-item outputs: the scorer pins PYTHONHASHSEED, seeds langdetect, and seeds Python's random. The last one matters because the official IFEval checkers are not reproducible out of the box: upstream google/IFEval items 1122 and 1129 carry non-alphabet "letters" ('#', '!') in keywords:letter_frequency kwargs, and the official checker replaces those with an unseeded random letter on every run — a ~±0.2pp jitter that affects anyone using the official scorer unseeded. We keep the official semantics (seeded) and disclose the two items.

Every accuracy carries a 95% interval, so ranks are not read off noise:

• Per-bench accuracy: Wilson 95% CI.
• The intelligence composite (mean of the three per-bench means): item-bootstrap, 2000 resamples, 95% interval. Shown as ±half-width in the table and as whiskers on the Pareto chart.
• Rank = 1 + (number of models strictly better), where "strictly better" means a model's CI lower bound is above this model's CI upper bound. Overlapping CIs are a statistical tie (marked =). Subset runs leave the top group tied; the full 596-item battery separates them.

Speed / memory / power

PipelinedBench (STATS prefill/decode) on iPhone 17 Pro / M4 Max. tok/s + peak footprint for v0; power (tokens/joule) is a v0.5 axis. This box ≈ M4 Max: 0.8B int8 = 207 tok/s decode, matching the published 204. A row's memory is marked est until device-measured, and decode is shown as Mac (○) until an iPhone number exists. Reported tok/s are warm-state: engine loaded and warmed, cold load and first-run specialization excluded; 128-token prompt / 256-token decode, two trials on a settled device. The built-in Foundation Model publishes no tok/s — its public API returns no token counts or streaming stats, so only wall-clock per answer exists (kept in the raw logs).

Runtime-neutral schema (day-1 multi-runtime)

Quality is an attribute of the artifact (model × quant × format), measured on its native runtime + float retention. Speed/memory/power are attributes of runtime × device. Two normalized, HF-dataset-ready tables: artifacts.jsonl and measurements.jsonl. v0 data is Core AI only (format=aimodel, runtime=coreai), but GGUF/llama.cpp and MLX/mlx-lm rows drop in as pure data additions.

Apple's built-in Foundation Model as a board row

Apple's on-device system model (SystemLanguageModel via LanguageModelSession()) runs the same battery through its public API — no weight extraction (runtime=foundation-models, format=system, DL size 0, OS-resident, retention=null). On the full 596-item battery it tops the composite point estimate (76.1%) with no weak bench (IFEval 82.1 / MMLU-Pro 64.8 / MATH 81.5 completed; refusals 0) — but it is a statistical tie with the best open 1–2B ports, and LFM2.5-1.2B beats it on IFEval (87.7). Correction note: an earlier subset run (n=40) showed FM "weak on IFEval" (67%); the full battery corrected that to 82.1% — small-n noise, kept here as a worked example of why the board reports CIs and refuses to rank inside them.

Refusal rate (a column for all, decisive for the built-in)

guardrailViolation and explicit textual refusals are separated from wrong answers (refusal_rate; accuracy is computed over non-refused items). Near-zero on the general battery for every model, but the column matters the moment a bench probes guarded topics.

Finding: reasoning-heavy ≠ instruction-following ≠ on-device-practical

Reasoning-heavy models (Qwen3.5-4B, the built-in FM) top MMLU-Pro but crater IFEval: Qwen3.5-4B is the smartest port on MMLU-Pro (68.9% completed) yet scores 31.5% on IFEval because it prepends a "Thinking Process:" meta-analysis that isn't the requested output — violating format constraints outright — and generates ~4000 tokens even at cap-4096. The composite includes IFEval deliberately: raw knowledge is not the same as being useful in your pocket.

Fairness & corrections

Publishing another model's score carries a duty of fairness. Every entry links its parity proof; if we measured the wrong quant, or a port lost parity and the number reflects a broken export rather than the model, model owners can contest it via the objection issue template. Only open-weight models are ported; the only closed system on the board is Apple's built-in FM, via its public API.

Contamination — which columns can and cannot be gamed

Public benchmarks may appear in any model's training data, and this is not verifiable from outside. The board's columns differ sharply in how much that matters:

• Immune — retention ③, tok/s, memory, energy. Retention compares the quantized and float versions of the same weights: whatever the model memorized, both sides memorized equally, so contamination cancels and only quantization damage remains. Speed/memory/energy are physical measurements.
• Resistant by construction — IFEval. Scoring is programmatic constraint checking on freshly generated output (word counts, forbidden characters, formats). There are no gold answers to memorize; a model must actually follow the instruction at inference time.
• Vulnerable — MMLU-Pro and MATH. Gold answers are memorizable. A model fine-tuned on the test set (deliberately or via crawl accident) inflates these two columns.

Mitigations, in order of deployment: the battery is versioned and rotated (each rotation bumps battery_version); a paraphrase probe is planned — the same items rewritten with different wording/numbers, scored pairwise, so a memorizer shows a public>paraphrase gap that genuine capability does not; and the roadmap's endpoint is self-authored, freshly-captured test sets that are contamination-free by construction. Submissions suspected of benchmark tuning are checked with the probe before listing.

Related work — what this board is and is not

Where DeviceMark sits among the tools that measure local/on-device AI. The matrix uses each project's own published scope (links below); "intelligence axis" means output quality is a reported, per-model comparison column, not merely an internal pass/fail check.

• MobileAIBench (Salesforce, 2024) — the closest conceptual neighbor: LLM/LMM task performance across quantization levels for mobile-sized models, with "a library for running evaluations on desktops and an iOS app for on-device latency and hardware utilization" (their split, quoted). Its task coverage (multimodal, trust & safety) is broader than this board's. Differences: it evaluates models the harness re-quantizes, with quality measured on the desktop side and no token-level proof that the phone runs the same computation; DeviceMark scores the shipped artifact on its native runtime and transfers quality to the device through a greedy token-parity gate, adds retention vs the same-weights float baseline, the built-in Apple FM row, CI-based ties, and a continuously updated board rather than a paper snapshot.
• MLPerf Client (MLCommons, v1.6, 2026) — the industry-standard client-side performance benchmark: TTFT and tokens/s on int4 models, now on Windows, macOS Tahoe 26, iPadOS 26 and Ubuntu. Its standardization and vendor governance are far stronger than this board's. Quality appears only as a qualification threshold (models must clear an MMLU score of 43–70 to be included) — it is not a reported per-system comparison axis, there is no phone tier, no memory/retention column, and no per-model intelligence ranking. DeviceMark is complementary: the quality×speed join on the iPhone tier.
• LocalScore (Mozilla Builders) — a llamafile/llama.cpp desktop benchmark combining prompt speed, generation speed and TTFT into one score, with a crowd-sourced hardware database. By its own docs it does not measure output quality; GGUF/desktop/single-GPU scope. Its hardware coverage breadth is something this board does not attempt.
• PocketPal — an on-phone GGUF app whose community leaderboard ranks devices (Glicko-2 over thousands of crowd benchmarks, recomputed every 6 h). Real crowd phones at a scale this board doesn't have — but it ranks hardware speed, not model intelligence, and has no quality or retention axis.
• Geekbench AI (Primate Labs) — cross-platform (Android/iOS/Windows/macOS/Linux) AI benchmark of ten fixed workloads (eight computer vision, two NLP classification/translation) scored at single/half/quantized precision with accuracy reported alongside speed. No LLM text generation, no model comparison (it compares devices on fixed models) — orthogonal to a model leaderboard, listed for completeness.
• Mobile-Bench (ACL 2024) — despite the name, an orthogonal axis: it benchmarks LLM-based mobile agents (UI + API task completion, CheckPoint metric), not on-device inference. Listed to avoid name confusion.

Lineage on the quality side. The Open LLM Leaderboard (2M+ unique visitors before HF archived it) established the "same harness, many models" quality board — on server GPUs, fp16/bf16, with no device axis; its retirement note cites benchmark saturation and the risk of "optimizing in irrelevant directions", which is exactly the contamination/goodharting problem the contamination section addresses. Chatbot Arena ranks by human votes — measurement here is deliberately not voting, which is why this project avoids the word "arena". LiveBench demonstrates the rotation defense this board's battery versioning follows: ~1/6 of questions replaced monthly, fully refreshed in ~6 months, with verifiable ground truth. DeviceMark applies that lineage to the on-device setting, where the artifact (quantized bundle), the runtime, and the physical device are part of what is being measured.

Per-entry proof strength (as of 2026-07-13)

The parity claim is per-entry, not global. Speed gates below are PipelinedBench greedy token-match runs on the same iPhone 17 Pro (iOS 27.0 24A5355q); quality runs are the full 596-item battery on the Mac engine (macOS 27 beta3 26A5378j, b2 toolchain, 2026-07-12/13).

Raw per-item outputs (model answers + our scores, no benchmark questions redistributed) are published in the results dataset under raw/, so every number on this page can be re-scored independently.

Benchmark data & licenses

The published dataset ships results only — scores, counts, and confidence intervals. Benchmark questions and gold answers are not redistributed; each benchmark keeps its own license: IFEval (Apache-2.0), MMLU-Pro (MIT), MATH-500 (MIT, from the Hendrycks MATH set). GPQA-Diamond is excluded from this tier entirely (floor effect at ≤1B, and its gated terms of use). The results tables are published under CC-BY-4.0 as a Hugging Face dataset.

Environment note (why bundles are re-exported locally)

The measuring box is on macOS 27 beta3, which requires b2-versioned IR; pre-2026-07-09 bundles are un-loadable. Every entry's ① bundle is re-exported locally with the b2 toolchain via the zoo's own recipe (zoo_convert.py) — bit-identical weights, just re-emitted IR.

---

Draft, v0. This page is the authoritative statement of method; the board links here from its footnotes.