← back to the leaderboard

Methodology

On-Device LLM Leaderboard · v0, iPhone tier

Proof strength is disclosed honestly: where a column rests on a proxy, it says so.

What the leaderboard measures

"Of the models that actually run on this phone, which is the smartest that stays usable?" — the intelligence × decode-speed Pareto front under real device limits. Intelligence is measured on Mac (fast) and transferred to the device by the zoo parity gate (device ≡ Mac ≡ HF greedy token-exact); only decode tok/s is measured on the device itself. Peak memory is a measured device number on two rows and the bundle's download size on the rest (mem_measured says which); power is not measured at all — see Speed / memory / power. Measuring smarts at ~200 tok/s on a Mac instead of dying at ~20 tok/s on the phone is only honest because the parity gate proves the phone emits the same tokens.

The three columns per entry

Quantization variants (int8 vs int4) are separate entries, never mixed. ① is always measured with each model's shipped recipe default (the quant actually in the ModelStore bundle), not a softer fallback.

② baseline — implementation decision, and its proof strength

The v0 family is Qwen3.5 (0.8B / 2B). Its checkpoint is a vision-language model with a hybrid linear-attention (gated-delta / SSM) + sparse full-attention text backbone (model_type: qwen3_5, mostly linear_attention with full_attention every 4th layer, MTP head, mrope). Three baselines were considered:

• (a) third-party HF transformers (bf16) — the most authoritative, but not available: transformers 4.57.6 has no qwen3_5 model_type (only qwen3, qwen3_moe, qwen3_next, qwen3_vl*). It existed only in the unreleased 4.57.0.dev0 the checkpoint was saved with, and the repo ships no custom modeling code. Chasing a fork for a VL+SSM hybrid is out of scope for v0.
• (b) coreai_models eager reference (float) — chosen. Qwen3_5StatefulForCausalLM is coreai_models' own PyTorch implementation of this exact architecture — the same definition that is quantized+exported for ①. Run unquantized (fp16), greedy, it is a genuine pre-quantization reference. Harness = ref_eval.py.
• (c) fp16-on-engine proxy — the fp16 Core AI bundle. Fast but still engine-compiled, so it isolates only int8-vs-fp16 weights on the same engine. Kept as a secondary cross-check, and validated to agree with (b) eager and, for LFM2.5, with (a) transformers — so it is used as the uniform fleet baseline with eager/ transformers anchors.

Two-tier cost design. The eager reference decodes at ~11 tok/s on the Mac GPU — ~18× slower than the engine — so it is not run on the full battery. Column ① runs the full battery; column ② runs a stratified subset, and retention ③ is computed on those shared keys. Subset size is disclosed per entry.

Evaluation battery v0

Greedy (deterministic), cap = 4096 output tokens on every downloaded model, counted in tokens on every runtime (changelog: a harness that capped in characters was fixed, and four rows still labelled 1024 were corrected to the 4096 they had actually run at). A problem that produces no answer within the budget is scored wrong and stays in the denominator, so a model cannot gain by giving up — see Neutrality.

The built-in Foundation Model row ran at 1024, because its driver predates the change. It makes no difference and the raw results show why: across all 599 responses (IFEval 301, MMLU-Pro 197, MATH 101) not one was truncated — the model never approached even the lower limit, and its accuracy is flat at every budget from 128 tokens upward. Stated here rather than quietly equalised, because it is the first thing worth checking about a row that ranks against models given four times the budget. 0-shot, chat template with thinking OFF (verified: on-device ChatSession runs thinking-off). Prompts instruct step-by-step reasoning in the visible answer.

GPQA-Diamond is excluded from the iPhone tier: at ≤1B it sits at/below chance (13.3% vs 25% chance, 60% never converge within cap) = floor effect, and it is gated (ToS). It moves to the Mac / v1 tier. Benchmark contamination is a real risk; the battery is versioned (battery_version) and rotated.

The cap lever + the retention confound (documented, not hidden)

max-tokens is an explicit design lever. Verbose small models spend the whole budget reasoning — the effect measured on an earlier 1024-token configuration, kept here because it is what motivated the uniform 4096 budget the board now runs: qwen3.5-0.8B int8 at cap-1024 on MMLU-Pro caps 69/196 and produces no boxed answer in 65/196 — those score as wrong (a fair "couldn't deliver a usable answer on-device in-budget" signal). Larger / less-verbose models cap far less.

Finding: ① and ② cap at different points (greedy paths diverge — int8 vs float weights, engine vs eager), so their no-answer rates differ and the retention ratio ③ can be dominated by cap-timing, not quantization quality — it can even exceed 100% at small n. On shared answered problems the two emit identical boxed answers. Two mitigations, both reported:

1. Two accuracies per bench: acc (no-answer = wrong; the on-device "usable" number) and acc_completed (only problems that produced an answer; cap-independent quality). Retention ③ is reported on acc_completed so it reflects quantization damage, not verbosity; acc + no-answer rate carry the usability story. When retention is within noise (≥0.97 or CI-consistent) the board shows "≈100% ▵" and keeps the raw value in the data.
2. Larger n + a per-tier cap: raise the cap when no-answer is the dominant term.

Confidence intervals & ranking

Reproducibility. Every number regenerates bit-identically from the raw per-item outputs: the scorer pins PYTHONHASHSEED, seeds langdetect, and seeds Python's random. The last one matters because the official IFEval checkers are not reproducible out of the box: upstream google/IFEval items 1122 and 1129 carry non-alphabet "letters" ('#', '!') in keywords:letter_frequency kwargs, and the official checker replaces those with an unseeded random letter on every run — a ~±0.2pp jitter that affects anyone using the official scorer unseeded. We keep the official semantics (seeded) and disclose the two items.

Every accuracy carries a 95% interval, so ranks are not read off noise:

• Per-bench accuracy: Wilson 95% CI.
• The intelligence composite (mean of the three per-bench means): item-bootstrap, 2000 resamples, 95% interval. Shown as ±half-width in the table and as whiskers on the Pareto chart.
• Rank = 1 + (number of models strictly better), where "strictly better" means a model's CI lower bound is above this model's CI upper bound. Overlapping CIs are a statistical tie (marked =). Subset runs leave the top group tied; the full 596-item battery separates them.

Speed / memory / power

PipelinedBench (STATS prefill/decode) on iPhone 17 Pro / M4 Max. tok/s for v0; power (tokens/joule) and a single-protocol peak-memory sweep are v0.5 axes. This box ≈ M4 Max: 0.8B int8 = 207 tok/s decode, matching the published 204. The board's size column is the shipped bundle's download size (exact bytes on the hosting hub, checked 2026-08-20); measured device peaks, where they exist, are in the dataset's measurements table. Decode is shown as Mac (○) until an iPhone number exists. Reported tok/s are warm-state: engine loaded and warmed, cold load and first-run specialization excluded; 128-token prompt / 256-token decode, two trials on a settled device. The built-in Foundation Model exposes no token counts, so its decode speed is published as an estimate (▵): fm-speed streams 8 long-form generations (max 512 tokens, greedy, fresh session each; 2 warmups discarded) and measures sustained chars/s from chunk arrival times, TTFC/prefill excluded — 311.4 chars/s median on iPhone 17 Pro (range 275–345; on-device via PipelinedBench PB_FM_SPEED) and 220.3 chars/s on M4 Max (range 198–232), TTFC 0.42 s median on both, 2026-07-15. tok/s is then derived through the empirical chars-per-token band of the 7 open ports’ full596 outputs (per-model medians 3.61–4.43, ≈4,700 generations): ≈70–86 tok/s on iPhone, ≈50–61 on Mac. A band, not a point, because the FM’s tokenizer is unknown — the spread is the systematic uncertainty, disclosed rather than hidden. (Two honest curiosities, also disclosed: the phone measures faster than the Mac on this protocol, and the iPhone per-run rates drift down 345→275 across the 8 runs — thermal, visible in the raw timings.) Raw chunk timings: raw/fm_speed_mac.jsonl + raw/fm_speed_iphone.jsonl in the dataset.

Runtime-neutral schema (day-1 multi-runtime)

Quality is an attribute of the artifact (model × quant × format), measured on its native runtime + float retention. Speed/memory/power are attributes of runtime × device. Two normalized, HF-dataset-ready tables: artifacts.jsonl and measurements.jsonl. v0 data is Core AI only (format=aimodel, runtime=coreai), but GGUF/llama.cpp and MLX/mlx-lm rows drop in as pure data additions.

Apple's built-in Foundation Model as a board row

Apple's on-device system model (SystemLanguageModel via LanguageModelSession()) runs the same battery through its public API — no weight extraction (runtime=foundation-models, format=system, DL size 0, OS-resident, retention=null). It runs the full 596-item battery through the same scorers with no refusals; as a closed system model it has no float baseline, so retention (③) is N/A. Its per-bench scores and 95% CIs sit in the board like every other row — we do not editorialize its ranking; where CIs overlap the board marks a tie and refuses to order within it. Correction note kept as a worked example of small-n noise: an earlier n=40 subset showed the FM "weak on IFEval" (67%); the full 300-item battery did not reproduce that — which is why the board reports CIs.

Refusal rate (a column for all, decisive for the built-in)

guardrailViolation and explicit textual refusals are separated from wrong answers (refusal_rate; accuracy is computed over non-refused items). Near-zero on the general battery for every model, but the column matters the moment a bench probes guarded topics.

Why the composite includes IFEval — reasoning-heavy ≠ instruction-following ≠ on-device-practical

A model can be strong on knowledge/reasoning yet weak at being useful on a phone. Qwen3.5-4B is the clearest case: it reasons at length — generating ~4000 tokens even at the 4096-token budget and often prepending a "Thinking Process:" preamble that isn't the requested output — so it lands near the bottom on IFEval and delivers a parseable final answer on only a fraction of items, despite competitive raw MMLU-Pro accuracy when it does answer. The composite includes IFEval and scores no-answer as wrong on purpose: raw knowledge is not the same as handing back a usable answer within a pocket-sized budget.

Fairness & corrections

Publishing another model's score carries a duty of fairness. Every entry links its parity proof; if we measured the wrong quant, or a port lost parity and the number reflects a broken export rather than the model, model owners can contest it via the objection issue template. Only open-weight models are ported; the only closed system on the board is Apple's built-in FM, via its public API.

Neutrality & limitations — what the board is an opinion about

The measurement is neutral: every model runs the identical 596-item battery through the identical scorers, greedy (temperature 0, deterministic), under one shared budget (4096 output tokens), and every raw output is published for re-scoring — no per-model tuning. But several choices are opinions, and they push the results in known directions. Disclosed here so a reader can judge for themselves.

• The board measures on-device delivery, not raw capability. A no-answer, an over-cap ramble, or an answer not in the requested format is scored wrong and kept in the denominator. This rewards models that reason and hand back a parseable answer within a fixed on-device budget, and penalizes verbose or format-loose models even when their reasoning is strong. Example: Gemma 4 E2B scores 85/100 on GSM8K on its native runtime, but lower on the capped battery because it overflows the budget on harder items — that gap is delivery, not intelligence, which is exactly what the per-row diagnostics (answered %, generated tokens) exist to show.
• The cap is a use-case choice. 4096 tokens models a tight on-device budget: a concise assistant fits; a reasoning deployment that lets the model think would give it more. Applied uniformly, but it favors concise models over verbose ones by design.
• Answer format is required. MATH is scored on a \boxed{} final answer (the standard MATH-dataset convention, prompted identically for every model); MMLU-Pro accepts several answer-letter phrasings. A model that reasons correctly but ignores the format loses the point — a deliberate instruction-following component of "delivery."
• Each model is measured on its own native runtime, not a common one. Quality can be runtime-dependent (e.g. Gemma's quantization is co-designed with LiteRT-LM's activation path — the same weights score ~85% there and ~48% on an fp16-activation runtime). The runtime is disclosed per row; cross-model comparison carries that wrinkle.
• Coverage is a selection, not a census. The board holds the on-device ports the author has built (plus Apple's built-in FM and cloud reference lines), not every on-device model. Which models appear — and the choice to measure "delivery" — are the author's decisions. The safeguards against that bias are the ones above: one identical battery and scorer set, published raw outputs, confidence intervals, and no editorializing of who wins.

Bottom line: read the board as "who delivers a correct, parseable answer within a fixed on-device budget" — a specific, honest question — not as a raw-intelligence ranking. Labelled that way it is fair; read as raw IQ it is not.

Contamination — which columns can and cannot be gamed

Public benchmarks may appear in any model's training data, and this is not verifiable from outside. The board's columns differ sharply in how much that matters:

• Immune — retention ③, tok/s, memory, energy. Retention compares the quantized and float versions of the same weights: whatever the model memorized, both sides memorized equally, so contamination cancels and only quantization damage remains. Speed/memory/energy are physical measurements.
• Resistant by construction — IFEval. Scoring is programmatic constraint checking on freshly generated output (word counts, forbidden characters, formats). There are no gold answers to memorize; a model must actually follow the instruction at inference time.
• Vulnerable — MMLU-Pro and MATH. Gold answers are memorizable. A model fine-tuned on the test set (deliberately or via crawl accident) inflates these two columns.

Mitigations, in order of deployment: the battery is versioned and rotated (each rotation bumps battery_version); a paraphrase probe is planned — the same items rewritten with different wording/numbers, scored pairwise, so a memorizer shows a public>paraphrase gap that genuine capability does not; and the roadmap's endpoint is self-authored, freshly-captured test sets that are contamination-free by construction. Submissions suspected of benchmark tuning are checked with the probe before listing.

Cloud sea-level lines (for scale)

To show how far the pocket tier is from a data-center model, two cloud APIs run the identical 596-item battery through the same scorers at temperature 0: Gemini Flash (mini-class) and Gemini Pro (frontier). They are drawn as horizontal reference lines, not ranked device entries — a cloud model has no on-device tok/s, memory, or quantization to measure, so those columns are n/a. Both land at ~93% composite (a tie), versus ~76% for the built-in FM and ~72% for the best pocket port: that gap is the price of fitting on the phone. Protocol caveat: cloud models do their own internal reasoning and are not device-budget-capped, so a generous output cap (8192) is used; a handful of items where the model never stopped are excluded (completed-only, same as every row). Raw cloud outputs ship in the dataset raw/ alongside the device rows. Model snapshots: gemini-3.5-flash / gemini-3.1-pro-preview, measured 2026-07-14.

Anchored-ops track (preview)

The chat battery above measures what a model knows. The anchored-ops track measures what it can do with device data: 48 single-turn chores across 7 skill clusters — compute a cross-time-zone due time, split a bill and put the exact amount in a reminder, album only the matching screenshots, extract only my open action items from a meeting transcript, merge scattered notes keeping the latest values, copy names/rooms/derived times into arguments verbatim, and create exactly one prep reminder per active calendar event.

Single source. Every task is compiled from the agent task suite's task definitions (the same JSON files run multi-turn in the agent harness); the world data the agent would gather through read tools is inlined into one prompt. Gold values are computed programmatically (time-zone math via the IANA database, money via exact decimal arithmetic), so the reference solutions and checks cannot drift from the task parameters. Before any model ran, every task passed a harness self-test: the reference solution must pass and a do-nothing run must fail, under the real checker.

Deterministic scoring. The model must reply with a JSON call list ({"calls": [{"tool": …, "args": …}]} — the same shape as agent-harness traces). Checks are deterministic: case-insensitive substrings, ISO-8601 instant equality (+09:00 and -07:00 spellings of the same moment are equal), exact counts and item sets. No judge model, no partial credit; an unparsable reply fails every check and stays in the denominator. MACRO = the mean of the 7 cluster pass-rates (each cluster weighs the same regardless of task count), so a model can't farm one easy skill. Per-model hover shows task-pass with its 95% CI, the parse rate, and the field-level rate.

Conditions & preview status. Same shipped artifacts as the board, Mac engine, greedy decoding, 4096-token cap (thinking counts against it; declared for every row). Preview means: not part of the intelligence composite, n is still small (CIs on hover), and iPhone runs are pending — the label comes off when on-device numbers land. Battery and scorer publish with the results dataset.

Related work — what this board is and is not

Where DeviceMark sits among the tools that measure local/on-device AI. The matrix uses each project's own published scope (links below); "intelligence axis" means output quality is a reported, per-model comparison column, not merely an internal pass/fail check.

• MobileAIBench (Salesforce, 2024) — the closest conceptual neighbor: LLM/LMM task performance across quantization levels for mobile-sized models, with "a library for running evaluations on desktops and an iOS app for on-device latency and hardware utilization" (their split, quoted). Its task coverage (multimodal, trust & safety) is broader than this board's. Differences: it evaluates models the harness re-quantizes, with quality measured on the desktop side and no token-level proof that the phone runs the same computation; DeviceMark scores the shipped artifact on its native runtime and transfers quality to the device through a greedy token-parity gate, adds retention vs the same-weights float baseline, the built-in Apple FM row, CI-based ties, and a continuously updated board rather than a paper snapshot.
• MLPerf Client (MLCommons, v1.6, 2026) — the industry-standard client-side performance benchmark: TTFT and tokens/s on int4 models, now on Windows, macOS Tahoe 26, iPadOS 26 and Ubuntu. Its standardization and vendor governance are far stronger than this board's. Quality appears only as a qualification threshold (models must clear an MMLU score of 43–70 to be included) — it is not a reported per-system comparison axis, there is no phone tier, no memory/retention column, and no per-model intelligence ranking. DeviceMark is complementary: the quality×speed join on the iPhone tier.
• LocalScore (Mozilla Builders) — a llamafile/llama.cpp desktop benchmark combining prompt speed, generation speed and TTFT into one score, with a crowd-sourced hardware database. By its own docs it does not measure output quality; GGUF/desktop/single-GPU scope. Its hardware coverage breadth is something this board does not attempt.
• PocketPal — an on-phone GGUF app whose community leaderboard ranks devices (Glicko-2 over thousands of crowd benchmarks, recomputed every 6 h). Real crowd phones at a scale this board doesn't have — but it ranks hardware speed, not model intelligence, and has no quality or retention axis.
• Geekbench AI (Primate Labs) — cross-platform (Android/iOS/Windows/macOS/Linux) AI benchmark of ten fixed workloads (eight computer vision, two NLP classification/translation) scored at single/half/quantized precision with accuracy reported alongside speed. No LLM text generation, no model comparison (it compares devices on fixed models) — orthogonal to a model leaderboard, listed for completeness.
• Mobile-Bench (ACL 2024) — despite the name, an orthogonal axis: it benchmarks LLM-based mobile agents (UI + API task completion, CheckPoint metric), not on-device inference. Listed to avoid name confusion.

Lineage on the quality side. The Open LLM Leaderboard (2M+ unique visitors before HF archived it) established the "same harness, many models" quality board — on server GPUs, fp16/bf16, with no device axis; its retirement note cites benchmark saturation and the risk of "optimizing in irrelevant directions", which is exactly the contamination/goodharting problem the contamination section addresses. Chatbot Arena ranks by human votes — measurement here is deliberately not voting, which is why this project avoids the word "arena". LiveBench demonstrates the rotation defense this board's battery versioning follows: ~1/6 of questions replaced monthly, fully refreshed in ~6 months, with verifiable ground truth. DeviceMark applies that lineage to the on-device setting, where the artifact (quantized bundle), the runtime, and the physical device are part of what is being measured.

Per-entry proof strength (as of 2026-07-13)

The parity claim is per-entry, not global. Speed gates below are PipelinedBench greedy token-match runs on the same iPhone 17 Pro (iOS 27.0 24A5355q); quality runs are the full 596-item battery on the Mac engine (macOS 27 beta3 26A5378j, b2 toolchain, 2026-07-12/13).

Raw per-item outputs (model answers + our scores, no benchmark questions redistributed) are published in the results dataset under raw/, so every number on this page can be re-scored independently.

Benchmark data & licenses

The published dataset ships results only — scores, counts, and confidence intervals. Benchmark questions and gold answers are not redistributed; each benchmark keeps its own license: IFEval (Apache-2.0), MMLU-Pro (MIT), MATH-500 (MIT, from the Hendrycks MATH set). GPQA-Diamond is excluded from this tier entirely (floor effect at ≤1B, and its gated terms of use). The results tables are published under CC-BY-4.0 as a Hugging Face dataset.

Environment note (why bundles are re-exported locally)

The measuring box is on macOS 27 beta3, which requires b2-versioned IR; pre-2026-07-09 bundles are un-loadable. Every entry's ① bundle is re-exported locally with the b2 toolchain via the zoo's own recipe (zoo_convert.py) — bit-identical weights, just re-emitted IR.

---

Draft, v0. This page is the authoritative statement of method; the board links here from its footnotes.