The smartest LLMs that
actually run on an iPhone

On-Device LLM Leaderboard — intelligence × decode speed, measured under real device limits, with Apple's built-in Foundation Model on the board as sea level.

Every row runs the same 596-item battery on its native runtime, under one fixed on-device budget — cap 4096, greedy, and no-answer scored wrong. The score is what a model actually delivers within those limits; the per-row diagnostics (answered %, median generated tokens) show why a number lands where it does. Cloud APIs on the same battery are a reference ceiling, not ranked.

Pareto front — intelligence vs decode speed

y = intelligence composite (item-bootstrap mean of IFEval mean-of-4, MMLU-Pro & MATH completed-only accuracy) with 95% CI whiskers. x = decode tok/s, device-measured on iPhone 17 Pro (PipelinedBench, numerics-gated; per-entry gates in proofs). Dashed line = Apple's built-in Foundation Model (no bundle to load). Download size per model is in the table below.

The board — shipped artifacts, one protocol

hover any score for its own 95% CI and n
IFEval 300 items

Follows written instructions exactly — e.g. "write 300+ words and never use a comma". Checked programmatically (official checkers); no answer key to memorize.

MMLU-Pro 196 items

Expert 10-choice questions across 14 fields (law, medicine, CS, …). Score = % correct of the questions it finished within budget.

MATH 100 items

Competition math — must produce the exact final answer (symbolically checked). Score = % correct of finished items.

Intelligence = mean of the three

What the model can do, absolute. Different from ③ retention, which is quantized ÷ float of the same model — what shipping it cost.

Reading notes — metrics, symbols, caveats

See what the scores mean — real answers

one item per benchmark · every model's raw output · pass / fail as scored

Model authors: embed your row — devicemark.github.io/badge/<slug>.svg submit a model objection
Cite this board 
@misc{devicemark2026,
  title  = {DeviceMark: an on-device LLM leaderboard — quality, speed, memory,
            and retention for verified on-device ports},
  author = {Majima, Daisuke},
  year   = {2026},
  howpublished = {\url{https://devicemark.github.io/}},
  note   = {Results under CC-BY-4.0}
}