On-Device LLM Leaderboard — intelligence × decode speed, measured under real device limits, with Apple's built-in Foundation Model on the board as sea level.
v0 finding: the top is a statistical tie — the built-in model does not clearly beat the best open 1–2B ports. And Google's official QAT int4 measures at parity with its bf16 checkpoint on MMLU-Pro/MATH.
y = intelligence composite (item-bootstrap mean of IFEval mean-of-4, MMLU-Pro & MATH completed-only accuracy) with 95% CI whiskers. x = decode tok/s, device-measured on iPhone 17 Pro (PipelinedBench, numerics-gated; per-entry gates in proofs). Dashed line = Apple's built-in Foundation Model (no bundle to load). Memory per model is in the table below.
Follows written instructions exactly — e.g. "write 300+ words and never use a comma". Checked programmatically (official checkers); no answer key to memorize.
Expert 10-choice questions across 14 fields (law, medicine, CS, …). Score = % correct of the questions it finished within budget.
Competition math — must produce the exact final answer (symbolically checked). Score = % correct of finished items.
What the model can do, absolute. Different from ③ retention, which is quantized ÷ float of the same model — what shipping it cost.