Pipetteとは？スマホのAI性能を実機で測定する新しいベンチマーク

Today, in partnership with Artificial Analysis , we release Pipette , an open-source platform for benchmarking foundation models on edge devices. ／ Pipette is built around a simple premise: on-device behavior is a property of the deployed system, not the model in isolation. ／ It contains five on-device performance metrics for more than 1,000 model × quantization × runtime × device × context configurations. The current data spans 30+ models, multiple quantization formats, llama.cpp builds for macOS, iOS, Windows, and Android, and context lengths from 256 to 8,192 tokens. Initial published results cover MacBook Pro with M5 Max, iPhone 17 Pro, and Galaxy S26 Ultra ／ Publication of community-submitted results is currently in beta. — 冒頭の公開内容に関する記述より

Performance benchmarks follow Pipette's published measurement methodology : timing and memory runs use fixed token shapes, greedy decoding, a discarded warm-up, five measured repetitions, and readiness gating. Pipette's methodology has been reviewed and verified by Artificial Analysis . ／ Before each timed repetition, a platform-specific readiness check verifies acceptable thermal and load conditions. Pipette publishes results only for runs that pass this check. — How Pipette Works 節より

We have independently validated Liquid AI's inference benchmarking methodology — What we're launching 節より

Intelligence is measured by a set of five evaluations selected to represent the types of function/tool calling, knowledge and reasoning tasks small models are often asked to accomplish. ／ Nanbeige4.2-3B and LFM2.5-2.6B share the top average evaluation scores (with a 16K context limit) at launch, both scoring 63, ahead of Ornith-1.0-9B on 62 and Qwen3.5 9B (Reasoning) on 61. ／ on an iPhone 17 Pro it answers a standard 1,024-token prompt in 8.0 seconds using 2.3 GB of memory, against 21.4 seconds and 4.0 GB for Nanbeige4.2-3B and more than 25 seconds and 6.9 GB for the two 9B models. ／ Capping each answer at what a model can generate in 60 seconds on the iPhone 17 Pro (its output speed × 60, ignoring prompt processing time) puts LFM2.5-8B-A1B first on 47, followed by LFM2-2.6B-Exp on 45 and Gemma 4 E4B (Non-reasoning) on 44. LFM2.5-2.6B falls to 38, and Nanbeige4.2-3B to 18: at 14 output tokens per second it gets about 850 tokens per answer, so most of its reasoning never finishes. ／ While we aren't using the one-minute limit as the primary filter for this benchmark data, it may be more in line with the maximum wait time mobile users are willing to tolerate. — Launch results for the iPhone 17 Pro 節より

At Q4_K_M on Galaxy S26 Ultra, Granite-4.0-H-350M retains 78.4% of its decode throughput from 256 to 4,096 input tokens, while Granite-4.0-350M retains only 33.8%. ／ At Q4_K_M and 2,048 input tokens on M5 Max, Granite-4.1-8B and Ministral-3-8B-Instruct-2512 differ by only 2.4% in decode throughput and 1.2% in peak RAM. In quality evaluations of the same Q4_K_M artifacts, Granite leads IFBench by 7.3 points while Ministral leads GPQA Diamond by 14.0 points. ／ Pipette keeps these tradeoffs separate and inspectable rather than collapsing deployment readiness into a single ranking. — 本文の構成比較の記述より

In this initial release, no NPU path supports enough of the published model set to provide consistent comparisons across model classes, so NPU results are not included. ／ Under the current benchmark setup and model coverage, none of the stable GPU backends tested on Android consistently outperformed the selected CPU path across the full model set. ／ We do not currently recommend interpreting cross-device results as controlled hardware comparisons. ／ Current Pipette results are most reliable for comparing configurations within the same device. — Limitations and Future Directions 節より