Liquid AI Pipette: Mobile AI Benchmark for Phones

Pipette is Liquid AI's open-source mobile AI benchmark

Pipette is an open-source platform from Liquid AI for benchmarking AI models on edge devices such as phones and laptops. Liquid AI released it on August 24, 2026 in partnership with Artificial Analysis. The benchmark targets small language models that run on phones, laptops, and embedded devices.

Context length, used in the table below, is the number of tokens the model reads before it starts to answer. The table below lists what shipped with the release.

What shipped with Pipette

Item	Detail
Dataset	Five on-device performance metrics for more than 1,000 configurations
Models	More than 30, in several quantization formats
Runtimes	llama.cpp builds for macOS, iOS, Windows, and Android
Context lengths	256 to 8,192 tokens
Devices	MacBook Pro with M5 Max, iPhone 17 Pro, Galaxy S26 Ultra
Clients	Open-source clients for macOS, Windows, iOS, and Android
Code	pipette-mgmt, pipette-clients, pipette-scores, under Apache 2.0
Dashboard	Interactive view of quality, throughput, latency, context scaling, and memory use

The Apache 2.0 license covers the pipeline code. The dataset is described as public.

Artificial Analysis combines Pipette's performance data with equally weighted quality evaluations chosen to represent real-world mobile use. Results for AMD Ryzen AI Max+ 395 and Radeon 8060S are listed as coming soon.

Pipette benchmarks model, quantization, runtime, and device

Pipette treats a model, its quantization, its runtime, and its device as one unit, because on-device behavior belongs to the whole deployed system. Every measurement starts from a tested configuration of those four. A benchmark then fixes the metric and the token shape, and the result is a latency, throughput, or memory figure for that configuration.

Quantization shrinks a model's weights so that it fits in a phone's memory. Model cards usually report quality for the original full-precision weights, while phones run the shrunken versions, and the loss in quality varies by model, format, and task.

Pipette therefore scores the quantized files on IFBench, GPQA Diamond, and MATH-500. Where a full-precision FP16 or BF16 result exists, it serves as the reference, so the score shows how much capability survives quantization.

The quality scores come from llama.cpp runs on NVIDIA H100 80GB reference systems, not from the phones. The dashboard pairs each score with compatible speed and memory measurements taken on the device. Pipette keeps speed, memory, and quality as separate readings instead of folding them into one ranking.

How Pipette keeps its phone benchmarks reproducible

Pipette fixes the test conditions and publishes a speed result only when the phone was in an acceptable state during the run. Speed and memory runs use fixed token shapes, greedy decoding (always choosing the most likely next token), a discarded warm-up pass, and five measured repetitions. Before each timed repetition, a readiness check confirms acceptable thermal and load conditions, and runs that fail the check are not published.

Quality evaluations follow a separate protocol. They use standard datasets, generate answers through a reference runner, and score them deterministically without knowing which model produced them.

Three open-source components carry this out. Each has a separate job:

• pipette-mgmt serves the versioned benchmark catalog and ingests submissions.
• pipette-clients runs benchmarks on the target device and generates the answers to be scored.
• pipette-scores supplies the evaluation prompts and scores the answers without access to how they were generated.

Each submission records the benchmark and token shape, the model file and quantization, the runtime version and settings, and the device hardware and operating system. Results from different conditions stay separate, and the basis of each comparison is visible. Liquid AI says Artificial Analysis reviewed and verified the performance methodology.

Small language models in Pipette: the first results

Liquid AI highlights four results where parameter count and model size fail to predict how a model behaves on a device. Each one concerns a pair of models that look alike on paper and behave differently in the measurements. Decode throughput, the speed at which a model generates new tokens, is the speed measure used throughout.

Two 350M models scale very differently with input length

On a Galaxy S26 Ultra, Granite-4.0-H-350M keeps far more of its decode throughput than Granite-4.0-350M as the input grows. At Q4_K_M quantization, the first model retains 78.4% of its decode throughput between 256 and 4,096 input tokens, and the second retains 33.8%. Both have 350 million parameters, so the size label does not tell a reader which one slows down on long inputs.

Decode throughput kept from 256 to 4,096 input tokens

Bars show the share of decode throughput each model keeps on a Galaxy S26 Ultra at Q4_K_M. Source: Liquid AI.

78.4%Granite-4.0-H-350M

33.8%Granite-4.0-350M

0204060%

Sparse activation is fast but doesn't shrink memory use

LFM2.5-8B-A1B decodes faster than much smaller dense models, but its memory use stays that of an 8B model. At 2,048 input tokens on a Galaxy S26 Ultra, it decodes 2.4x faster than Qwen3.5-4B and 2.6x faster than Ministral-3B-Instruct-2512.

The model activates only 1.5B of its 8.5B parameters for each token, which explains the speed. It still peaks at 5.29 GiB, because every expert weight counts toward the memory the model needs.

Two small iPhone models split on speed and quality

On an iPhone, MiniCPM5-1B finishes a fixed workload sooner, while LFM2.5-1.2B-Instruct scores higher on mathematics. The workload is 2,048 input tokens and 256 output tokens at Q4_K_M. MiniCPM5-1B completes it in 3.47 seconds against 4.12 seconds, a 15.8% reduction in elapsed time.

In the quality evaluation of the same Q4_K_M files, LFM2.5-1.2B-Instruct scores 9.0 points higher on MATH-500. Neither configuration is ahead on both axes, so the choice depends on which axis matters for the application.

Time to finish 2,048 input and 256 output tokens on an iPhone

Bars are in seconds and scaled to the longer time. Q4_K_M. Source: Liquid AI.

3.47MiniCPM5-1B

4.12LFM2.5-1.2B-Instruct

024

Granite and Ministral split on task quality

Two 8B models with almost the same speed and memory profile lead on different tasks. On an M5 Max at Q4_K_M and 2,048 input tokens, Granite-4.1-8B and Ministral-3-8B-Instruct-2512 differ by only 2.4% in decode throughput and 1.2% in peak RAM.

In the quality evaluations of the same files, Granite leads IFBench by 7.3 points, while Ministral leads GPQA Diamond by 14.0 points. IFBench measures instruction following and GPQA Diamond measures science reasoning. A pair with matching deployment profiles can therefore still differ on the task that decides the choice.

Where the first Pipette release falls short

Liquid AI lists four limits of this release: no NPU results, a CPU-focused Android path, no controlled cross-device comparison, and a narrow quality suite. Liquid AI frames them as the scope within which the current results should be read.

• NPUs are not covered. NPU support depends on model-specific kernels and operators. No NPU path supports enough of the published models to give consistent comparisons, so NPU results are left out.
• Android runs on the CPU. None of the stable Android GPU backends tested consistently beat the selected CPU path across the full model set. The llama.cpp Android path is therefore CPU-focused, while the iPhone path uses the Metal backend.
• Cross-device numbers are not hardware comparisons. Android and iOS runs differ in flash-attention support, thread counts, accelerator use, and execution environment. Results are most reliable when comparing configurations within one device.
• The quality suite is narrow. It covers instruction following, science reasoning, and competition mathematics. It does not fully cover agentic behavior, knowledge-intensive tasks, or multimodal workloads.

Running Pipette on your own phone or laptop

The Pipette dashboard lets a reader filter the published results by the device they plan to target, and the open-source clients let them add measurements of their own. The dashboard has three surfaces: a per-device Leaderboard, a Results explorer with configuration-level tables, and a Submissions browser with the records behind each result. Data can be narrowed by model, quantization, runtime, device, benchmark, context length, and thinking mode.

The comparison charts plot IFBench, GPQA Diamond, or MATH-500 against time to first token, end-to-end latency, prefill throughput, decode throughput, or peak RAM. A constraints panel filters by minimum quality or throughput and maximum latency or memory. Every Leaderboard chart generates a shareable link that keeps its filters, so one comparison can be sent to a colleague as it is.

Anyone can run Pipette on their own hardware with the open-source clients, the native iOS app, or the Android app, which also makes it usable as a local LLM benchmark for a specific machine. Publication of community-submitted results is in beta. A missing model, runtime, device, or benchmark can be requested from Liquid AI.

More configurations will be added as they complete the measurement and publication process, so the dashboard changes over time. Saving the web page as Markdown keeps a record of the numbers as they stood on a given day.

A dated copy of the launch post helps when a figure is questioned later. To keep one, try the tool that converts a web page URL into Markdown.

LFM2.5-2.6B is one of the models Pipette launched with, and it is covered in Liquid AI LFM2.5-2.6B: Fast Agents, Not for Coding. Practical coding use of local models is covered in Local LLM Coding Now Hits 75% of Cloud AI.