RollTab piano autocomplete: a 125M on-device model

What RollTab does and how fast it runs on an iPhone

RollTab is a free app for anyone with a MIDI keyboard and an iPhone or iPad. You play a few notes, and the model continues the piece.

Edwardsson's comparison is GitHub Copilot, but for piano. The idea started almost a year before the post, and fourteen experiments went by before he was happy enough with the result to write about it.

The post reports about 108 notes per second for the 125M-parameter model on an iPhone 15, and the final app runs entirely on-device. According to the author, the biggest gains came from three choices: the MIDI representation, aggressive data cleaning, and DPO post-training.

RollTab at a glance

Item	Detail
Model	Decoder-only transformer, about 125M parameters
Speed	About 108 notes/sec on an iPhone 15
Execution	Entirely on-device (Core ML, INT8 weights)
Training data	A few hundred thousand MIDI files, roughly 300M note events
Post-training	DPO on pairwise preferences scored by Gemini
Needs	A MIDI keyboard and an iPhone or iPad
Post date	20 August 2026

A MIDI file (MIDI is short for Musical Instrument Digital Interface) stores no recorded sound. It stores music as a sequence of events: a key pressed at a given pitch and velocity (how hard the key is struck), a key released, the sustain pedal changing state. Other events switch instruments or change volume, and the events are often organized into several tracks.

A pop or game file might carry melody, chords, bass, drums, strings, and several synth parts. This project focuses on piano continuation, so Edwardsson mostly kept piano-like material and removed or reduced the rest.

MIDI tokenization: why one token per note is fast

RollTab is fast because one musical note costs one pass through the model instead of about four. A transformer reads and writes tokens, which are discrete symbols from a fixed vocabulary, and it produces one token per pass. How many passes a note costs depends on how notes are turned into tokens.

The most obvious scheme gives every MIDI event its own token. Put pitch and velocity directly into a note-on token and the vocabulary can grow to 128 × 128 + 128 = 16,512 tokens, counting only note-on and note-off. Velocity could be bucketed, but many combinations stay rare, so the model has to learn structure from sparse tokens.

Why the note-on and note-off encodings drifted

Edwardsson tried three schemes before the final one, and each failed in a different way. Event tokens were slow, note-on and note-off events drifted, and explicit durations cost about four passes per note.

• Event tokens with grammar-factored fields: a note-on, a pitch and a velocity as separate tokens. The output spaces shrink to 128 pitches, about 16 velocities and about 100 durations. Invalid next tokens can be masked during generation, which guarantees valid output, but the stream was slow.
• Note-on and note-off events. These models tended to drift: they forgot to emit a note-off, left notes hanging, or lost track of which notes were active. He calls this especially bad for his target, a small model running close to real time on a laptop or phone.
• Explicit duration, written as a note with its pitch, velocity and duration plus a time-shift token. Drift went away and the music was better, but one note took roughly four autoregressive transformer steps and used up the context window quickly. Autoregressive means each step feeds on the output of the one before.

One token per note, with sustain folded into duration

The final design uses one token per note, so the backbone runs once per note and not once per field. The representation is NOTE(pitch, delta_onset, duration, velocity), with no separate time-shift event.

Silence is carried by delta_onset, the time since the previous note onset, on the next note. Timing uses 24 steps per quarter note, which covers common straight and triplet subdivisions, including live timing that lands almost but not quite on the beat.

Chords are several notes with a delta of zero, sorted by pitch. Sorting means training does not penalize one song for writing a C-major chord as CEG and another as EGC.

Each note has five categorical fields with their own vocabularies. Each field gets an embedding, and the note token is the sum of those embeddings. The model has a separate output head per field, and a small nested decoder lets later fields condition on earlier predicted fields.

The expensive transformer backbone runs once per note, not once per field. That turns four passes per note into one, and it is the source of the 108 notes per second on the iPhone.

The sustain pedal never became a token. During preprocessing, a note released while the pedal is down is extended to the pedal-up time, and a repeated pitch cuts the earlier note off. The pedal gesture is lost, but the model only has to predict pitch, onset, duration and velocity.

Training data: cleaning beat a 5x larger corpus

A corpus about 5x larger produced worse models, so cleaning and selecting the data came first. Edwardsson searched public collections of MIDI files, focusing mostly on older classical music in the public domain.

Quality varied wildly, so he wrote quite a few cleaning scripts. The final dataset had a few hundred thousand files and roughly 300 million note events.

He tried the 5x larger corpus hoping for better results. In his words, cleaning and selecting the data mattered more than adding more of it.

How the piano MIDI corpus was cleaned and deduplicated

The final pipeline had five steps. They ran in this order:

• Select piano-focused material.
• Remove or reduce pathological multi-track mixtures.
• Filter by density and by pitch and time coverage.
• Deduplicate with fingerprints that ignore global transposition and uniform tempo changes, so the same piece in another key or tempo counts once.
• Group alternate versions of the same composition into the same split, so that a variant of a training piece does not end up in the validation set.

Augmentation for notes that arrive early, late, or hard

Augmentation mattered because live input is not a pristine MIDI file. It comes from a person playing, so notes may be slightly early, late, or too hard. Edwardsson settled on four augmentations: global transposition, uniform tempo scaling, duration and velocity jitter, and dropped prompt notes.

The decoder-only transformer: 33M, 64M, and 125M sizes

The architecture is essentially a fairly standard decoder-only transformer: RMSNorm, rotary positional embeddings, causal self-attention, SwiGLU/MLP blocks, and autoregressive generation. Training starts as cross-entropy over the five output heads, summed, which makes it easy to track pitch, duration and velocity accuracy separately.

Cross-entropy has a limit for music. A held-out song supplies exactly one correct next note, while many continuations would work musically. The loss teaches the mechanics of music but is not a great proxy for how good a full continuation sounds.

Edwardsson trained three sizes, shown below. The small model suited quick experiments, and the medium one almost always beat it.

The large model did better again, though not by a huge margin. He is now trying to bring the medium model close to the large model's quality, mostly to reduce footprint and latency in the iOS app.

Model sizes trained (parameters)

Bars show parameters in millions, scaled to the 125M model as 100. Source: SimEdw's Blog.

M050100

33MSmall

64MMedium

125MLarge

Edwardsson says he deliberately avoided reading too deeply into music-generation papers at first, because he wanted to work through the problem himself. Afterward he compared his approach with recent transformer-based models for symbolic MIDI generation, which he lists as Aria, Moonbeam, MIDI-GPT, Anticipatory Music Transformer, PianoBART, and MIDI-LLM.

Scheduled sampling raised loss but improved continuations

Scheduled sampling made validation loss slightly worse but made the continuations better. Normally the duration and velocity predictions see the correct pitch during training. At inference they must work with whatever pitch the model actually predicted.

Scheduled sampling closes that gap: sometimes the model is fed its own predicted pitch instead. Edwardsson started at 0% for the first few epochs, then raised the rate gradually, up to 50% in the best model.

Validation loss got slightly worse, 2.9998 against 2.9495 without it. Continuations got better: Gemini preferred the scheduled-sampling model 64.3% to 35.7%.

Scheduled sampling at 50% against none

Validation loss (lower is better)

Bars are scaled to the higher loss as 100.

0123

2.9998Scheduled 50%

2.9495Without

Gemini preference (higher is better)

Bars show percent of pairwise judgments won.

%0204060

64.3%Scheduled 50%

35.7%Without

Direct preference optimization: preference up to 69.05%

Direct preference optimization (DPO) took pairwise preference from 24.55% for the pretrained base to 69.05%. It is a post-training step that learns from pairs of outputs.

For each prompt, Edwardsson generated several continuations and used pairwise evaluation to label one chosen and one rejected. DPO then makes the chosen continuation more likely than the rejected one, while keeping the model reasonably close to the original.

In his account, DPO made the biggest difference after pretraining. It took the model from occasionally producing a good continuation to doing so much more reliably. After DPO, more than 69% of continuations were preferred over the base model in his pairwise evaluation.

Pairwise Gemini judging beat absolute scores

Asking Gemini which of two continuations was better worked; absolute scores did not. At first, evaluation meant listening to continuations from held-out songs, with prompts of 4 to 32 notes. That was slow, and after a while everything sounded like noise.

He then wrote automatic metrics: repeated pitch n-grams, pitch entropy, pitch-class entropy, pitch range, note density, long pauses, and chord density. They caught obvious failures but could not select the best model.

Gemini 3.5 Flash became the judge, but a single absolute score from it was inconsistent. Asking which of two continuations was better worked much better, especially with every comparison mirrored to reduce position bias. In one test over 200 songs, Gemini gave the same preference 70% of the time after A and B were reversed.

Gemini also over-weighted how good a continuation sounded in isolation, and not how well it followed from the prompt. Edwardsson split the evaluation into a continuation score and a sounds-good score, and used the continuation score as the main signal for DPO.

The β sweep and the consensus dataset

The consensus dataset at β=0.03 gave the best result, 69.05%. The β value controls how strongly DPO penalizes moving away from the base model. In the sweep, β=0.01 and β=0.03 improved the model, and β=0.10 pushed too hard and made it worse.

A "consensus" dataset kept only the preference pairs where the evaluator agreed consistently, instead of trusting every noisy judgment. That gave the best result. Edwardsson's gut feeling is that the base model had already learned a reasonable picture of music, just not what makes a good continuation.

Pairwise preference after DPO

Bars show percent of continuations preferred by Gemini in pairwise scoring. Source: SimEdw's Blog.

24.55%Pretrained base

61.08%β = 0.01

57.14%β = 0.03

38.10%β = 0.10

69.05%Consensus (β = 0.03)

0204060%

Core ML quantization: INT8 weights and a 384-note window

Edwardsson exported the PyTorch model to Core ML, Apple's format for running models on its devices, and quantized the weights to INT8. Quantization stores each weight as an 8-bit integer instead of a longer number, which shrinks the model. The first launch is still annoyingly slow while Apple's runtime optimizes the model for the available hardware.

The model was trained only on contexts of up to 512 notes, but he wanted longer sessions. Whenever the context gets close to that limit, the app keeps the most recent 384 notes, rebuilds the context from them, and continues.

This means rebuilding the KV cache, the stored attention state that spares the model from reprocessing earlier notes. The model is fast enough that the rebuild has not been a major problem.

Because the model uses RoPE for positional encoding, a neater solution is possible in theory, with shifted positions and a ring buffer. Core ML does not expose Q, K, and V (the query, key and value tensors of attention) directly, so Edwardsson left it there and was mostly happy that it worked.

What failed and where RollTab still struggles

Edwardsson lists what did not work next to what did, and says RollTab is still far from perfect. Everything in the table comes from the post.

What worked, and what did not

Worked	Did not work
One token per note	Note-on and note-off encodings drifted too much for small real-time models
Cleaning and selecting the data	Grammar-masked token streams were valid but slow
DPO with a consensus dataset at β=0.03	Broader data made results worse when the data was noisy
Scheduled sampling at 50%	Bigger models helped, but did not solve loops
Pairwise Gemini judging	Mirostat reduced repetition but often made outputs incoherent
Splitting the judge into a continuation score and a sounds-good score	Extra local auxiliary losses made training slower without clear listening wins
Core ML export with INT8 weights	Absolute scalar Gemini ratings were worse than pairwise judging
	Validation loss alone missed differences in rollout quality
	Born-again networks (retraining a model on its own soft predictions) did not improve quality

RollTab loops occasionally, and short prompts are difficult. Four-note prompts were the hardest, because there is little musical context to work with.

Eight notes worked better, and prompts of 16 to 32 notes were substantially more reliable. Unprompted generation is very much hit or miss, which is not the use he is aiming for.

On the app side, the first version took 11 days to get approved. A newer version pending review adds a choice between top-k, top-p, min-p, XTC, top-h, and Mirostat v2 sampling.

Saving a web page as Markdown keeps its headings, code blocks and tables as plain text, which is easy to search, quote, or hand to an AI model. Edwardsson's post has code blocks and result tables worth keeping that way.

To convert the page, try the tool that turns a page URL into Markdown. The address to enter is the post's own URL, given in the source link under the title.

Running models on your own hardware is the subject of another article. Local LLM Coding Now Hits 75% of Cloud AI covers coding with local models.