オンデバイスAIをiPhoneで動作させたRollTabの自動補完設計

"TL;DR: I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15)."／"The biggest improvements came from finding the right MIDI representation, cleaning the training data aggressively, and adding DPO post-training."／"Almost a year ago, I started tinkering with an idea: connect my MIDI piano to my phone, play something, and have AI autocomplete the song for me."／"Think GitHub Copilot, but for piano."／"Fourteen experiments later, it is finally at a point where I am happy enough with it to write about."／"The architecture is essentially a fairly standard decoder-only transformer: RMSNorm, rotary positional embeddings, causal self-attention, SwiGLU/MLP blocks, and autoregressive generation."（TL;DR／導入／Model 節）— SimEdw's Blog より

"There are 128 MIDI pitches and 128 velocity values, so the naive combined note-on vocabulary has up to: 128 * 128 + 128 = 16,512 tokens just for note-on and note-off."／"A common improvement is to factor the representation with a grammar"／"You can enforce the grammar during generation by masking invalid next tokens."／"I tried note-on/note-off style representations, but my models tended to drift."／"They would forget to emit note-off, leave hanging notes, or lose track of active state."／"That was especially bad for my target: a small model running close to real time on a laptop or phone."／"This worked better musically, but it was slow."／"One musical note took roughly four autoregressive transformer steps."／"It also burns through the context window quickly."（How Do You Tokenize Music? 節）— SimEdw's Blog より

"The representation I eventually settled on was: NOTE(pitch, delta_onset, duration, velocity) There is no separate TIME_SHIFT event in the final version."／"Silence is represented by delta_onset on the next note: the time since the previous note onset."／"We sort by pitch so that during training we don't get penalized when one song encoded a C-major chord as CEG and another as EGC."／"Each field gets its own embedding."／"There is a small nested decoder between the fields, so later fields can condition on earlier predicted fields."／"But the expensive transformer backbone runs only once per note, not once per field."／"Instead, sustain is baked into note duration during preprocessing."／"If the key is released while the sustain pedal is down, the note is extended to the pedal-up time."／"This loses the explicit pedal gesture, but it makes the modeling problem much simpler: the model only has to predict pitch, onset, duration, and velocity."（The final representation 節／Sustain Pedal 節）— SimEdw's Blog より

"I searched through a lot of publicly available datasets and collections, focusing mostly on older classical music in the public domain."／"The final dataset contained a few hundred thousand MIDI files, representing roughly 300 million note events."／"deduplicated by fingerprints that ignore global transposition and uniform tempo changes"／"I tried scaling the dataset to roughly 5x the size, hoping it would improve performance, but the resulting models were worse."／"Cleaning and selecting the data mattered more than simply adding more of it."／"Augmentation was important because the live input is not a pristine MIDI file."／"It is me playing piano, badly enough that notes might be slightly early, late, too hard, etc."（Dataset 節／Augmentation 節）— SimEdw's Blog より

"Still, the training objective has an important limitation: music continuation does not have a single correct answer."／"A held-out song only gives the model one "correct" next note, even though there are often many continuations that would work musically."／"Cross-entropy is useful for learning the mechanics of music, but not a great proxy for how good a full continuation sounds."／"So during training I sometimes fed the model its own predicted pitch instead."／"Validation loss ↓ scheduled 50% 2.9998 without scheduled 2.9495 Gemini preference ↑ scheduled 50% 64.3% without scheduled 35.7%"／"Scheduled sampling hurt validation loss, but improved rollout quality."（Training 節／Scheduled Sampling 節）— SimEdw's Blog より

"These metrics were useful for catching obvious failures, but they were not good enough to select the best model."／"Eventually I used Gemini 3.5 Flash for pairwise evaluation."／"Asking it to give a single absolute score was inconsistent."／"Asking instead, "given A and B, which continuation is better?" worked much better, especially when I mirrored every comparison to reduce position bias."／"In one test over 200 songs, Gemini gave the same preference 70% of the time after reversing A and B."／"Initially, Gemini overindexed on how good a continuation sounded in isolation, rather than how well it followed from the prompt."／"a continuation score, measuring how well the output follows from the prompt, and a sounds-good score, measuring its musical quality in isolation."／"I used the continuation score as the primary signal for DPO."（Evaluation 節）— SimEdw's Blog より

"DPO made the biggest difference after pretraining."／"It took the model from occasionally producing a good continuation to doing it much more reliably."／"After DPO, more than 69% of continuations were preferred over the base model in my pairwise evaluation."／"The β value controls how strongly DPO penalizes moving away from the base model."／"In my sweep, β=0.01 and β=0.03 improved the model, while β=0.10 pushed too hard and made it worse."／"I also tried a "consensus" dataset: instead of trusting every noisy preference judgment, I only kept preference pairs where the evaluator agreed consistently."／"pretrained base 24.55% β = 0.01 61.08% β = 0.03 57.14% β = 0.10 38.10% consensus(β = 0.03) 69.05%"／"My gut feeling is that the base model had already learned a reasonable mental model of music, just not what makes a good continuation."（DPO 節）— SimEdw's Blog より

"I exported the PyTorch model to Core ML and quantized the weights to INT8."／"The first launch is still annoyingly slow while Apple's runtime optimizes the model for the available hardware."／"The model was only trained with contexts of up to 512 notes, but I wanted to support longer sessions."／"Whenever the context gets close to the limit, I keep the most recent 384 notes, rebuild the context from those, and continue from there."／"This means rebuilding the KV cache, but the model is fast enough that it hasn't been a major problem."／"Unfortunately Core ML does not expose Q, K, and V directly."（Packaging it 節）— SimEdw's Blog より

"Four-note prompts were the hardest: there simply was not much musical context to work with."／"Eight notes worked better, while 16–32 note prompts were substantially more reliable because the model had enough structure to infer what was happening."／"Note-on/note-off drifted too much for small real-time models."／"Grammar-masked token streams were valid but slow."／"Broader data made results worse when the data was noisy."／"Bigger models helped, but did not magically solve loops."／"Mirostat reduced repetition but often made outputs incoherent."／"Extra local auxiliary losses made training slower without clear listening wins."／"Absolute scalar Gemini ratings were worse than pairwise judging."／"Validation loss alone missed important differences in rollout quality."／"Born-again networks (retraining a model on its own soft predictions) didn't improve quality here."／"It is still far from perfect."／"It loops occasionally, short prompts are difficult, and there is plenty I want to improve."（Evaluation 節／What Did Not Work 節／Conclusion 節）— SimEdw's Blog より