再帰的自己改善はどこまで来たか、AIがAIの学習を改良する153回の実験

"We want to measure how well frontier models can conduct research."／"Claims about recursive self-improvement are becoming more common, yet we still lack convincing evaluations of autonomous research."／"To our knowledge, this is the first public experiment of its kind at this scale: runs lasting up to eight days, 8xH200s per run, and coverage of 18 models."／"For comparison, Anthropic's internal automated AI R&D evaluation optimizes a model on a CPU node, while OpenAI reports using nanoGPT Track 1 with a single H100 for less than a day in the GPT-5.6 Sol system card"／"One potential reason is that they over-focused on existing PRs."／"This time, we didn't give them access to the internet at all."（冒頭の各段落）— Prime Intellect より

"While we don't have strong conviction that methods developed in this kind of speedrun are inherently scalable or would be used in real model training, we think the tight feedback loop and hill-climbing aspect make it an interesting testbed for evaluating AI research capabilities."／"The speedrun trains a 124M parameter GPT and counts how many steps it takes to reach validation loss 3.28."／"Our baseline is the leaderboard's tuned-baseline entry, accepted there at 3,250 steps; under our own verification bar it passes at 3,290, and that is the number the agents start from."／"The latest record claim sits in an open PR at 2,600 steps."／"The agents get the training script with baseline hyperparameters and know that a better method exists."／"Everything below the baseline they have to find on their own."／"The improvements that win are optimizer work: better preconditioning, caps and floors on weight and update magnitudes, schedules that keep the learning rate hot for longer, weight averaging near the end of training, etc."（冒頭／Context 節）— Prime Intellect より

"The most striking result is the gap between models."／"It appears at every stage of the research process: which experiments they choose, how carefully they execute them, and how they interpret noisy results."／"Give each model's best final run the same resource budget and compare the best validated record it reached within that budget."／"Fable and Opus 5 lead however the budget is measured, and swapping hours for experiments barely changes the order, so the gap is not only about volume."（冒頭の要点／Results 節）— Prime Intellect より

"The models all find similar ideas. What separates them is how they run experiments."／"They kill families on one seed, treat their own crashes as proof the idea is bad, and throw away small gains that don't clear the bar alone. Grok 4.5 lost row normalization twice because of its own scaling bugs."／"The stronger models test borderline results on three seeds instead of one, and only pay for eight when their noise model says it's worth it."／"Opus 5 re-opened β2 tuning under a new recipe and it became a new record."／"K3 deleted two mechanisms that had led to the previous record after a new normalization made them useless."／"When Fable couldn't find gains from single knobs anymore, it started testing pairs that were individually worse but jointly better; one late re-probe was worth thirty-one steps."／"Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind."／"A good research decision is sometimes not to spend another GPU run."（Results 節／Prime Agent 節）— Prime Intellect より

"The rulebook, program.md (public), defines what can be edited, what counts as a record, and how to use the node."／"Each model+harness launches on a GPU node (8xH200s) in headless mode inside a simple sandbox (bwrap + network namespace)."／"The agent only sees its own working directory, the read-only dataset and the Python environment."／"The only route to the outside is a logging proxy that allows the model's API and nothing else."／"To claim a record, the model runs bash run.sh 8 which trains the recipe eight times on fixed seeds it can't touch and writes a logfile with the exact source and all eight losses."／"A frozen verify.py accepts the claim if the eight-run mean beats 3.27859 instead of 3.28, a margin that makes passing on luck alone roughly one-in-a-thousand, close to the statistical rule of the upstream repo."／"These constraints come from earlier runs where models would abuse the number of samples to pass the statistical test, kill runs way earlier than they should have, and so on."／"After hundreds of reports and no cheating or sandbox escapes, we stopped running it and check results and cheating when looking at progress or exporting the traces."／"Everything is public: traces, scratchpads, reasoning streams of the open-weights models, monitor reports, per-run ledger, and the harness in the shared research repository."（冒頭／Harness 節）— Prime Intellect より

"One other detail is that we gave an estimation of the speedrun noise in program.md that was slightly too large."／"62 out of ~100 runs measured it themselves instead of trusting our number, and these runs are concentrated at the top of the results table."／"42 went further and discovered something we never mentioned (on purpose): rerunning the same recipe on the same seed also moves the loss because GPUs are not deterministic."／"This noise is much smaller than seed-to-seed noise, so a model that finds it can compare two recipes on a shared seed and resolve differences a normal screen can't for the same cost."／"Several models rebuilt their screening protocol around this."（Harness 節）— Prime Intellect より

"One important disclaimer is that our benchmark has a lot of variance."／"This is due to the inner noise of the nanoGPT speedruns (hard to distinguish between improvement and seed noise) and also the randomness of the model on such a complex process."／"Running several seeds for several days would have made this first experiment too compute intensive. We launch at least three seeds, compare them after ~24 hours, and continue only the most promising one."／"Two runs of the same model and harness land about 54 steps apart at 24 \"agent-hours\", 43 apart at 100 experiments, and 40 apart at 300k output tokens."／"We didn't see major impact from any of these"／"some models did react differently to errors and restarts"／"Models also have different knowledge cutoffs which limits access to certain papers. This was a deliberate choice."／"We tried a few runs with a CLI tool for searching papers but found that restricting internet access including arxiv made models slightly more creative."／"A few ablation runs on models like Fable showed that high/xhigh/max often led to close results."／"We can't average over many replicates due to compute constraints, but the results still clearly separate models into different tiers."／"We are working on making this cleaner: more seeds per model and more models/harnesses."（Results 節／Noise and confounding factors 節／Limits and conclusion 節）— Prime Intellect より

"None of the runs produced a fundamentally new method; the winning ingredients are all similar to existing ones in the literature."／"Even so, models such as Fable 5 and Opus 5 performed dramatically better than the rest."／"We were again surprised by the lack of novelty."／"The models clearly understand the objects they manipulate at a deep level, and yet very few genuinely new ideas emerge, which makes it hard to tell if this is an artifact of the speedrun setup or a real capability limit."／"Intology's NanoGPT-Bench gives agents about 2.7 days on one 8xH100 node with no internet, they find agents recover less than 10% of five months of human progress."（冒頭の要点／Related work 節／Limits and conclusion 節）— Prime Intellect より