Sakana AIのマイクラ生成AI、300億ブロック学習で遊べる世界に

We are excited to share our latest work, together with New York University: Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes. ／ Technical Blog: https://pub.sakana.ai/dream-cubed ／ Paper: https://arxiv.org/abs/2604.22847 ／ Code: https://github.com/SakanaAI/DreamCubed — 公開告知の冒頭より

Generative AI has made incredible progress in language modeling, far beyond other modalities, where words and tokens offer a natural compositional unit for scalable training. This is similar to Minecraft and many other popular video games, where developers rely on cubes, tiles, and other discrete primitives to build rich, interactive worlds. — 言語モデルとゲームのブロックを対比した段落より

Generative AI has made incredible progress in the fields of image, video, and text generation. Despite success in these modalities, the interactive 3D worlds of video games have received much less research attention. ／ The 3D world of Minecraft is made up of voxels: discrete, compositional building blocks like sand, stone, and water. These semantically meaningful block types provide a perfect abstraction for generative modeling: each piece of the game world is simply a collection of block tokens that come together to create a simulacrum of the natural world. — 3D世界が研究の手薄な領域であることと、ブロック構造が生成モデルに向く理由の記述より

We create automated data collection and curation pipelines, collecting 1.8+ million ／ chunks of procedurally generated Minecraft worlds, and 200,00+ chunks from human-authored maps, comprising over 30 billion total blocks. ／ We collect chunks from the surface level of the world and assign a class label based on the biome they are in, creating a dataset for biome-conditional generation. ／ Villages, for example, are both the most structurally complex and the least common areas of the game world. To overcome this data balancing problem, we set up targeted collection pipelines to identify and collect specific biomes to ensure adequate representation, which we show can have a very significant impact on downstream models. ／ chunks generated by our top-performing discrete MD4 model and continuous DDPM model for each of the 15 biome classes. ／ We reached out to one such professional map creator, who granted permission to use several of their large and high-quality maps for scraping. We release all our data and models in accordance with the map creators for the larger research and Minecraft community. — データセットの規模、バイオームの分類と種類数、偏りへの対処、および人が作ったマップの許諾に関する記述より

A key design decision is not to compress the spatial dimensions of our data. Commonly, diffusion models operate on the compressed representations from models like Variational Autoencoders, which can massively improve the efficiency of diffusion models. We choose to train directly at the native block resolution, feeding each ／ chunk of the world into a 3D Diffusion Transformer with ~280M parameters. While we miss out on the performance gains of compressed representations, we find this choice is crucial for enabling fine-grained control over generation. — 空間の次元を圧縮しないという設計判断と、その代償および狙いに関する記述より

To understand what makes a great foundation model for Minecraft, we study both diffusion formulations, using an identical Diffusion Transformer backbone ／ MD4 discrete masked diffusion: We add a special [MASK] state to the vocabulary, and corrupt chunks by gradually replacing real blocks with [MASK] tokens. To generate new chunks, the model "unmasks" blocks by predicting the original block type to replace the [MASK] tokens. ／ DDPM in embedding space: We map each block type to a 16-dimensional vector obtained by a frozen text embedding model, using the block's in-game name (e.g., "stone"). Chunks are corrupted by gradually adding Gaussian noise, and we generate chunks by gradually removing noise back to clean embeddings. We perform a nearest neighbor lookup to recover discrete block types from denoised embedding vectors. ／ With our carefully curated data and after hyper-parameter tuning, we show that both models perform well on the training task, generating believable ／ we find the discrete diffusion formulation unlocks a variety of interactive generation workflows beyond the single-chunk generation case, making it a more compelling choice for an interactive world model. — 比較した2方式の手順と、どちらも学習課題をこなせたうえで離散マスク拡散を選んだ理由に関する記述より

This simple property gives us a fine-grained block-conditioning on top of class-conditioning as a free byproduct of training. ／ In practice, this means we can write any pattern of blocks as an initial state and even during the generation process, which becomes a hard constraint that the model must work around. ／ you can regenerate parts of a chunk taken directly from the game world, or hand-craft custom patterns, then let the model fill in the rest. ／ We can make spirals, waves, and slopes of blocks to see how our model works with these constraints to create a realistic chunk of the game world. Even suggestive patterns like a volcano, which don't naturally occur in Minecraft, are no problem and result in successful extrapolation. ／ We can use a simple sliding window procedure to generate arbitrarily-sized worlds, generating overlapping chunks in sequence and stitching them together: — ブロックを指定した生成が学習の副産物として得られること、その実例、および任意の大きさのワールド生成に関する記述より

When training on human-authored maps, we treat each map as its own additional "biome", and compress the thousands of block types present in this data down to a smaller, more manageable vocabulary of 171 unique blocks. As map creators use a much larger palette of blocks to capture small visual details, compressing maps to a limited vocabulary results in data that is simpler, allowing our models to focus instead on capturing the complex and detailed human structures. We acknowledge that learning to generate from data that encodes human creativity comes with a new set of challenges and even ethical considerations, and we present our models and results as a first step in this domain. — 人が作ったマップの語彙を171種類へ絞った理由と、その扱いに伴う倫理面の論点に関する記述より

To test how important data composition is, we create three variants of our dataset, each with 1 million chunks. The Balanced dataset represents even distribution between the 15 natural biome classes, while the Boosted dataset doubles the proportion of village samples. The Natural dataset represents the naive approach, collecting chunks at their natural spawn rate in the game world, leading to large imbalances between biome labels. ／ We can see a significant performance drop in our patch 4 models in the village biome, suggesting that the finer resolution at lower patch sizes is important for modeling this more complex data. Similarly, we see differences regarding dataset composition: the Natural dataset model performs well on the overrepresented ocean biome (35% of the dataset), and underperforms in the village biome (< 1% of the dataset). We find that doubling the number of village samples compared to the rest in the Boosted model, we see a significant improvement in FID. — データセット3変種の中身、パッチサイズの影響、およびデータの構成による成績の違いに関する記述より

We create an experiment where humans view two sets of chunks for the same biome, and pick the one they think is better. We include samples generated by two of our discrete models, our DDPM model, and real samples taken from our training data. ／ MD4 (patch 2) Real 67.1% 173 ／ MD4 (patch 4) Real 57.1% 163 ／ DDPM (patch 2) Real 55.2% 192 ／ Model vs. real. Win rate is the fraction of trials where participants preferred generated chunks from Source A over real Minecraft chunks from the same biome. ／ Across all three models, generated chunks were actually preferred to real ones. This may be partly due to our process of data curation and generation, but we view it as evidence that generated chunks are at least on par with the distribution we're trying to model. — 人による比較実験の設計、モデル別の勝率と試行数、およびその結果の解釈に関する記述より

Lacking established metrics for 3D voxel worlds, we adapt Fréchet Inception Distance (FID) ／ Using this data, we validate the render-based FID metric we use. If image-based FID is a good proxy for 3D chunk quality, we'd expect that lower-FID models would win more against higher FID models. ／ And we do find a limited but significant relationship between FID and win rate. In particular, we find that for biomes where there are small FID differences across models, this relationship is weaker. However, focusing on more structured biomes where the FID differences are larger, alignment of our metric with human judgment becomes stronger, with the lower-FID model winning 66.1% of the time when the gap is > 15 points. — 3Dのブロック世界に確立した指標が無いことと、人の選択で指標そのものを検証した結果に関する記述より

We use our data to train a family of powerful transformers for efficient generation of interactive 3D environments at cube resolution. We show how our models allow players to mold the world around them by generating structures, terrain, and maps that are immediately editable and playable. — 生成物がそのまま編集・プレイできる点に関する記述より