Sakana AI「Dream-Cubed」とは？AIが遊べるMinecraftワールドを生成する仕組み

We are excited to share our latest work, together with New York University: Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes. ／ Technical Blog: https://pub.sakana.ai/dream-cubed ／ Paper: https://arxiv.org/abs/2604.22847 ／ Code: https://github.com/SakanaAI/DreamCubed — ニューヨーク大学との共同研究であること、論文の正式題、および公開物の所在に関する記述より

Generative AI has made incredible progress in the fields of image, video, and text generation. Despite success in these modalities, the interactive 3D worlds of video games have received much less research attention. ／ The 3D world of Minecraft is made up of voxels: discrete, compositional building blocks like sand, stone, and water. These semantically meaningful block types provide a perfect abstraction for generative modeling: each piece of the game world is simply a collection of block tokens that come together to create a simulacrum of the natural world. — 3D世界が研究上の空白であること、およびボクセル構造が生成モデルに適する理由に関する記述より

We create automated data collection and curation pipelines, collecting 1.8+ million 32³ chunks of procedurally generated Minecraft worlds, and 200,00+ chunks from human-authored maps, comprising over 30 billion total blocks. ／ We collect chunks from the surface level of the world and assign a class label based on the biome they are in, creating a dataset for biome-conditional generation. ／ Villages, for example, are both the most structurally complex and the least common areas of the game world. To overcome this data balancing problem, we set up targeted collection pipelines to identify and collect specific biomes to ensure adequate representation, which we show can have a very significant impact on downstream models. ／ Biome-conditioned 32³ chunks generated by our top-performing discrete MD4 model and continuous DDPM model for each of the 15 biome classes. ／ We reached out to one such professional map creator, who granted permission to use several of their large and high-quality maps for scraping. We release all our data and models in accordance with the map creators for the larger research and Minecraft community. — データセットの規模、バイオームによる分類とその種類数、偏りへの対処、および人間制作マップの許諾に関する記述より

A key design decision is not to compress the spatial dimensions of our data. Commonly, diffusion models operate on the compressed representations from models like Variational Autoencoders, which can massively improve the efficiency of diffusion models. We choose to train directly at the native block resolution, feeding each 32³ chunk of the world into a 3D Diffusion Transformer with ~280M parameters. While we miss out on the performance gains of compressed representations, we find this choice is crucial for enabling fine-grained control over generation. — 空間次元を圧縮しないという設計判断と、その代償および狙いに関する記述より

We implement two diffusion formulations: ／ MD4 discrete masked diffusion: We add a special [MASK] state to the vocabulary, and corrupt chunks by gradually replacing real blocks with [MASK] tokens. To generate new chunks, the model "unmasks" blocks by predicting the original block type to replace the [MASK] tokens. ／ DDPM in embedding space: We map each block type to a 16-dimensional vector obtained by a frozen text embedding model, using the block's in-game name (e.g., "stone"). Chunks are corrupted by gradually adding Gaussian noise, and we generate chunks by gradually removing noise back to clean embeddings. We perform a nearest neighbor lookup to recover discrete block types from denoised embedding vectors. ／ With our carefully curated data and after hyper-parameter tuning, we show that both models perform well on the training task, generating believable 32³ conditioned on a target biome, as seen below. — 比較した2つの拡散方式の手順と、どちらも学習課題をこなせたという結果に関する記述より

This simple property gives us a fine-grained block-conditioning on top of class-conditioning as a free byproduct of training. ／ In practice, this means we can write any pattern of blocks as an initial state and even during the generation process, which becomes a hard constraint that the model must work around. Because we work at the native resolution using the in-game block vocabulary, this translates to simply placing a handful of Minecraft blocks in a desired pattern. ／ We can use a simple sliding window procedure to generate arbitrarily-sized worlds, generating overlapping chunks in sequence and stitching them together: — ブロック指定生成が学習の副産物として得られること、および任意サイズのワールド生成に関する記述より

To test how important data composition is, we create three variants of our dataset, each with 1 million chunks. The Balanced dataset represents even distribution between the 15 natural biome classes, while the Boosted dataset doubles the proportion of village samples. The Natural dataset represents the naive approach, collecting chunks at their natural spawn rate in the game world, leading to large imbalances between biome labels. ／ We can see a significant performance drop in our patch 4 models in the village biome, suggesting that the finer resolution at lower patch sizes is important for modeling this more complex data. Similarly, we see differences regarding dataset composition: the Natural dataset model performs well on the overrepresented ocean biome (35% of the dataset), and underperforms in the village biome (< 1% of the dataset). We find that doubling the number of village samples compared to the rest in the Boosted model, we see a significant improvement in FID. — データセット3変種の中身、パッチサイズの影響、およびデータセット構成による成績の違いに関する記述より

We create an experiment where humans view two sets of chunks for the same biome, and pick the one they think is better. We include samples generated by two of our discrete models, our DDPM model, and real samples taken from our training data. ／ MD4 (patch 2) Real 67.1% 173 ／ MD4 (patch 4) Real 57.1% 163 ／ DDPM (patch 2) Real 55.2% 192 ／ Model vs. real. Win rate is the fraction of trials where participants preferred generated chunks from Source A over real Minecraft chunks from the same biome. ／ Across all three models, generated chunks were actually preferred to real ones. This may be partly due to our process of data curation and generation, but we view it as evidence that generated chunks are at least on par with the distribution we're trying to model. — 人間による比較実験の設計、モデル別の勝率と試行数、およびその結果に対する解釈に関する記述より

Lacking established metrics for 3D voxel worlds, we adapt Fréchet Inception Distance (FID) [4] to measure the differences between our models. ／ Using this data, we validate the render-based FID metric we use. If image-based FID is a good proxy for 3D chunk quality, we'd expect that lower-FID models would win more against higher FID models. ／ And we do find a limited but significant relationship between FID and win rate. In particular, we find that for biomes where there are small FID differences across models, this relationship is weaker. However, focusing on more structured biomes where the FID differences are larger, alignment of our metric with human judgment becomes stronger, with the lower-FID model winning 66.1% of the time when the gap is > 15 points. — 3Dボクセル世界に確立した指標が無いこと、FIDをレンダリング画像へ適用したこと、および人間の選好で指標そのものを検証した結果に関する記述より

When training on human-authored maps, we treat each map as its own additional "biome", and compress the thousands of block types present in this data down to a smaller, more manageable vocabulary of 171 unique blocks. As map creators use a much larger palette of blocks to capture small visual details, compressing maps to a limited vocabulary results in data that is simpler, allowing our models to focus instead on capturing the complex and detailed human structures. We acknowledge that learning to generate from data that encodes human creativity comes with a new set of challenges and even ethical considerations, and we present our models and results as a first step in this domain. — 人間制作マップの語彙を171種類へ圧縮した理由と、その扱いに伴う倫理的な論点に関する記述より

We use our data to train a family of powerful transformers for efficient generation of interactive 3D environments at cube resolution. We show how our models allow players to mold the world around them by generating structures, terrain, and maps that are immediately editable and playable. — 生成物がそのまま編集・プレイ可能である点に関する記述より