PuM — Action Baselines
June 22, 2026
PuM is a Minecraft agent with an explicit 3D voxel world model. The perception module predicts a 32×32×32 block grid from a single RGB frame; the action head (not yet added) will map that to keyboard + mouse outputs.
These are the pixel-only baselines, evaluated on 10 atomic block-interaction tasks from the MCU benchmark via MineStudio on an NVIDIA A100. Success = target item in inventory at episode end (programmatic). Max 600 steps. 10 episodes/task.
Results
| Task | VPT (BC) | VPT steps | Steve-1 | Steve-1 steps | ACT (BC) | ACT steps |
|---|---|---|---|---|---|---|
| mine oak log | 0/10 | — | 10/10 | 50.8 | 0/10 | — |
| mine stone | 2/10 | 50.0 | 10/10 | 213.0 | 1/10 | 513.0 |
| mine coal ore | 4/10 | 114.0 | 6/10 | 111.2 | 0/10 | — |
| mine iron ore | 2/10 | 60.0 | 5/10 | 165.2 | 0/10 | — |
| mine gravel | 4/10 | 35.2 | 2/10 | 102.5 | 9/10 ★ | 96.7 |
| mine sand | 4/10 | 190.5 | 2/10 | 29.0 | 5/10 ★ | 32.0 |
| mine dirt | 6/10 | 181.2 | 10/10 | 109.0 | 4/10 | 137.8 |
| navigate to water | 1/10 | 400.0 | 0/10 | — | no demos | |
| mine deepslate | 0/10 | — | 0/10 | — | no demos | |
| mine diamond ore | 3/10 | 68.7 | 0/10 | — | 0/10 | — |
| kill zombie | 2/10 | 76.5 | 5/10 | 151.6 | 0/10 | — |
| Total | 28/110 | 50/110 | 19/90† | |||
★ ACT beats both VPT and Steve-1 on gravel and sand despite training on only 633 demonstrations.
* ACT total excludes navigate_to_water and deepslate (no mine-block events in index); † also excludes kill_zombie (no combat demos).
Kill zombie: midnight, night_vision, Strength I + iron sword, zombie summoned 3 blocks ahead, max 400 steps. Success = kill registered in MineStudio kill_entity event.
VPT and ACT receive no goal input. Steve-1 receives the task name as a text prompt (MineCLIP). Mining tasks give an iron pickaxe; zombie task gives an iron sword.
ACT: frozen VPT-2x encoder + action-chunking transformer head (window=16, chunk=8), BC on 633 task-filtered VPT contractor demos.
Rollouts
One representative episode per task per model — shortest success, or best attempt if 0 successes. Click a task to compare all three models.