PuM — Data Collection Samples
June 15, 2026
PuM (Perception Under Motion) is a Minecraft AI agent project. The goal is to train a perception module that maps raw first-person video frames to a structured voxel-grid representation of the surrounding world — the input layer for a downstream policy.
These videos were captured by the Blockscope data collection pipeline: a headless Mineflayer bot acts as the controller (moves, looks around, generates interesting trajectories) while a separate Fabric client runs a record-only mirrored camera via ReplayMod, capturing clean first-person video. Per-tick block-state labels are extracted from the server and uploaded to a central store via Hopper, giving aligned (video, voxel-grid) training pairs.
The sessions below each mark a milestone in getting the pipeline production-ready.
Mineflayer → VPS: first end-to-end session
The proof-of-concept that the VPS architecture works. A Mineflayer controller bot connects to the Minecraft server hosted on the VPS, navigates naturally through 1.8-style overworld terrain, and the mirrored Fabric camera (also on the VPS) follows it frame-for-frame. Lodestone syncs the view; Hopper uploads ticks. First time all components (controller + camera pair + upload) ran together without manual intervention.
Productionized collector — no modal
~70 seconds of validated output from the hardened collector. Key fix: the ReplayMod
"recover recording?" modal that appeared on reconnect used to block the camera from
starting. The session manager now races a file-existence check against the modal and
dismisses it, so recordings start cleanly every time. The resulting .mcpr
is clean with no corrupted frames at the head.
1.8 natural terrain session
Classic 1.8 overworld terrain (converted to 1.19.4 via Amulet) used as the target visual domain for PuM's first training run. The limited palette — grass, dirt, stone, oak trees, sky — reduces visual complexity and matches the block set the perception module will be evaluated on. This session shows the Mineflayer bot traversing hills and forest, generating the kinds of view transitions the model needs to learn from.
VPS architecture working — 80s void-scatter session
First fully clean end-to-end session using the final VPS architecture: the Mineflayer
controller connects to the VPS first, Lodestone assigns it a void-scatter world (classic
1.5.2 block palette), the record-only Fabric camera joins into the same world directly
(no cross-world teleport → no Respawn packet → ReplayMod records the full session).
80 seconds, 14 MB video, valid .mcpr, no recovery modal, no WASD hints.
The void-scatter path shows the hand-placed walkway of old-era blocks — coloured wool,
terracotta, mushroom stem, minerals — which the perception module trains on.
Skywars island — minigame map from the world library
A session that landed on a Skywars map (skywars_map_solo_clearing)
from Lodestone's 35-world library rotation. Unlike the void-scatter and natural-terrain
sessions, this is a hand-built floating-island minigame map: cobblestone spawn cages,
densely decorated terrain, and neighbouring islands suspended over a starfield void.
It's a useful stress case for the perception module — tight built geometry and a hard
sky/void boundary instead of open natural horizon. The Mineflayer controller explores
the spawn island (no inter-island bridging yet), and the mirrored camera captures the
full 120s at 640×360 / 20fps with a clean .mcpr.
This run also exercises three new collector fixes: a per-second position logger with stuck detection, slow ~5°/s look-arounds during the 90s camera-warmup wait (so the mirror records moving views instead of a frozen frame), and a sprint cap that drops the bot from ~5.6 m/s to a steadier ~4.3 m/s walk for more watchable footage.
Hermitcraft composite episode — void-scatter world (round-robin miss)
First run of the new hermitcraft composite episode, which rotates
interior_hunt, rare_block, and vertical
sub-cycles on 30-second intervals. However, Lodestone's round-robin world pool
assigned this session to a void-scatter world (bs_5_5c4543)
rather than one of the hermitcraft maps — so the bot spawned on a floating
platform at 8,65,8. The interior_hunt and rare_block
sub-cycles immediately errored ("not iterable" — no loaded chunks containing
eligible blocks), while vertical sub-cycles succeeded and drove the
bot off the platform edge and up to rooftop height. 200 seconds, 48 MB, clean
.mcpr. Key finding: interior_hunt and
rare_block need a guard against empty block-scan results when
the world is sparse; and a world-type check at episode start would let the
hermitcraft composite bail early to a simpler fallback on non-hermitcraft maps.
Hermitcraft episode — interior_hunt + rare_block fix verified
Follow-up to the previous session, which crashed with TypeError: blocks is not iterable
in both interior_hunt and rare_block. The root cause:
bot.findBlock (singular) was being called with a count: N
option it doesn't support — it returns a single Block | null, and iterating
over that with for (const b of blocks) threw on the first non-null result.
The fix was to switch to bot.findBlocks (plural), which accepts
count and returns a Vec3[], then map back to Block
objects with bot.blockAt(pos).
This session was assigned to a hermitcraft world by Lodestone (spawned at
-63.5,100.0,0.5, surface Y ~93, surrounded by light_blue_stained_glass
— a typical hermitcraft decorative build). Five sub-cycles ran across 180 seconds:
vertical → rare_block → vertical →
interior_hunt → rare_block. No crashes.
rare_block resolved 24 tier-1 / 70 tier-2 / 65 tier-3 block IDs and
successfully navigated to a T2 light_blue_stained_glass target on both
runs. interior_hunt found no signature-block clusters in the initially
loaded chunks and fell back to an explore walk gracefully — also no crash, just the
expected "no signature blocks" fallback path.
Furnace — smelting raw sessions into voxel labels
Raw sessions are video + tick data. To train PuM, each tick needs a paired voxel label: a 32×32×32 grid of block IDs centred on the player, plus a visibility mask — which of those 32,768 voxels are actually visible from the player's camera position.
Furnace is the GPU labeling pipeline that generates these pairs. It runs
in two stages: a Node.js stage (parse_mcpr.js) replays the
.mcpr recording frame-by-frame via PrismarineJS to extract per-tick
world states into a compact binary (world_states.bin); then a Python
stage (labeler.py) drives an OpenGL renderer to cast rays from each
tick's camera pose and compute which voxels are hit, producing one
tick_NNNNN.npz per tick.
Each .npz contains:
b (32,32,32 int32 block state IDs),
m (32,32,32 uint8 visibility counts),
pose (full player state JSON),
and inventory.
The video below shows the first 5 seconds of the Skywars session (100 ticks at 20 fps).
Left: the raw game frame as captured from the Minecraft client.
Right: the Furnace GPU render — Renderer.render_rgb() called with the
same camera pose and the precomputed visibility mask, rendering only the voxels that were
actually visible from the player's perspective. Sky blue fill is the OpenGL clear colour.
Session: session_1781571293 •
2,553 ticks • 2,553 NPZ files • 569 MB labels •
31 unique block types at tick 1000 •
GPU 1 (RTX class) • ~11 min wall time •
command: CUDA_VISIBLE_DEVICES=1 uv run python -m pipeline.labeler --session /data/vvm33/BLOCKSCOPE_DATA/session_1781571293/