PuM — System Architecture

June 24, 2026

2026-06-24 — current state

PuM (Player You Mean) is a Minecraft agent that receives the game screen and outputs keyboard + mouse events — no privileged client-side access at inference time. The active research focus is the perception module: predicting a dense 32×32×32 voxel grid of semantic block types from a single 640×360 RGB frame.

The pipeline spans four machines. Data flows from clients → Hopper → Furnace → training. Baselines (VPT, Steve-1, ACT) run headlessly on the same GPU server via MineStudio.

flowchart TD
    Git(("Git"))
    Tidbits(("tidbits.vivime.info"))

    subgraph MAC["MAC"]
        ClaudeDev["Claude Dev Assistant"]
    end
    ClaudeDev -->|"commit / push"| Git
    ClaudeDev -->|"publish results"| Tidbits

    subgraph WINDOWS["WINDOWS"]
        WinClient["Player client
Blockscope mod + ReplayMod · humans"]
    end

    subgraph VPS["VPS (mc.vivime.info:25566)"]
        subgraph Paper["Paper 1.19.4"]
            subgraph Lodestone["Lodestone plugin"]
                SessionMgr["SessionManager
(world assign, pairing, mirror)"]
            end
        end
        WorldPool["World pool
(pre-staged copies)"]
        McConverter["mc_converter
(1.8→1.19.4, no DFU)"]
        WorldGenNode["World generator
(void-scatter, configurable)"]

        Lodestone <-->|"draw / refill"| WorldPool
        McConverter --> WorldPool
        WorldGenNode --> WorldPool
    end

    subgraph BREV["BREV (GPU server — A100s)"]
        subgraph Camera["Camera container"]
            BlockscopeMod["Blockscope mod
(record-only)"]
            ReplayModNode["ReplayMod"]
        end
        Controller["Controller
(Mineflayer bot)"]
        Hopper["Hopper
(FastAPI :9000)"]

        subgraph DATA["/data/vvm33"]
            Raw["BLOCKSCOPE_DATA/
raw sessions"]
            Smelted["SMELTED_DATA/
per-tick NPZs"]
            Weights["pum_checkpoints/
model weights"]
            BaselineData["act_training_data/
contractor demos"]
        end

        subgraph FURNACE["Furnace (pipeline.labeler)"]
            Baker["Baker
(one-time, reads MC jar)"]
            Labeler["Labeler
(replays .mcpr + ticks.jsonl)"]
            Renderer["Renderer
(ModernGL GPU)"]
            Baker -->|"geometry + texture cache"| Renderer
            Labeler -->|"world grid + camera pose"| Renderer
        end

        subgraph TRAIN["Training (train.sh)"]
            ModelCode["viblock5
(SigLIP + voxel slots)"]
        end

        subgraph BASELINES["Baselines (MineStudio)"]
            VPTBaseline["VPT (BC)"]
            Steve1Baseline["Steve-1"]
            ACTBaseline["ACT (BC)
(frozen VPT-2x + transformer head)"]
            MCUEval["MCU eval
(10 tasks, inventory check)"]
            VPTBaseline --> MCUEval
            Steve1Baseline --> MCUEval
            ACTBaseline --> MCUEval
            BaselineData -->|"task-filtered demos"| ACTBaseline
        end

        GPUs[("GPUs")]

        BlockscopeMod -->|"ticks / video"| Hopper
        ReplayModNode -->|".mcpr"| Hopper
        Hopper -->|"write"| Raw
        Raw -->|"read"| Labeler
        Renderer -->|"write"| Smelted
        Smelted -->|"read"| ModelCode
        ModelCode -->|"write"| Weights

        Camera -.->|"uses"| GPUs
        Renderer -.->|"uses"| GPUs
        ModelCode -.->|"uses"| GPUs
        BASELINES -.->|"uses"| GPUs
    end

    Controller <-->|"connect ctrl_id"| SessionMgr
    Camera <-->|"connect cam_id"| SessionMgr
    WinClient <-->|"connect + world assign"| SessionMgr
    WinClient -->|"uploads (HTTP)"| Hopper

    wandb(("wandb"))
    ModelCode -->|"experiment runs"| wandb
    MCUEval -->|"result table + videos"| Tidbits

Components

MAC: Dev machine. All builds run here (Lodestone, Blockscope mod, mc_converter). Claude Code assistant. Deploys jars to VPS + Brev.
VPS: Paper 1.19.4 server at mc.vivime.info:25566. Lodestone plugin manages world assignment, controller/camera pairing, and session lifecycle. 35-world map library.
WINDOWS: Human data collection PC. Blockscope mod + ReplayMod + Baritone. Uploads to Hopper.
BREV: Shared A100 GPU server. Runs headless Camera containers (Docker + Xvfb + VirtualGL), Mineflayer controller bots, Hopper upload API, Furnace labeling pipeline, PuM training, and MineStudio baseline evaluation.
Hopper: FastAPI server on Brev (:9000). Receives streamed video, ticks.jsonl, frame_mapping.jsonl, and .mcpr from all clients.
Furnace: GPU OpenGL pipeline (ModernGL). Replays .mcpr to reconstruct world state per tick, ray-casts visibility masks, outputs per-tick NPZs (32³ block grid + visibility + pose).
viblock5: Current active experiment. SigLIP-B/16 with 8³ voxel slot tokens injected at layer 10, decoded to 32³ via transposed convolutions. Visibility-masked loss.
Baselines: MineStudio headless eval: VPT (BC, 26/100), Steve-1 (45/100), ACT (BC on task-filtered contractor demos, 19/80). MCU block tasks, programmatic inventory check. Results at pum-baselines.