Reading & Steering a VLA's Behavior Manifold

August 6, 2026

OpenHA is a vision-language-action model that plays Minecraft. Before it acts, it writes down what it intends to do — a Chain-of-Action line like Grounding: Mine oak_log — and then emits the concrete keyboard/mouse action. That intermediate "intent" makes it an unusually good subject for a question I care about: does a VLA represent abstract behavior as a smooth, low-dimensional manifold you can read and steer, or as a pile of disconnected task-specific reflexes?

The short answer, from a few dozen experiments on one model: there is a real manifold, at layer 21 — the point where the model decides what to do. You can read intent off it, and you can steer it (the same "mine" direction still works in the Nether). But it has a hard edge: the intent layer is writable, the motor layer that executes skilled actions is essentially read-only. This note walks through it with the actual renders and gameplay clips.

The manifold

Collecting layer-21 activations across thousands of gameplay steps and projecting them down, behavior lives on a continuous, low-dimensional (~6-D) manifold that runs through the whole network. It isn't discrete clusters and it isn't a clean circle — it's an open gradient organized by the action-state the model is in: approach → orient → strike. Drag to rotate; click a state in the legend to isolate it; the bold thread is the model's central intent-path.

Interactive — drag to rotate, scroll to zoom. Open fullscreen ↗

The manifold, in motion. A dot marks the model's real layer-21 activation — projected into the same PCA frame as the render above — tracing a comet-trail as the agent plays. It climbs from approach toward strike as the decoded intent flips to Mine and it breaks the block.

What's real: a genuinely low-dimensional (~6-D nonlinear), continuous, whole-network manifold whose coordinate is discrete action-state, not stimulus geometry (camera angle and distance-to-target don't decode from here at all). The render follows Goodfire's method: a smoothing spline through real per-state centroids over a demoted point cloud — no fitted surface.

Honest slice. The 3-D view is a PCA projection capturing only 23.6% of the variance of a truly ~6-D object; the cloud really is that diffuse. The spline is a smoothing fit through real approach→strike centroids, not a surface, and it's an open arc, not a loop.

Reading intent

Because the model declares its Grounding: verb before acting, you can literally read its intent per frame. It tracks the scene: a block in front reads Mine; an animal in the distance reads Approach. Bars are the model's real next-token probabilities.

Decoded Grounding: intent + P(Approach / Mine / Kill) over four scenes. No steering — just reading.

Steering it — across worlds

Averaging activations for "mining" over a handful of blocks gives a direction. Add that direction into layer 21 during live play and the model's mine-intent, and its actual mining, both go up — from stated intent all the way to breaking blocks.

The clean causal test: same world, same prompt, same seed — the only difference is one vector added at layer 21. The orange arrow on the steered manifold is that injected vector (unsteered point → steered point, in the same PCA frame). Left stays idle (0 blocks); right's intent goes to Mine and it breaks a block. Every number on screen is a real logged value.

The striking part: the vector was built entirely in the overworld, yet it still works when the visuals and the block type are completely out of distribution. Same vector, in the Nether:

Matched seed, netherrack world. Left: unsteered — idle, P(Mine) 0.06, 0 blocks. Right: +mine-vector at layer 21 — P(Mine) 0.97, mining netherrack. "Mine" is a biome-invariant concept, not an overworld-pixel feature.

Overworld — idle → mining. (Representative: baseline mined 0 in every seed.)

The same steer across four worlds — overworld, Nether, mushroom, glowstone cave.

Steering is real, biome-invariant, and directional — but it only works for behaviors the model forms a distinct, non-default intent for. "Mine" steers cleanly; "approach" is the model's saturated default verb (nothing to push); movement is marginal. And it's modest: dose-limited, above a random-vector null but not dramatic. The render / clips are honest about which episodes are typical vs favorable.

The edge: what won't steer

The clean boundary of the whole project is this: you can steer what the model decides, but not how skillfully it executes. Killing a mob is the case that fails — and unpacking why gives the sharpest result.

no intent: In a "kill" scene the model's declared verb is Approach (P(Kill) ≈ 0.02) — there is no distinct "kill" intent at layer 21 to amplify. It can kill unsteered (5/6), so it's a control gap, not a capability gap.
read-only motor: The competent attack cadence (rapidly toggling the click) is perfectly readable downstream (~1.0 decode) yet not writable by any static injection at any layer — it just disrupts.
why: it's temporal: The click is autoregressively persistent — the model's default is to repeat the last action, and the "toggle" direction is literally the negation of that. A constant push can't create an alternating pattern. A dynamic, persistence-breaking intervention does write the cadence (toggled edges 2.4 → 10.4) — competent kills stay gated by a second aiming problem.

In one line

A VLA's behavior is a readable, low-dimensional, distribution-shift-robust intent layer sitting over a read-only procedural motor layer — you can read both, but you can only write the intent (and there it generalizes across worlds), while skill needs an intervention matched to how the motor actually computes it. Reading a concept is not the same as controlling the behavior; the gap between them is exactly where safety and interpretability get interesting.

Caveats. One model (OpenHA / Qwen2-VL-7B), one environment (Minecraft via headless MineStudio), modest effect sizes, a narrow usable steering range, and constructed biome scenes (the "Nether" is a netherrack build, not the true dimension). Full method, numbers, and every honest negative are in the repo (see M0/M0_MANIFOLD_CATALOG.md and M0/M0_IMPLICATIONS.md). Written up with Claude Code.