RL Carves the Space of Possible Futures

August 14, 2026

2026-08-14 — representation geometry · goal-conditioned RL · a pilot, prior-art-first

A goal-conditioned agent has to answer "how far is this goal?" To a network that was only shown lots of goals, two goals that look alike are close. But two goals that look alike can be very differently reachable — one's behind a wall, one isn't. The question I wanted to test: does reinforcement learning reorganize the agent's internal geometry away from appearance and toward what it can actually reach?

(A note on how I ran this, after a lesson: the previous tidbit I posted turned out to be mostly-known because I built first and searched the literature late. This time I ran a dedicated adversarial prior-art hunt before building. It found one close paper — "Beyond Euclidean Proximity" (2026), which shows a static appearance≠reachability separation with a frozen-encoder control — so I cite-and-beat it: my claim is only the parts it doesn't do.)

The setup

A compact goal-conditioned visual agent in a capability-gated task: a goal reachable only with a tool (paired worlds, pixel-identical except the inventory). I train two ways — imitation ("before RL") and online goal-directed RL ("after") — and ask, with the sim's exact reachability graph as ground truth (never shown to the agent): does the agent's latent distance to a goal track directed hitting-time, and does that emerge from RL? Everything's grounded in real Minecraft 1.11.2 pixels (via a bit-verified from-scratch renderer) for the substrate + demo.

Side-by-side Minecraft clip: with a pickaxe the agent mines through a wall to a glowing goal; without, it is blocked — The capability gate, in real Minecraft 1.11.2 RGB (rendered from a legally-owned game jar by a from-scratch, bit-verified C/CUDA renderer). Both panels are identical — a glowstone goal behind a stone wall — and the *only* difference is hotbar slot 0: **with a pickaxe** the agent mines a doorway and walks through; **without**, the bare hand can't break the stone and it stays blocked. That's the sim's real tool-gated block hardness, not a cosmetic edit — goal-reachability genuinely depends on the inventory capability, with exact reachability ground truth for the experimenter.

What holds: RL carves reachability geometry

Confirmed, and it's the real result. Latent distance goes from anti-correlated with reachability after imitation (partial Spearman −0.15 — the geometry is organized by appearance) to strongly aligned after RL (+0.47) — and the reorganization emerges over the course of RL training across all three seeds, rising from ~0 to 0.4–0.65. It is not the image encoder: a frozen/random encoder stays at ~0.00–0.05. This dynamic, training-time reorganization is exactly what the static single-checkpoint prior work couldn't show.

Three panels: capability-edit locality fails; reorganization confirmed; reorganization emerges over training — **Middle & right:** the reorganization — imitation's geometry anti-correlates with reachability (−0.15), RL's aligns (+0.47), the frozen-encoder control stays flat, and the right panel shows it building up *over RL training* (3 seeds). **Left:** the honest failure — see below.

What doesn't: you can't cleanly steer the capability

The tempting flashy version of this project was "decode a have-tool direction, edit it, watch the agent plan as if the wall were passable." It doesn't hold up. The capability is perfectly decodable (AUROC 1.0) and, after RL, editing it is even stronger and beats a random edit on 88% of worlds — but the edit is not local: it bleeds into unrelated navigation more than a random edit of the same size (leak 0.70 vs random 0.35, and worse than imitation's 0.33). So RL makes capability more causally potent without disentangling it — it's a potent-but-entangled correlate, not a clean, isolable, steerable variable.

I tried to fix it, two principled ways, and it wouldn't fix — which turned out to be the interesting part. The have-tool direction is ~98% contained inside the navigation-variance subspace of the hidden state. So (1) editing in the navigation-orthogonal complement drops the leak to ~0.03 but the effect goes inert (it flips a probe, not the plan), and (2) retraining with an explicit disentanglement penalty barely moves the leak (0.70→0.53, still far above the bar) and weakens the edit. The mechanism is almost obvious in hindsight: deciding to mine through vs go around is a navigation decision, so in a reachability representation the "have-tool" variable is bound to the navigation manifold by function — the causally-potent part of it is its overlap with navigation. You can't peel it off without killing it.

The one-line lesson, which is the through-line of everything on this blog lately: decodable + causally potent ≠ clean, local, steerable. RL carves the geometry of reachable futures into the representation; it does not hand you a tidy knob for each capability — and here that entanglement is structural, not an artifact of a naive edit.

Prior art, up front

cite-and-beat: "Beyond Euclidean Proximity" (2026) — the static appearance≠reachability decode with a frozen-encoder control. My delta: the training-time emergence of it under RL, and the capability-gated + causal-edit (negative) pieces it doesn't do.
cleared: Ghosh et al. Actionable Representations (symmetric metric, no reorganization-over-training, no edit); Halvagal & Chung Task-Induced Representational Invariances (algorithm-dependent invariance from scratch, decode-only) — both orthogonal. The quasimetric-RL line builds directed geometry but statically.

Caveats. A compact goal-conditioned CNN policy, imitation-vs-RL, one capability-gated task family; the RL sweep ran in a fast custom RGB gridworld for throughput, with real Minecraft for the substrate/grounding and the demo. Exact reachability ground truth is the experimenter's only, never supervised into the agent. Pre-registered before building (claims + rigor bar + go/no-go frozen up front); the causal-edit falsification is a deliberate pre-registered outcome, not a post-hoc excuse. Full code, preregistration, and every number in the repo. Written up with Claude Code.