L1 / L5 / L6 / L7 — Four VLA Experiments

July 7, 2026

Notes to self on four mechanistic interpretability experiments I ran this week on LLaVA-1.5-7B and OpenVLA-7B. Two falsified hypotheses that left something better behind (L1, L5), two clean kills with mechanistic causes (L6, L7).

L1 — The Projector Isn't Rotating Anything

This is the good kind of dead hypothesis — the one that dies and leaves something better behind. The pitch I went in with: in LLaVA-1.5-7B, the little 2-layer MLP projector that glues CLIP to Llama ($1024 \to 4096$) is a coordinate change. If that were true, a diff-of-means concept direction you build on the CLIP side would be useless on the LLM side and vice versa — you'd have to transport the vector through the projector's Jacobian to move it across. Instead I found the projector barely rotates anything at all, and the actual coordinate change is happening inside the LLM, layers deep.

Setup, and one architecture gotcha

LLaVA-1.5 runs CLIP at 336px → 577 tokens (CLS + 576 patches). The projector consumes hidden_states[-2][:, 1:] — CLIP layer $-2$ with the CLS token dropped. So the CLS token never even reaches the projector, which killed my original plan to inject on CLS. I switched to the 576 patch tokens the projector actually eats:

• CLIP-space repr = mean over the 576 projector-input patch tokens (1024-d).
• LLM-space repr = mean over the 576 projector-output tokens (4096-d).
• CLIP injection = add $\alpha \cdot \hat{d}$ to all projector-input tokens.
• LLM injection = add $\alpha \cdot \hat{d}$ to all projector-output tokens.
• Jacobian transport = per-token projector Jacobian $J \in \mathbb{R}^{4096\times1024}$ at the mean patch feature; $\hat{d}_{\text{trans}} = \text{normalize}(J\,\hat{d}_{\text{clip}})$.

Concept directions are diff-of-means on Places365 indoor vs outdoor (200/class train, 40 eval). $\alpha$ scales by the empirical diff-of-means magnitude (CLIP $\approx 8.3$, LLM $\approx 12.0$).

Both kill criteria fired

Kill conditions: (1) post-projector native LLM injection steers as well as CLIP injection; (2) $\cos(\text{transported}, \text{native}) > 0.7$. Both fired, hard. The cosine came out $0.979$. Dose-response slopes:

Reanalysis — is $\cos = 0.98$ trivial? No.

A 2-layer MLP with a GELU could be near-linear if the nonlinearity is mostly asleep, making the high cosine boring. Test: compare Jacobian transport vs pure-linear transport $\text{normalize}(W_2 W_1\,\hat{d}_{\text{clip}})$, throwing the GELU away entirely:

Only 19–22% of GELU units sit in a linear regime (mean $|g''| \approx 0.66$). The map is genuinely nonlinear locally — the $0.98$ cosine is a real property of the nonlinear map, not a linear-algebra tautology. Also: $\cos > 0.7$ holds for warm_cool and colorful_muted (pixel-statistic attributes CLIP wasn't trained on), so the projector is geometry-preserving beyond just the dominant semantic axis.

The coordinate change is inside the LLM

Layer sweep — inject native direction ($\alpha=2$) at different decoder layers:

The concept direction is legible right after the projector (probe $R^2 = 0.995$ at layer 0) but injecting it there does almost nothing to what the model says. It only becomes causally load-bearing from layer 15 up. The projector hands off clean geometry; the LLM spends fifteen layers deciding what to do with it.

L5 — OpenVLA Knows Where It Is in the Episode

This one has a plot twist, so I want to write down the whole arc. The claim: OpenVLA-7B encodes the normalized within-episode timestep $t_{\text{norm}} = \text{frame\_index} / (\text{episode\_length}-1) \in [0,1]$ as a linear direction in its LLM hidden states — and crucially, not just as a proxy for where the arm is. The story: it looked killed, then I found the shuffle null was buggy, then a clean re-run on a totally different dataset replicated it. The methodology is the finding here.

Method

OpenVLA-7B forward with neutral prompt on recorded frames. Capture decoder layers $\{0,4,8,12,16,20,24,28,31\}$, average over 256 image-token positions → $(4096,)$ per frame per layer.

• Time probe: ridge $\text{hidden} \to t_{\text{norm}}$, $R^2$ on 20% held-out.
• Proprio probe: ridge $\text{hidden} \to \text{proprio}$ (multi-output), mean-over-dims $R^2$.
• Residualize (Frisch–Waugh): fit $B: \text{proprio}\to\text{hidden}$, subtract $\hat{H} = \text{proprio}\cdot B + c$, re-probe $\text{hidden}_{\text{resid}} \to t_{\text{norm}}$. If timestep is only arm pose, residualizing kills it.
• Shuffle null: permute labels, re-run. This is where it went wrong.

Kill criterion: residualized $R^2 \leq$ shuffled-residualized $R^2$.

Act one — LIBERO, and it looks dead

First dataset: lerobot/libero_10, 843 frames, 8-dim joint state as proprio. Raw time probe $R^2 \approx 0.99$ everywhere — but so is proprio. Here's the layer sweep with the original shuffle:

I almost wrote it up as a clean negative. The thing that stopped me: a properly shuffled label should give $R^2 \approx 0$. A null floor at $0.93$ means the shuffle didn't destroy anything.

The bug — episode-swap shuffle on a stereotyped task

The original shuffle permuted labels by swapping whole episodes. LIBERO tasks are visually stereotyped — every episode of a given task traces nearly the same trajectory through nearly the same images — so within-episode temporal structure survives an episode swap. Frame 5 of episode A and frame 5 of episode B are basically the same image at the same point in the motion. The "shuffled" labels still line up with the hidden states. Fix: shuffle at the frame level — a full random permutation of $t_{\text{norm}}$ across all frames, ignoring episode boundaries.

Act two — DROID, clean null, replicates

Rather than patch the null on LIBERO, I validated on lerobot/droid_100: 2000 frames across 10 Franka Panda episodes in real lab scenes, 7-dim end-effector state, no LIBERO stereotypy. Frame-level permutation null:

The progress representation is a mid-layer phenomenon — strongest around layer 16, degrading to $0.585$ at layer 31 while the null falls off to $-1.381$. The accidental clock is real, and it's not LIBERO-specific. The methodological moral: an episode-swap shuffle is not a null on a visually stereotyped task. Frame-level permutation is the right null, and I should have sanity-checked that the null was near zero before trusting the kill criterion.

L6 — No Lateral-Sign Axis (Killed)

Clean kill — both criteria — but a clean kill is still a result. The pitch: there ought to be a single low-dimensional lateral-sign axis in OpenVLA-7B such that flipping the input image left-right reflects both vision activations and predicted action logits along the same direction. A shared cross-modal "which way is left." It's not there.

Setup

300 DROID frames, OpenVLA-7B fp16, LLM layer 16, first action-token logits (256 x-axis bins), Ridge ($\alpha=1.0$), 80/20 split. Three conditions:

• hflip — horizontal flip (the hypothesis).
• vflip — vertical flip (magnitude-matched control, nothing to do with left/right).
• color jitter — non-geometric control.

Kill-1: PC1 of action-logit delta must exceed $0.70$ for hflip. Kill-2: $R^2(\Delta a \sim \Delta v)$ for hflip must beat vflip control.

Kill-1 — high-dimensional response

Kill-2 — the coupling isn't lateral-specific

There is a vision→action coupling under geometric perturbation — both flips give $R^2 \sim 0.5$–$0.7$, well above the color-jitter's $0.067$ — so mirroring does move both modalities in cross-modally predictable ways. But it's a generic geometric-perturbation effect, not a lateral one. Vertical does it better. The Pearson $r$ between PC1-projections of $\Delta v$ and $\Delta a$: hflip $= +0.210$, vflip $= -0.239$ — same magnitude, no lateral-specific shared axis. The scalar $dx$ shift: hflip $= 0.209$, vflip $= 0.235$ — flipping vertically moves the predicted x-action just as much.

L7 — OpenVLA Can't Say What It's About to Do (Killed)

Killed — but this is the most satisfying kind, because the negative comes with a crisp mechanistic reason. The pitch: patch OpenVLA's diff-of-means "move-left − move-right" motor direction into a text forward pass and watch $\text{logit}(\text{"left"}) - \text{logit}(\text{"right"})$ shift monotonically. The motor latent, causally wired to the language head. It doesn't work. The reason is that OpenVLA's language head isn't in word space anymore.

Setup

Data: lerobot/droid_100, filtered on x-axis EEF delta — left $< -0.15$, right $> +0.15$ — 200 fit + 50 eval per class. Patch: $\alpha \cdot \hat{d}_{\text{action}}$ (scaled to diff-of-means magnitude $\approx 3.62$) added at layer-16 output on all 256 image-token positions, $\alpha \in [-3,3]$ (and $\pm 8$ in diagnostics). Readout: signed $\Delta\text{logit} = \text{logit}(\text{"left"}) - \text{logit}(\text{"right"})$, sign-folded by true label. Controls: random-direction (3 seeds), LLaVA-1.5-7B (no-action-finetuning control).

The decisive finding — the LM head got repurposed

Before looking at dose-response, the diagnostics told the whole story. OpenVLA emits discretized action tokens (ids $> 31744$, top of vocab). The English "left"/"right" word logits are vestigial:

Dose-response — flat, as predicted

For completeness, signed $\Delta\text{logit}$ at the image-token patch:

α = −8:  −0.042
α = −4:  −0.027
α = −2:  −0.019
α =  0:  −0.017
α = +2:  −0.020
α = +4:  −0.021
α = +8:  −0.026

No monotone shift. Symmetric U-shape in the $\pm 0.02$–$0.04$ band — signature of norm perturbation noise. Note the symmetry in $\alpha$: pushing "toward left" and "toward right" nudge the readout the same tiny amount in the same direction. That's what you get when the patch is just perturbing the residual-stream norm and the direction's semantics don't matter. OpenVLA's slope is at the random-control level. LLaVA-1.5-7B keeps its LM head in word space ("left"/"right" rank $\ll 6696$), so the degeneracy is specifically the action fine-tuning, not the Llama architecture.

Two follow-ups worth doing: (1) read out on action tokens instead — does patching the motor direction shift the x-axis action-bin logits? That's the version of the hypothesis that respects where the probability mass lives. (2) Use a checkpoint that co-trains the LM head with action supervision so the word head stays alive; then the motor→language channel could plausibly be testable. The current negative is really a statement about this specific checkpoint, not VLAs in general.

— back to this tomorrow.