Action → camera-image motion: learning a dense, state-dependent ActionImage
A small encoder that maps the robot action into the optical-flow field on the camera image, on the world model's 16×16 latent grid. It is the visual analog of the tactile action→gel-flow encoder, and it exists to answer one question: the analytic ActionImage projects only the two sensors (a sparse prior), so it can describe hand motion but not the motion of the manipulated PCB — the exact thing the world model hallucinates. Can a model learn that propagation?
Answer: yes, and it needs both inputs. The ActionImage alone reaches 3.76 px error and the visual state alone 3.85 px — but together 2.42 px, against 5.45 px for the analytic prior and 7.07 px for predicting no motion at all. In the object (PCB) region specifically the learned encoder is 2.3× better than the analytic prior. Super-additivity is the point: neither the action nor the image alone determines how the board moves — their interaction does.
Task and data
Per latent transition (5 frames at 6 fps): predict the pooled AllTracker optical flow
of the middle camera. Inputs are the 12-channel ActionImage the video
expert already uses — 6 position Gaussians (2 sensors × gel centre / normal / up) plus
6 endpoint-motion channels (2 sensors × du, dv, weight) — the visual state
(the Wan2.2 VAE latent of the frame the transition starts from), and an
analytic prior (the ActionImage's own weight-averaged displacement,
fading to zero away from the sensors). Prediction is
residual + gain · prior, so the model learns a correction to the geometry
rather than the field from scratch.
Targets: AllTracker flow at 6 fps for all 36 episodes, cropped to the region the VAE actually encodes and pooled to the latent grid (units: px in the 256×256 crop). Validation is two fully held-out episodes. 150–810 K parameters per variant; minutes to train each.
Ablations — active-pixel end-point error (lower is better)
| variant | inputs | EPE active | EPE all | direction cos |
|---|---|---|---|---|
| zero baseline | — | 7.07 | 4.45 | — |
| analytic prior | geometry only, no learning | 5.45 | 3.76 | — |
| A | prior only | 4.24 | 2.62 | 0.754 |
| B | ActionImage | 3.76 | 2.31 | 0.807 |
| Bm | motion channels only | 3.82 | 2.35 | 0.807 |
| C | visual state | 3.85 | 2.35 | 0.760 |
| D | ActionImage + state | 2.45 | 1.52 | 0.868 |
| E | ActionImage + state + prior | 2.42 | 1.51 | 0.865 |
Bm drops the position Gaussians and keeps only the motion channels — a near-tie with B, so at this resolution where the sensors are adds little once how they move is given.
Where the gain lives: hand vs object
| region | zero | analytic prior | learned (E) |
|---|---|---|---|
| hand / sensor cells | 14.02 | 8.07 | 3.85 |
| object (PCB) cells | 6.02 | 5.05 | 2.20 |
The analytic prior barely beats predicting nothing in the object region (5.05 vs 6.02) — as expected for a two-point projection. The learned encoder cuts that error to 2.20.
Predicted vs ground-truth flow fields
Left to right: analytic prior · ActionImage only (B) · visual state only (C) · all inputs (E) · ground truth. One arrow per latent cell (all 16×16), drawn on the camera crop the model sees; the vertical sign is image convention (down = down, verified against the sensors' measured displacement). Arrow length is magnitude, exaggerated 5× — one grid cell of arrow = 5 px of flow in the 256-crop ≈ 9 px in the source frame ≈ 14 mm of physical motion over the transition's 0.83 s. Colour also encodes magnitude (pale yellow ≈ 0, dark red ≈ 12 px), so sub-pixel flow reads as faint without being hidden. Switch Split to compare held-out episodes (0510-ep005/006, never seen in training) against training episodes — the two look similar, which is the point: a 150–810 K-parameter encoder on 34 episodes is not memorising.
Why there are arrows off the motherboard
Three distinct causes, worth separating because only one of them is a model artifact:
- Tracker noise on the untextured table (most of it). Measured on a held-out episode: ground-truth flow on low-texture cells has a median of 0.42 px versus 12.4 px on textured cells (board, hands), and texture correlates with flow magnitude at +0.53. Optical flow is ill-posed on plain wood, so AllTracker returns small drifting vectors there. They are near-zero, but a 5× display exaggeration makes them visible — hence the colour scale, which keeps them pale.
- Resolution bleed. One latent cell covers 30×30 source px, so a cell straddling the board's edge averages moving and static content and lands somewhere in between. At 16×16 the object boundary is inherently a cell wide.
- Genuine over-spreading by the models (the real artifact). The analytic prior is a broad Gaussian and the encoder has a convolutional receptive field, so the prediction columns push motion further from the sensors than ground truth does. Compare the prior column against ground truth in the panels above: the prior fans motion across the whole left side, while ground truth concentrates it on the board and hands. That over-spread is part of the error the metrics report.
Toward a learned ActionImage
The world model currently conditions on the sparse analytic ActionImage. This encoder's output is a dense, state-dependent one — the same representation, but able to say what the action does to the object, not just to the sensors. The natural next step is to feed this predicted field back as the video branch's ActionImage (or inject its frozen motion feature), which is now justified by measurement rather than hope.
Caveats: two held-out episodes, one task (motherboard assembly), same session as training — this measures "unseen trajectory, seen conditions". Errors are per latent transition, in px of the 256×256 crop, on the 16×16 grid the world model operates on.