Temporally Centered SIGReg Improves LeWorldModel Representations for Robot Policy Learning

Two-panel overview. Left: the encoder and action-conditioned predictor, with each latent split into a temporal mean and a centered residual that SIGReg is applied to. Right: under Raw LeWM the persistent component and the residual compete for one near-unit variance budget and the residual is suppressed; under TC-LeWM their variance allocation is decoupled. — **Overview of TC-LeWM.** We keep the LeWM encoder and action-conditioned predictor unchanged. For each latent, we compute a local temporal mean z̄_t and centered residual r_t = z_t − z̄_t. Raw LeWM applies SIGReg to z_t; TC-LeWM applies the same regularizer to r_t instead.

Why this matters

LeWorldModel (LeWM) learns predictive representations directly from pixels and uses SIGReg to prevent representation collapse. We ask a different question: are these world-model representations also well suited for downstream robot policy learning?

On LIBERO-Long, Raw LeWM remains limited even as world-model training scales up. A frozen Raw LeWM encoder gives 26.1% average success under independent single-task training, 44.7% under suite-wise 10-task training, and 52.0% when the world model is trained on all 40 LIBERO tasks. More data helps, but does not remove the downstream representation gap.

Our analysis points to a specific mechanism: the Raw LeWM objective biases variance allocation toward temporally persistent information and suppresses the temporally centered residual—the part of the representation that captures complementary within-window change.

Why Raw LeWM suppresses residual variation

We decompose each latent as z_t = z̄_t + r_t, where z̄_t is the temporally persistent component and r_t is the centered residual. Along a random SIGReg projection, the same decomposition becomes x = c + ε.

We then construct a controlled Monte Carlo surrogate of the Raw LeWM objective. The Epps–Pulley criterion used by SIGReg favors a projected marginal close to a standard Gaussian and therefore favors projected variance near one. Under the zero-cross-covariance setting, var(x) = var(c) + var(ε). This couples the persistent and residual components through a shared near-unit variance budget. At the same time, the prediction objective favors smaller residual variation. Together, the two terms bias variance allocation toward the persistent component.

Three panels. (a) The Raw LeWM loss over the plane of persistent variance against residual variance, with its minimum at low residual variance. (b) EP loss rising with persistent variance for the raw target and staying flat for the residual target. (c) Trained LIBERO systems: Raw LeWM points lie on a line of constant total variance with small residual variance, TC-LeWM points sit at roughly constant residual variance with persistent variance free to vary. — **Monte Carlo and empirical variance allocation.** **(a)** The projected Raw LeWM surrogate has a minimum biased toward persistent variation and low residual variation. **(b)** Increasing persistent variance raises the EP loss on the full projected latent, while the EP loss of the temporally centered residual is unchanged. **(c)** Trained Raw LeWM representations show a nearly fixed total variance dominated by the persistent component, whereas TC-LeWM maintains a stable residual variance while persistent variance is free to vary.

This matters for downstream manipulation because suppressing centered-residual variation can make fine-grained robot state and dynamics less accessible from the learned representation. In our trained models, Raw LeWM indeed shows lower linear decodability of robot state and dynamics, particularly gripper dynamics.

TC-LeWM: change where SIGReg acts

The fix is simple: regularize the temporally centered residual instead of the full latent representation. The encoder, predictor, prediction loss, and downstream policy protocol are unchanged.

Method	Objective
Raw LeWM	L_pred + λ · SIGReg( { z_t } )
TC-LeWM (ours)	L_pred + λ · SIGReg( { r_t } ), r_t = z_t − z̄_t

In Raw LeWM, the persistent component and centered residual must jointly satisfy the SIGReg target. In TC-LeWM, SIGReg acts directly on the residual, so persistent variation no longer trades off against residual variance to satisfy the same projected target.

TC-LeWM also retains an effective anti-collapse property: if the representation becomes constant within a temporal window, then r_t = 0 and every projected residual degenerates at zero, which is strongly penalized by the EP criterion.

Closed-loop robot policy learning

TC-LeWM improves downstream behavior-cloning performance at all three world-model training scales. Under suite-wise 10-task training, it raises the four-suite average from 63.6% to 83.8%; under unified 40-task world-model training, it raises the average from 72.6% to 85.4%.

Suite-wise 10-task training

Method	Spatial	Object	Goal	Long	Avg.
Pre-trained methods
Octo (fine-tuned)	78.9	85.7	84.6	51.1	75.1
OpenVLA (fine-tuned)	84.7	88.4	79.2	53.7	76.5
π_0.5 (fine-tuned)	98.8	98.2	98.0	92.4	96.9
From-scratch methods
Diffusion Policy	78.3	92.5	68.3	50.5	72.4
Raw LeWM	62.4±7.5	93.9±2.3	53.4±5.3	44.7±4.3	63.6
TC-LeWM (ours)	86.0±4.1	97.0±0.1	88.4±7.5	63.6±6.8	83.8

Closed-loop success rates (%). Raw LeWM and TC-LeWM are mean±std over three full-pipeline training seeds. Diffusion Policy, Octo, and OpenVLA results are taken from Kim et al.; π_0.5 results are taken from Black et al. TC-LeWM is trained from scratch and outperforms Diffusion Policy, Octo, and OpenVLA in this evaluation, while pretrained π_0.5 remains stronger.

Independent single-task training

Method	T0	T1	T2	T3	T4	T5	T6	T7	T8	T9	Avg.
Raw LeWM	53.1	37.5	12.4	11.8	11.6	13.6	29.8	28.7	17.8	44.7	26.1
TC-LeWM (ours)	85.1	68.5	75.6	25.6	36.2	37.8	42.9	39.1	34.9	55.3	50.1
Δ (TC − Raw)	+32.0	+31.0	+63.2	+13.8	+24.6	+24.2	+13.1	+10.4	+17.1	+10.6	+24.0

Per-task closed-loop success rates (%) on the ten LIBERO-Long tasks, averaged over three training seeds and three evaluation seeds. TC-LeWM improves all ten tasks.

Unified 40-task world model training

Method	Spatial	Object	Goal	Long	Avg.
Raw LeWM	62.1±11.1	95.1±1.0	81.0±4.7	52.0±4.3	72.6
TC-LeWM (ours)	92.5±3.0	97.1±1.3	89.7±2.5	62.3±4.3	85.4
Δ (TC − Raw)	+30.4	+2.0	+8.7	+10.3	+12.8

All four LIBERO suites share one encoder and latent predictor, while a separate task-conditioned downstream policy is trained for each suite. Mean±std over three full-pipeline training seeds, each evaluated with three evaluation seeds and 50 rollouts per task.

Closed-loop examples

Example rollouts from the suite-wise 10-task setting. Raw LeWM and TC-LeWM use the same task, initial scene, and evaluation protocol.

2× speed, looping

“put both the cream cheese box and the butter in the basket”

Raw LeWM

TC-LeWM (ours)

What changes in the representation

The downstream gains are accompanied by consistent changes in the learned representations: more centered-residual variation, less trajectory-identity variation, and higher linear decodability of robot state and dynamics.

Four panels: variance allocation between persistent and residual components at three training scales; the difference in variance assigned to task identity, within-episode progress and trajectory identity; linear decodability of end-effector, gripper and joint state, velocity and acceleration; and closed-loop success split into grasping and placement stages. — **Representation analysis.** **(a)** Raw LeWM allocates 75%, 84%, and 95% of the variance to the persistent component at the 1-, 10-, and 40-task scales, leaving the centered residual strongly suppressed. TC-LeWM maintains substantial variation in both components. **(b)** Raw LeWM assigns 2.5–6.9× more variance to trajectory identity, while TC-LeWM assigns relatively more variation to within-episode progress under multi-task training. **(c)** Robot state and dynamics are more linearly decodable from TC-LeWM representations, with especially large gains for gripper dynamics. **(d)** TC-LeWM also achieves higher closed-loop success at both grasping and placement stages.

Attention rollout maps at the grasping moment for four LIBERO-Long tasks, external and wrist views, comparing Raw LeWM against TC-LeWM. — **Attention rollout at the grasping moment.** Across the first four LIBERO-Long tasks, TC-LeWM more consistently attends to manipulation-relevant regions such as the robot arm, gripper, and manipulated object, while Raw LeWM more often attends to background or task-irrelevant regions.

Grid of 3D PCA projections for ten LIBERO-Long tasks, comparing Raw LeWM against TC-LeWM, colored by within-trajectory progress. — **Per-task latent geometry.** 3D PCA visualizations on LIBERO-Long under suite-wise 10-task training, colored by normalized within-trajectory progress. TC-LeWM often exhibits smoother and more clearly progress-aligned structures, whereas Raw LeWM more often produces diffuse geometry. Percentages denote per-task closed-loop success for the displayed training seed, averaged over three evaluation seeds.

BibTeX

@misc{liu2026tcsigreg,
  title         = {Temporally Centered SIGReg Improves LeWorldModel Representations for Robot Policy Learning},
  author        = {Chang Liu and Fei Suo and Yanzhou Jin and Zeyu Ping and Yusuke Iwasawa and Yutaka Matsuo and Yaonan Zhu},
  year          = {2026},
  eprint        = {2607.26924},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2607.26924}
}