Understanding the Limits of Latent World Models

World models are widely proposed as core components for enabling agents to reason over long temporal and spatial horizons. Yet current evaluations often emphasize task performance while providing limited insight into whether learned world models acquire predictive latent dynamics that reflect the underlying structure of the environment. Here we introduce the concept of faithful latent world models as an equivariance condition between the environment’s transition map and the learned latent dynamics. We propose latent rollout consistency (LRC) as a metric to evaluate action-conditioned latent prediction. Strikingly, we find that across the studied state of the art latent world models, none solves the task by actually faithfully modeling the dynamics of the environment.

PDF22.8 MB

What does it take to model the world?

Current evaluations of latent world models emphasise task performance and give limited insight into whether the learned latent dynamics reflect the structure of the environment. We call a latent world model faithful when its representation preserves the environment state and its latent dynamics evolve consistently with the true dynamics under the same actions.

Commutative diagram of a faithful latent world model, with state space and latent space connected by r = h ∘ g, and latent rollout consistency comparing predicted with encoded latents — Fig. 1 · (a) In a faithful latent world model, applying the true transition f and then encoding with r = h ∘ g gives the same result as encoding first and applying the learned latent dynamics under the same action. (b) Latent rollout consistency (LRC) compares autoregressive latent predictions with the representations inferred from the corresponding realised observations.

The environment evolves as $\boldsymbol s_{t+1} = \mathbf f(\boldsymbol s_t, \boldsymbol a_t)$ and is observed as $\boldsymbol x_t = \mathbf g(\boldsymbol s_t)$ . A latent world model infers $\boldsymbol z_t = \mathbf h(\boldsymbol x_{\leq t}, \boldsymbol a_{\lt t})$ from observations and predicts $\hat{\boldsymbol z}_{t+1} = \hat{\mathbf f}(\boldsymbol z_t, \boldsymbol a_t)$ . With the state-to-latent map $\mathbf r = \mathbf h \circ \mathbf g$ , the model is faithful if it satisfies two conditions. The first, dynamics equivariance, requires $\hat{\mathbf f}(\mathbf r(\boldsymbol s), \boldsymbol a) = \mathbf r(\mathbf f(\boldsymbol s, \boldsymbol a))$ for all states $\boldsymbol s$ and actions $\boldsymbol a$ : encoding after a true transition agrees with predicting in latent space. The second, state sufficiency, requires $\mathbf r$ to be injective, so that the representation retains the full state.

We measure dynamics equivariance with latent rollout consistency (LRC), the $R^2$ between the latent predicted autoregressively over $H$ steps and the latent inferred from the realised trajectory:

\mathrm{LRC}(H) = R^2\big(\mathbf h(\boldsymbol x_{\leq H}, \boldsymbol a_{\lt H}),\, \hat{\boldsymbol z}_H\big)

A faithful model has an LRC of 100% at every horizon. High LRC alone is not sufficient, because collapsed representations can be easy to predict, so post-hoc state decoders test how much of the state the representation retains.

Reward against LRC, linear decoding R² and nonlinear decoding R² for three ways a world model can fail to be faithful: noisy dynamics with an identity encoder, noisy dynamics with a nonlinear encoder, and a noisy encoder with exact dynamics — Fig. 2 · Three ways a world model can fail to be faithful. (A) Identity encoding, noisy dynamics. (B) Nonlinear encoding, noisy dynamics. (C) Noisy encoder, exact dynamics. Reward against LRC at H = 30 (top), linear decoding R² (middle) and nonlinear decoding R² (bottom); colour indicates increasing noise.

The three columns use CartPole swing-up with an oracle planner, model-predictive control on the true simulator, so there is no learned world model. In column A the encoder is the identity and noise is added to the oracle transitions. Linear and nonlinear decoding stay at $R^2 = 1$ , so probes cannot see the failure, while reward falls with LRC. Column B applies the same dynamics noise after an invertible nonlinear mix. Linear decoding is poor even without noise, because the coordinates are nonlinear, whereas nonlinear decoding stays at 1 and reward again tracks LRC.

Column C keeps the transitions exact and corrupts the inverse encoder. Decoding remains high and reward can stay high even as LRC collapses: encoder errors are amplified over the rollout before reward or probes break. Both the representation and the dynamics model can limit task performance, which motivates evaluating latent prediction, state decoding and reward together.

High reward does not imply faithful latent dynamics

01Reward does not reveal whether the dynamics were learned. Task reward alone does not show whether a world model learns accurate latent dynamics or preserves the underlying physical state; LRC and state decoding expose these failures directly.
02Pixel models struggle to recover velocity. Providing velocity explicitly improves latent prediction, but does not fully close the gap to state-based models.
03Nonlinear observations make dynamics hard to learn. Even when the underlying state is fully observed, nonlinear transformations make both the representation and its dynamics substantially harder to learn.

Three world-model families on three control tasks

DreamerV3 is reconstruction-based and reward-driven, TD-MPC2 is reconstruction-free but reward-driven, and LeWM is reconstruction-free and reward-free: it learns through self-supervised latent prediction and plans with model-predictive control as a post-hoc step.

Pendulum, CartPole and Reacher with their state variables, and the architectures of DreamerV3, TD-MPC2 and LeWM — Fig. 3 · (a) Pendulum, CartPole and Reacher from the DeepMind Control Suite, each evaluated with state and pixel observations. (b) The three world-model families, grouped by reconstruction and training objective: TD-MPC2 and DreamerV3 are task-specific (RL), LeWM is task-agnostic (dynamics learning + MPC).

Pendulum and CartPole have nontrivial unforced dynamics, whereas Reacher stays at rest without control. Every model is trained on raw state vectors and on rendered pixels, with four seeds per model, modality and environment. LRC is computed on 1,000 evaluation episodes of 100 steps from a random policy with broad state-space coverage, and reward is summed over episodes of 500 steps.

These environments are deliberately simple. Doing well on them does not imply that a model will scale to harder settings, but if it cannot learn accurate dynamics here, it is difficult to expect it to do so in more complex systems.

Task performance does not imply accurate latent prediction

Reward versus LRC at H = 30, LRC versus rollout horizon, and position and velocity decoding errors for state and pixel inputs — Fig. 4 · (a) LRC at H = 30 versus reward, averaged across environments; 95% CIs by bootstrap over four model seeds. (b) LRC over rollout horizons for state and pixel inputs. (c) Position-decoding errors of state- and pixel-based models across environments, normalised by a shuffled-target baseline. (d) Velocity-decoding errors, shown as in (c).

Reward and LRC capture distinct aspects of a world model. Models can reach high reward while their latent rollouts are inaccurate only a few steps into the future. The dissociation is clearest for the task-trained TD-MPC2 and DreamerV3, whose controllers solve the task despite very low LRC from pixels: their objectives may permit task-specific shortcuts, or policy and value networks may compensate for inaccurate latent dynamics.

LeWM is pretrained and then plans with model-predictive control, so it cannot take this shortcut by design. It needs sufficiently high LRC to obtain reward, and from pixels its rollouts do not reach that bar on the tested environments.

Pixel observations degrade latent prediction

State inputs provide the physical variables directly, whereas pixel inputs require the model to infer them from a high-dimensional nonlinear observation and from temporal context. Prediction degrades in both modalities, but pixel-based rollouts deteriorate faster and approach the persistence baseline (Fig. 4b).

CartPole rollouts decoded to physical space, state-space occupancy, and prediction errors binned over state space for state and pixel inputs — Fig. 5 · (a) Predicted latent rollouts are mapped to physical state with frozen nonlinear probes. (b) Example CartPole rollouts for DreamerV3, TD-MPC2 and LeWM, with the recorded actions above each trajectory. (c) Evaluation-set occupancy over pole angle and angular velocity (top) and cart position and velocity (bottom). (d) Prediction errors over the same state spaces for models trained from states or pixels; colour shows the error in each bin relative to the median error, annotated in physical units.

Across all three model families, predictions from pixels depart from the realised trajectory within 30 control steps, only 0.3 s of simulated time.

Accurate latent prediction needs an encoder that retains the predictive state and dynamics that evolve it correctly. Decoding the state from frozen latents separates the two. Positions are decoded with low error (0–7%) in most conditions, with small differences between state and pixel inputs. Velocities are decoded less accurately (3–22%), and state models outperform pixel models by a large margin (Fig. 4c–d).

On CartPole, velocity errors grow with large velocities, where the system evolves fastest and errors compound over the rollout. The pattern appears for both input modalities, but the median error is roughly 2–15× larger for pixel-based models.

Missing velocities explain part of the pixel–state gap

The physical state is Markovian, a single pixel frame is not: velocities have to be inferred from temporal context. On CartPole, pixel models receive ground-truth cart and pole velocities, at evaluation only at initialisation. Reducing the temporal context from three frames to one serves as a negative control.

Velocity fused with the visual representation, example CartPole rollouts with and without velocity, and LRC over the horizon for DreamerV3, TD-MPC2 and LeWM — Fig. 6 · (a) Ground-truth velocity is fused with the visual representation. (b–c) Example CartPole rollouts with and without velocity for TD-MPC2 and LeWM. (d–f) LRC over the rollout horizon for DreamerV3, TD-MPC2 and LeWM.

Adding velocity substantially improves latent prediction for TD-MPC2 and LeWM, with both reduced and full temporal context. For LeWM with three frames, LRC at $H = 30$ rises from 1.2% to 96.3%. It has no such effect on DreamerV3, possibly because the velocity reaches its RSSM posterior but not its prior.

The intervention changes the representation as intended. With a single frame, the normalised velocity-decoding error is about 130% for TD-MPC2 and LeWM, and providing velocity reduces it to about 1–2%.

Reward does not follow consistently. With three frames, TD-MPC2 changes little, LeWM stays far below its state-input reward (178 versus 757), and DreamerV3 performs worse. CartPole swing-up needs coordinated actions over more than 30 steps, so better LRC at $H = 30$ is not enough to plan the full motion. Velocity explains part of the gap between pixel and state inputs, but not all of it.

Learning latent dynamics from nonlinear observations

Pixels are also a nonlinear transformation of the physical state. On CartPole, we replace rendering with an invertible nonlinear map of the state through zero to three mixing layers: the input keeps all state information, only in a coordinate system that is harder to invert.

Rendering versus invertible state mixing, a CartPole episode under increasing mixing, and LeWM reward, LRC and decoding errors versus mixing depth — Fig. 7 · (a–b) Pixel rendering contrasted with approximately invertible mixing of the underlying state. (c–f) LRC at H = 10, task reward, normalised position error and normalised velocity error for LeWM versus mixing depth, with pixels as a reference.

We show LeWM, whose self-supervised objective most directly approximates the faithfulness condition. Task reward and LRC decline together as the number of mixing layers grows, and the controller solves only the conditions with zero and one layer.

With one mixing layer, the normalised position-decoding error is about 0.5% and the task is solved; with two layers it exceeds 1.5% and the task fails. The pixel model, slightly above 1%, does not solve the task either.

Because the mixing is invertible, these failures cannot be attributed to missing information in the input. The evaluated objectives struggle to learn representations from which the physical state is recoverable.

Read the paper

Preprint, 2026

Understanding the Limits of Latent World Models

Rodrigo González Laiz, Steffen Schneider

PDF22.8 MB