CAST: Alternating State-Value Targets and Expanded Policy Gradients for Model-Based RL

Abstract

Model-based reinforcement learning (MBRL) is a family of RL methods that learn a model of the environment and use it for action selection, making it well suited to robotics due to its sample efficiency. Combining learned models with online planning can further improve action selection, as the planner can exploit the model to find better actions than the learned policy alone. Recent methods combining learned policies with online planning typically learn the value of the policy rather than the stronger planner-guided behavior. We present CAST (Critic with Alternating State-value Target), which uses planner-guided behavior to improve value learning while regularizing the value estimate with the current policy. CAST replaces the action-value critic with a state-value critic, trained using a target that combines a real planner-guided transition and an imagined transition under the current policy. The resulting value function corresponds to an alternating process between planner-guided behavior and the current policy, allowing it to benefit from the stronger planner behavior while being regularized by the policy being learned. We evaluate CAST on the DeepMind Control and HumanoidBench Suites against several state-of-the-art methods, and demonstrate successful transfer to a physical Unitree Go2 quadruped performing a dynamic handstand.

Method

CAST builds on TD-MPC2, retaining its latent world model and MPPI planner, while replacing the action-value critic $Q(z,a)$ with an ensemble state-value critic $V(z)$ trained toward the value of the planner-augmented behavior policy $\beta$.

Overview of CAST. A world model learns latent dynamics, rewards, and a state-value critic trained with alternating state-value targets that combine a replay transition with an imagined policy transition. The planner evaluates candidate trajectories using the terminal state value, while the policy is optimized through expanded policy gradients propagated across imagined rollouts.

The behavior policy $\beta$ (the planner) generally outperforms the learned policy $\pi_\theta$, especially early in training. We argue the critic should learn the value of $\beta$ rather than $\pi_\theta$, so that policy improvement can draw on a stronger critic. But bootstrapping directly from $\beta$ destabilizes training with overly large policy updates — a common RL failure mode. CAST resolves this with a regularized target that mixes one real transition under $\beta$ with one imagined transition under $\pi_\theta$:

$$y_t = r_t + \gamma\Big[R_\xi(z_{t+1}, \tilde a_{t+1}) + \gamma\, \bar V_\psi(\tilde z_{t+2})\Big], \qquad \tilde a_{t+1} \sim \pi_\theta(\cdot \mid z_{t+1})$$

Under fixed policies and exact dynamics, the resulting operator $\mathcal T_\text{CAST} = \mathcal T_\beta \mathcal T_\pi$ is a $\gamma^2$-contraction admitting a unique fixed point $V_\text{CAST}$—neither the value of $\beta$ nor of $\pi_\theta$ alone, but the value induced by their alternating composition. A state-value critic also removes the need to sample a terminal action during planning, and gradients for policy improvement are instead propagated through $k$ imagined transitions of the learned dynamics before bootstrapping with $\bar V_\psi$.

Ablations

We isolate the contribution of the hybrid value target and the expanded policy-gradient horizon on Dog-run and Humanoid-run.

Value-target ablation. Our hybrid target vs. purely on-policy and purely off-policy alternatives ($H_\pi{=}1$ throughout). The purely off-policy variant (bootstrapping directly from $V_\beta$) is consistently the worst of the three — it yields abrupt policy updates that destabilize learning, confirming the need for the alternating construction.

Behavior-to-policy ratio ablation. $H_V{=}1$ (CAST) vs. $H_V{=}2$ and $H_V{=}3$, i.e. anchoring the value target to more real transitions under $\beta$ before the single imagined step under $\pi_\theta$. On Dog-run, $H_V{=}2,3$ remain competitive with $H_V{=}1$; on Humanoid-run both lag behind, indicating that anchoring to a single real transition is a well-balanced choice.

Expanded-gradient-horizon ablation. $H_\pi{=}1$ (ours) vs. $H_\pi{=}2$ and $H_\pi{=}3$, with the hybrid value target fixed. Longer horizons track each other and $H_\pi{=}1$ closely, with $H_\pi{=}1$ holding a small lead and smaller confidence intervals on both tasks.

Policies Learned with CAST

CAST policies rolled out in the MuJoCo physics simulator, on both a quadruped (Unitree Go2, dynamic handstand) and a humanoid (Unitree R1, locomotion), including robustness to external pushes applied during the handstand.

@unpublished{crestaz2026cast,
  title  = {{CAST: Alternating State-Value Targets and Expanded Policy Gradients for Model-Based Reinforcement Learning}},
  author = {Crestaz, Pietro Noah and Kabouri, Mohamed Yassine and Mansard, Nicolas and Del Prete, Andrea},
  url    = {https://hal.science/hal-05739032},
  note   = {Working paper or preprint},
  year   = {2026},
}