Mental Imagery That Matters: Curing Latent Collapse in Multimodal Reasoning

Do Latent Visual Thoughts Actually Matter?

When humans reason about spatial layouts, physical dynamics, or transformations, we often simulate the answer in our head before speaking. A promising line of work brings this idea to multimodal LLMs by letting them emit continuous visual latent blocks between text, i.e., compact internal images the model can consult while reasoning:

<text₀, vis₀, text₁, vis₁, …, answer>

Architecturally, these visual states look load-bearing. But an architecture capable of using latents is not the same as one that does. We ask a simple diagnostic question: if we corrupt the model's own generated visual latents, does its answer change?

Across four recent latent-reasoning MLLMs, the answer is largely no. Dropping the visual blocks, replacing them with another sample's, or shuffling their order barely affects the model's final-answer probability. The same perturbations applied to the textual reasoning trace are substantially more damaging. We call this failure Mental Simulation Collapse (MSC): models exhibit the surface form of interleaved visual reasoning, but route almost all answer-relevant computation through text.

Diagnosing Mental Simulation Collapse

Vision vs. text perturbation impact across four latent-reasoning MLLMs

We probe four representative latent-reasoning models (Mirage-7B, SkiLa-7B, CoVT-7B, and Monet-7B) with a controlled perturbation test. For each model we capture its own greedy-decoded interleaved trajectory, then rescore the answer span after independently perturbing either the vision block or the reasoning text block under three conditions: Drop (remove), Cross (replace with another sample's), and Shuffle (permute).

The pattern is consistent across all four models. Perturbing the visual latents leaves the answer nearly untouched; perturbing the text trace collapses it. This asymmetry is striking because the visual perturbations directly target the very states these architectures were designed to introduce. If the models were genuinely using their visual mental states, corrupting those states should hurt. Instead, the model quietly recovers the answer from the linguistic stream. Per-sample regression fits (vision slopes 0.87–0.94 vs. text slopes 0.60–0.92) confirm this is not driven by outliers: at the level of individual trajectories, the visual latent content is largely inert.

Why the Latent Pathway Collapses

The current recipe for training latent-reasoning MLLMs is two-stage: Stage 1 aligns each generated latent \(z_\theta\) to a helper-image embedding \(\hat z\); Stage 2 drops the alignment loss and optimizes only the text likelihood on the interleaved trajectory. Neither stage actually forces the final prediction to depend on the latent.

Formally, text-only supervision cannot identify latent dependence: for any latent-dependent solution, a sufficiently expressive decoder can achieve nearly identical likelihood while ignoring the latent, \[p_{\theta'}(y \mid q, \mathbf{v}, t, z) \;\approx\; p_\theta\bigl(y \mid q, \mathbf{v}, t, G_\theta(q, \mathbf{v}, t)\bigr) \quad \forall z.\] From an information-theoretic view, collapse corresponds to \(I_\theta(y; z \mid q, \mathbf{v}, t) \approx 0\), i.e., the latent contributes no useful information about the output. Worse, once the decoder becomes invariant to \(z\), gradients through the latent pathway vanish and the collapse is stable. Adding visual supervision constrains how the latent looks, but not how it is used.

AnchorSim: Anchoring Reasoning to Simulation

AnchorSim targets the collapse mechanism directly. It is built on two complementary signals: a contrastive anchor that forces predictions with and without the visual pathway to differ, and a simulation-anchored teacher that supplies reasoning targets which actually consume those latents.

1. Contrastive Latent Anchoring

For every reasoning step we form two next-token distributions from the student policy \(\pi_S\): one conditioned on the full context \(C_{<k}\), and one on a visual-blind context \(\tilde C_{<k}\) where all previously generated latents \(\{z_1,\ldots,z_{k-1}\}\) are masked out. We then penalize agreement between the two, since agreement means the visual stream contributed nothing:

\[\mathcal{L}_\text{anchor}(\theta) = -\,\mathbb{E}_{\mathcal{T},k}\!\left[\frac{1}{|t_k|}\sum_{n=1}^{|t_k|} \mathrm{JSD}\!\Bigl(\pi_S(\cdot \mid C_{<k}, t_{k,<n}) \,\Big\|\, \mathrm{sg}\bigl[\pi_S(\cdot \mid \tilde C_{<k}, t_{k,<n})\bigr]\Bigr)\right]\]

The stop-gradient on the blind branch prevents the model from degrading its blind pass to "win" the contrast, so the only way to increase the divergence is to make the latent-aware pass carry real, consequential visual content.

2. Anchored Reasoning Distillation

Contrastive anchoring says use the latents; distillation says use them well. We keep an exponential-moving-average teacher \(\pi_T\) that shares the student's architecture but is conditioned on the ground-truth answer as a privileged prefix. This lets the teacher interpret each student-generated latent \(z_{k-1}\) and produce a textual reasoning chunk that coherently bridges the visual state to the correct answer. The student distills this answer-anchored reasoning:

\[\mathcal{L}_\text{distill}(\theta) = \mathbb{E}_{\mathcal{T},k}\!\left[\frac{1}{|t_k|}\sum_{n=1}^{|t_k|} \mathrm{JSD}\!\Bigl(\pi_T(\cdot \mid C_{<k}, a, t_{k,<n}) \,\Big\|\, \pi_S(\cdot \mid C_{<k}, t_{k,<n})\Bigr)\right]\]

The AnchorSim Objective

The full training loss combines the two signals:

\[\mathcal{L}_\text{AnchorSim}(\theta) = \mathcal{L}_\text{distill}(\theta) + \lambda\,\mathcal{L}_\text{anchor}(\theta)\]

At inference, the visual-blind branch is discarded and the model simply generates standard interleaved trajectories, now with latents that are causally consequential to the answer.

Results

We post-train Qwen2.5-VL-7B with AnchorSim and evaluate on nine public visual reasoning benchmarks spanning perception (BLINK, MMVP, V*), 2D/3D spatial reasoning (CV-Bench, SAT-Real), abstract and multimodal STEM (EMMA, VisuLogic), and visual planning and step-wise manipulation (VSP, CoMT). ANCHORSIM-7B achieves the strongest overall performance, attaining an average of 62.4%, outperforming the Qwen2.5-VL-7B base by +13.5 points and the Stage-1 interleaved model by +2.7 points. It ranks first on seven of nine benchmarks and second on two.

The gains are largest on benchmarks that require multi-step visual state tracking (VSP (+68.7) and CoMT (+18.5) over base), exactly where genuine mental simulation should help. Crucially, when we re-run the perturbation diagnostic on ANCHORSIM-7B, corrupting the visual latents now meaningfully degrades predictions: the model has learned to rely on what it draws in its head.

@article{luo2026anchorsim, title={Mental Imagery That Matters: Curing Latent Collapse in Multimodal Reasoning}, author={Luo, Mi and Dugad, Rohan and Xue, Zihui and Dimakis, Alex and Grauman, Kristen}, year={2026} }