When Thinking Drifts: Evidential Grounding for Robust Video Reasoning

Romy Mi Luo1, Sherry Zihui Xue1, Alex Dimakis2,3, Kristen Grauman1

1UT Austin2UC Berkeley3Bespoke Labs

NeurIPS 2025

[Paper]First page of the paper [Video]Still from the overview video [Code]GitHub repository page for luomi777/Video-VER [BibTeX]@inproceedings{ luo2025videover, title={When Thinking Drifts...}, year={2025} }
Two storyboard examples where a model answers correctly without Chain-of-Thought, then talks itself into the wrong answer: one hallucinates that water is hot, the other ignores the most recent frames.

TL;DR: Chain-of-Thought can make video reasoning worse. We call the failure visual thinking drift — traces that storytell instead of look. Visual Evidence Reward rewards traces that stay grounded, and ranks first on 9 of 10 benchmarks.

A 6-minute introduction to visual thinking drift and Visual Evidence Reward.

Visual Thinking Drift

Chain-of-Thought reliably helps on text. On video, our study across 10 benchmarks, 3 MLLMs and 20 video-QA subtasks finds the opposite: asking the model to think before answering often lowers accuracy, especially for open-source models.

The traces are fluent and plausible, and wrong. As a chain grows longer it leans more on hallucinated facts and stale temporal context — concluding from language patterns rather than from the video. Through a Bayesian lens, the trace stops being conditioned on the frames and starts amplifying the model's own priors.

Bar charts on ten benchmarks: the Chain-of-Thought prompt scores below the direct-answer prompt for Qwen2.5-VL-3B, Qwen2.5-VL-7B and Video-R1 on most of them.
Figure 2. Compared to directly prompting for an answer, instructing the model to “think before answering” leads to a noticeable performance drop in open-source MLLMs such as Qwen2.5-VL and Video-R1 across multiple benchmarks (10 shown here).
Hallucinated detailThe trace asserts visual specifics the video never shows, then reasons confidently from them.
Overridden intuitionA correct first-glance answer gets argued away over several steps of ungrounded deliberation.

Visual Evidence Reward

VER targets the cause rather than the symptom. Rewarding only a correct final answer also rewards a lucky story, so VER explicitly rewards reasoning traces that are verifiably grounded in visual evidence.

An auxiliary LLM acts as a judge, scoring how well each intermediate thought aligns with visual evidence generated for that specific question. That score becomes a reward signal for RL post-training, so traces that drift into language priors stop paying off.

VER training loop: a policy model produces a group of candidate answers, an LLM judge compares them against generated visual evidence to produce the Visual Evidence Reward, which drives group relative gradient optimization against a reference model.
The VER training loop. The policy model samples a group of reasoning traces; an LLM judge scores each one against separately generated visual evidence; the resulting Visual Evidence Reward drives group-relative policy optimization, regularized against a frozen reference model. (Slide from the overview talk.)
Reward the evidence, not the storyThe judge checks whether the claims in a trace can actually be verified against the video.
Question-dependent evidenceEvidence generated for the specific question beats a generic caption on 9 of 10 benchmarks.

Large multimodal models should not only “think before answering”, but also “see while thinking”.

Results

Table comparing Video-VER with baselines across ten video benchmarks; Video-VER ranks first on nine of them.

Table 1. Comparison of Video-VER with baselines across 10 video benchmarks. Margins in small offset font are gains over its base model Qwen2.5-VL-7B, both with CoT. Note that for most base models CoT prompting lowers accuracy relative to direct answering — Video-VER maintains or improves it.

Video-VER-7B ranks first on 9 of 10 benchmarks, the exception being VSI-Bench. Against its own base model it gains up to +9.0 points absolute and +4.0 on average across all 10.

Example

Side-by-side reasoning traces: Qwen2.5-VL-7B and Video-R1-7B add speculative details and answer wrongly, while Video-VER-7B stays with observable evidence and answers correctly.
Figure 6. Reasoning traces from baselines and Video-VER. Baselines fold in speculative or hallucinated details (pink) that the video does not support; Video-VER keeps intermediate steps aligned with observable evidence (green).

BibTeX

@inproceedings{luo2025videover,
  title={When Thinking Drifts: Evidential Grounding for Robust Video Reasoning},
  author={Luo, Mi and Xue, Zihui and Dimakis, Alex and Grauman, Kristen},
  booktitle={Advances in Neural Information Processing Systems},
  year={2025}
}