AdaAlloc
Adaptive Visual Token Allocation for Long-Video Question Answering
The University of Texas at Austin
NeurIPS 2026
AdaAlloc is a training-free agentic framework for long-video reasoning. An MLLM plans how to split a fixed visual-token budget between global context and high-resolution evidence, calls a temporal grounding tool to find that evidence, refines what comes back, and then answers.
points over budget-matched uniform sampling on LVBench (32K tokens), the hour-scale benchmark.
benchmark-and-budget settings improved: LVBench, MLVU, VideoMME, and LongVideoBench at 16K and 32K.
faster than a recent frame-selection method at 16K tokens, with 39% fewer FLOPs.
An agentic framework that plugs into an existing MLLM, with no fine-tuning.
An MLLM’s context window is limited. Where should the visual tokens go?
What does the person add to the pan right after chopping the onion?
See broadly, inspect selectively.
Abstract
Not just which frames, but how many tokens to spend where
Long-video question answering forces multimodal large language models to reason under strict visual-token budgets, creating a tradeoff between broad temporal coverage and fine-grained spatial detail. Uniform sampling preserves coverage but often obscures decisive local cues, while caption-based or frame-similarity retrieval operates over lossy proxies that can miss motion, state changes, and before–after relations. We recast long-video reasoning as a visual-token allocation problem: given a fixed budget, decide how much to spend on global context versus high-resolution local evidence, and where in the video to draw that local evidence from. We introduce AdaAlloc, a training-free inference framework that adaptively allocates visual tokens between a global video overview and localized visual detail. AdaAlloc first plans a global/local allocation policy, then uses temporal grounding to locate candidate evidence segments and refines them into a compact temporal memory. The selected policy controls both frame sampling density and resolution, combining sparse, lower-resolution global frames with denser, higher-resolution local frames drawn from the refined memory. The final answerer reasons over the allocated global context and refined local evidence. Across LVBench, MLVU, VideoMME, and LongVideoBench, AdaAlloc improves over budget-matched uniform sampling by up to 10.5 percentage points and outperforms the compared frame-selection and agentic methods on their reported benchmarks.
Method
Three stages, one budget
AdaAlloc splits a fixed visual-token budget B between low-res global context (blue) and high-res local evidence (orange). Like an agent, the MLLM plans, calls a temporal grounding tool, checks what comes back, and answers. Every MLLM call—planning, refinement, and answering—is capped at the same B.
From a sparse, low-res overview O, an MLLM planner picks one of five allocation policies π, writes grounding queries 𝒬, and proposes temporal seeds 𝒮 from timestamps or overview-visible events.
UniversalVTG, a ~60M-parameter temporal grounder (try the demo), localizes each query in the full video. The MLLM then keeps, discards, adjusts, and merges these hypotheses into a compact memory ℳ*—here dropping an oven clip and the first return to the stove.
The policy sets Bg = ρgB for sparse low-res frames over the whole video and Bl = ρlB for dense high-res frames from ℳ*. One MLLM call answers from both streams.
Five allocation policies. How the planner can split the budget
Each frame is encoded into visual tokens, image patches that take up room in the context window. Here a standard frame costs 2 tokens, a lower-res frame 1, and a high-res frame 4, so every close look is paid for with coverage.
Uniform sampling is the global-only endpoint, and retrieval-only pipelines approximate the local-only endpoint. When uniform sampling is already dense enough for a video under the budget, AdaAlloc skips grounding and answers directly from the global frames.
A real example from LVBench
Reading one pencil-written number in a 58-minute video
The deciding clue is a boxed number on a hand-drawn chart; the benchmark’s annotated evidence for it lasts 9 seconds. Step through what AdaAlloc does.
“What number is highlighted with rectangle in the chart written with pencil in numerical increasing order for each name?”
Sparse overview O · low-res
Planner output · policy π
Reading a handwritten number needs pixels, so most of the budget goes to local detail.
Evidence queries 𝒬
- A chart written with pencil was visible.
- A number was highlighted with a rectangle on the chart.
- The highlighted number was clearly visible.
Candidates 𝒞 from UniversalVTG · 11 proposals merged into 6
Global Eg · 0.3B · low-res, sparse over the full video
Local El · 0.7B · high-res from ℳ*
Answer
Uniform sampling, with the same model and budget, answers 5.
Frames are from the LVBench video; the close-up is enhanced for legibility. The paper version of this example is Figure 3.
Results
Consistent gains under matched budgets
Unless noted, every method uses Qwen3-VL-8B-Instruct with the same visual-token budget for the final answer. Video only, no subtitles. Accuracy in %.
Budget-matched uniform sampling against AdaAlloc. Gains are largest on the longest videos.
Same backbone, recent frame-selection methods. Their reported 128- and 256-frame settings (≈19K and ≈38K tokens) give them more visual tokens than AdaAlloc.
End-to-end wall-clock runtime on NVIDIA A40 GPUs, including AdaAlloc's planning and refinement calls. AdaAlloc costs more than single-pass uniform sampling, but is 3.1× (16K) and 2.1× (32K) faster than DIG.
Compute (TFLOPs) — 16K: uniform 468, DIG 1,871, AdaAlloc 1,146. 32K: uniform 1,261, DIG 2,664, AdaAlloc 2,873. DIG's query classifier (Qwen3-Next-80B-A3B) runs on eight A40s; everything else runs on one.
Analysis
What does the planner choose?
Mixed global/local policies dominate on every benchmark. Local-heavy is the most common choice on LVBench, MLVU, and LongVideoBench, while VideoMME splits between balanced and local-heavy.
Share of planner-routed questions. Questions that AdaAlloc answers directly, because uniform sampling is already dense enough, are excluded.
By NExT-QA question family (val, 16K)
Families are benchmark metadata the planner never sees. Descriptive questions pick global-oriented policies most often; temporal ones have the highest local-heavy rate.
Citation
BibTeX
@inproceedings{an2026adaalloc,
title = {AdaAlloc: Adaptive Visual Token Allocation for
Long-Video Question Answering},
author = {An, Joungbin and Grauman, Kristen},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026}
}