Not All Tokens Should Be Treated Equally:
Context Credits Reassignment

Abstract

Multimodal reinforcement learning with verifiable rewards (RLVR) algorithms assign the same advantage to every token in a generated sequence, providing no differential signal for tokens with varying degrees of dependence on visual evidence. A simple experiment confirms this issue: scaling token advantages by uninformative random noise already outperforms uniform credit, suggesting that uniform credit is empirically suboptimal. Beyond that, we observe that RLVR tuning under uniform credit weakens visual grounding along three complementary axes: attention, gradient attribution, and functional dependence on image embeddings, collectively indicating insufficient utilization of visual evidence. To understand this degradation, we provide a theoretical analysis showing that, under language-prior dominance and regularity conditions, uniform credit can amplify a pretrained text-favored gradient imbalance, contributing to weaker visual attention. In response, we propose Hierarchical Context Credit Reassignment (HiCCR), which reassigns credit at two levels: a token-level weight amplifies the advantage for visually grounded tokens, and a trajectory-level weight upweights rollouts with stronger visual engagement. The entire mechanism adds less than 3% wall-clock overhead and applies to mainstream RLVR algorithms without auxiliary models or additional data. On ten benchmarks spanning mathematical and general multimodal reasoning, HiCCR consistently improves upon its corresponding base algorithm and achieves state-of-the-art results among open-source models of comparable scale, while alleviating the weakening of visual grounding observed under RLVR training.

Motivation

GRPO and DAPO broadcast one trajectory-level advantage  to every token, whether the token reads a clock hand from the image or merely writes “Therefore”. Two observations suggest this is a poor default.

1. Uniform credit is suboptimal. Multiplying each token's advantage by uninformative noise w_t ∼ 𝒩(1, 0.2²) already outperforms uniform credit in all four settings (GRPO/DAPO × Qwen3-VL-4B/8B), e.g., 68.29 → 71.40 AVG@8 for DAPO on 8B.
2. Uniform credit weakens visual grounding. Compared with the base model, RLVR-tuned models attend less to the image, assign a smaller share of gradient attribution to image tokens, and lose less accuracy when image embeddings are zeroed out.

Base → GRPO → HiCCR-G on Qwen3-VL. Uniform credit erodes all three signals of visual grounding; HiCCR restores each beyond the base model. Values are read from Figure 1: attention is relative to GRPO (= 100), gradient share is the image fraction of Input×Gradient attribution, and the drop is measured on CLEVR-Count.

Method

HiCCR derives every weight from a single quantity, the context credit C_V(t): one minus the normalized entropy of token t's attention restricted to the visual positions V, averaged over heads of the last two layers. A token that locks onto a few image regions scores high; a token whose visual attention is diffuse scores low.

The two weights are applied asymmetrically: only positive advantages are reshaped, so that incorrect rollouts are not penalized merely for attending to the image.

HiCCR is a drop-in change to the advantage. We denote it HiCCR-G on GRPO and HiCCR-D on DAPO. Defaults: λ_tok = 0.5, τ = 0.5. The score reuses queries and keys from the forward pass, so no extra backward pass is required.

Main Results

We train on ViRL39K and compare against nine RLVR algorithms reproduced under identical settings. Accuracy is averaged over eight runs (AVG@8) on nine reasoning benchmarks: five mathematical (Geometry3K, MathVerse, MathVista, WeMath, MathVerse-Vision) and four general/logical (MMK12, CLEVR-Count, LogicVista, MMMU).

• Consistent gains. HiCCR improves its base algorithm in all four matched settings: +7.99 / +1.37 on 4B and +4.70 / +4.47 on 8B (GRPO / DAPO).
• Comparison with larger models. HiCCR-D on Qwen3-VL-8B (72.76) surpasses Qwen2.5-VL-72B (61.65), InternVL2.5-78B (55.26), GPT-4o (60.95) and Gemini-2.0-flash (61.40) on the same benchmarks (Table 7 in the paper).
• Generalization. Gains carry over to InternVL3-8B (+3.21 / +3.23), to training on Geometry3K only (+2.7 to +3.2), and to perception and hallucination benchmarks never used for training (MMStar 72.2%, HallusionBench 62.7%).
• Efficiency. On Qwen3-VL-8B, HiCCR adds 2.7% time per step (619 → 636 s) and 4.5% peak memory, while the perturbation-based PAPO adds 26.3% time.

Analysis

Which credit signal matters?

We keep the training recipe fixed and only change how token-level credit is assigned. Random and heuristic weightings give small gains; applying the same formula to text context or using unnormalized attention energy is clearly worse. Concentration within the visual tokens is the signal that matters.

Further ablations in the paper: TCR and TAR are complementary (Table 3); soft weighting beats hard top-k masking (Table 8); the last two layers give the best signal (Table 9); the asymmetric rule outperforms symmetric reshaping by 5.1 / 4.0 points (Table 10); results are stable for λ_tok ∈ [0.5, 0.8] (Table 11).

Where does the credit go?

HiCCR receives no token-level supervision, yet the highest credit lands on tokens that refer to visual content (e.g., triangle, angle, diagram). Visually grounded tokens receive significantly more credit than other tokens (median β 1.303 vs. 1.236, Cohen's d = 0.547, p < 0.001).

BibTeX

@inproceedings{gao2026hiccr,
  title     = {Not All Tokens Should Be Treated Equally: Context Credits Reassignment},
  author    = {Gao, Shujian and Wang, Yuan and Yan, Jiamei and Zhou, Penghao and
               Wang, Qinglei and Yan, Jiangtao and Wu, Zuxuan and Jiang, Yu-Gang},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}