VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Models

Wang, Chenyu; Chen, Tianle; Wu, Yuxin; He, Qijia; Chen, Mingyu; Ahmad, H. M. Sabbir; Batmanghelich, Kayhan; Li, Wenchao

Vision-language models · Uncertainty quantification · Benchmark

VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Models

Chenyu Wang^1*, Tianle Chen^1*, Yuxin Wu², Qijia He³, Mingyu Chen¹, H. M. Sabbir Ahmad¹, Kayhan Batmanghelich¹, Wenchao Li¹

CVPR 2026 TRUE-V Workshop

¹Boston University ²Amazon ³University of Washington ^*Equal contribution

Benchmark composition. Upper left: data annotation, rule-based filter, and expert-level filter. Right: VizWiz examples for the Clean, Image, Text, and Cross subsets, and unambiguous and ambiguous examples from VQ-FocusAmbiguity. Bottom left: a CLEVR scene with attribute, existence, counting, and relation questions. — **Benchmark composition and curation workflow.** Human annotation, rule-based filtering, and expert curation (upper left) yield four VizWiz-based subsets with modality-specific uncertainty labels (Clean, Image, Text, Cross); VQ-FocusAmbiguity provides grounded cross-modal ambiguity (middle right), and CLEVR scene graphs give attribute, existence, counting, and relation hallucination subsets (bottom left).Swipe to see the whole figure.

Abstract

Uncertainty quantification (UQ) is vital for ensuring that vision-language models (VLMs) behave safely and reliably. A central challenge is to localize uncertainty to its source, determining whether it arises from the image, the text, or misalignment between the two. We introduce VLM-UQBench, a benchmark for modality-specific and cross-modal data uncertainty in VLMs. It consists of 600 real-world samples drawn from the VizWiz dataset, curated into clean, image-, text-, and cross-modal uncertainty subsets, and a scalable perturbation pipeline with 8 visual, 5 textual, and 3 cross-modal perturbations. We further propose two simple metrics that quantify the sensitivity of UQ scores to these perturbations and their correlation with hallucinations, and use them to evaluate a range of UQ methods across four VLMs and three datasets.

Empirically, we find that: (i) existing UQ methods exhibit strong modality-specific specialization and substantial dependence on the underlying VLM, (ii) modality-specific uncertainty frequently co-occurs with hallucinations while current UQ scores provide only weak and inconsistent risk signals, and (iii) although UQ methods can rival reasoning-based chain-of-thought baselines on overt, group-level ambiguity, they largely fail to detect the subtle, instance-level ambiguity introduced by our perturbation pipeline. These results highlight a significant gap between current UQ practices and the fine-grained, modality-aware uncertainty required for reliable VLM deployment.

Where Does the Uncertainty Come From?

Effective UQ must indicate both how uncertain the model is and where that uncertainty arises.

Image

Visual uncertainty

Low-quality image artifacts such as blur, occlusion, or poor lighting.

“What title of this book?”

Next step: retake the image.

Text

Textual uncertainty

Invalid or subjective questions that are vague or incoherent.

“face. I am trying to ask him a question.”

Next step: rephrase the question.

Cross

Cross-modal uncertainty

Ambiguous references or insufficient visual evidence given an otherwise clear question and image.

“What is it that i am looking at?”

Next step: defer.

The Benchmark

VLM-UQBench combines four components into a unified benchmark for uncertainty in VLMs.

600
Human-annotated modality subsets (VizWiz). Three domain-expert PhD annotators curate 150 samples for each modality-specific subset, plus a clean subset.
140
Grounded cross-modal ambiguity (VQ-FocusAmbiguity). 60 ambiguous and 80 unambiguous examples with segmentation-based grounding of all valid referents.
4
Hallucination-focused subsets (CLEVR). Attribute, existence, counting, and relation questions generated from scene graphs.
16
Synthetic perturbation pipeline. Converts standard datasets into uncertainty-aware benchmarks without additional labels, with 8 visual, 5 textual, and 3 cross-modal perturbations.

Pipeline: a clean image of a clock with the question 'What time is it?' is perturbed visually (blur, dark, pixelate, salt and pepper, cutout), textually (shuffle, typo, subjective rewrite), and cross-modally (ambiguous reference, insufficient evidence in text or image); all variants go to open and closed VLMs, whose token probabilities, entropy, and sampled generations feed white-box and black-box UQ methods evaluated with AUROC, F1, URR, and HCC. — **Overview of our pipeline.** A clean image–question pair is processed through three stages: uncertainty perturbation, variant generation, and uncertainty calculation. UQ scores computed from the VLM outputs are evaluated with AUROC, F1, URR, and HCC.Swipe to see the whole figure.

Visual · 8 Blur · Brightness (dark) · Brightness (bright) · Cutout · Noise · Pixelate · Salt & pepper · Solarize

Textual · 5 Drop words · Invalid rewrite (INV) · Subjective rewrite (SBJ) · Shuffle · Typos

Cross-modal · 3 Ambiguous reference (AMB) · Insufficient visual evidence (IVE-text, IVE-visual)

Comparison with existing uncertainty benchmarks

Benchmark	Task / modality	Annotation type	Contrast granularity
AmbigQA	Open-domain QA	Ambiguity	Group
AbstentionBench	Open-domain QA	Abstention scenarios	Group
UNK-VQA	Open-ended VQA	Answerability	Group
VQ-FocusAmbiguity	Free-form VQA	Ambiguity (grounded)	Group
MM-UPD	Multiple-choice VQA	Unsolvability categories	Group
MuirBench	Multiple-choice VQA	Answerability	Sample
CertainlyUncertain	Free-form VQA	Aleatoric–epistemic uncertainty	Sample
VLM-UQBench (ours)	Free-form VQA	Modality-specific & cross-modality uncertainty	Sample

Metrics

AUROC and F1 assess how well each UQ method distinguishes uncertain samples from clean ones. Two new measures link UQ to modality-targeted perturbations and hallucination behavior.

Uncertainty Reflection Rate (URR)

A higher URR indicates that method more consistently increases its uncertainty under perturbation .

Hallucination Consistency Coefficient (HCC)

The point-biserial correlation between the uncertainty change and the hallucination label . A positive HCC indicates that hallucinated cases tend to exhibit larger uncertainty increases.

Models Kosmos-2, LLaVA, Qwen-VL, GPT-4o-mini · UQ methods 9 (6 white-box, 3 black-box) · Data VizWiz, VQ-FocusAmbiguity, VQAv2, CLEVR

Results

RQ1 · UQ methods specialize by modality and depend on the model

UQ methods exhibit clear modality-specific specialization, and their performance is strongly model-dependent. On the curated VizWiz subsets, Semantic Entropy is particularly effective for textual uncertainty.

Swipe to see all four models.

AUROC for discriminating uncertain vs. clean samples. Outlined: best in the column. “–”: missing token-level statistics.

All values on the VizWiz subsets (AUROC / F1)

Estimator	Kosmos-2			LLaVA			Qwen-VL			GPT-4o-mini
Estimator	Image	Text	Cross	Image	Text	Cross	Image	Text	Cross	Image	Text	Cross

Overt ambiguity: UQ rivals chain-of-thought

On grounded cross-modal ambiguity (VQ-FocusAmbiguity), the best UQ estimator matches or exceeds zero-shot, ZS-CoT, and ZS-ECoT prompting.

F1 for VQ-FocusAmbiguity detection: strongest UQ estimator vs. prompting
Model	UQ-best	ZS	ZS-CoT	ZS-ECoT
Qwen-VL	0.69	0.63	0.63	0.62
GPT-4o-mini	0.66	0.62	0.63	0.64

All estimators on grounded ambiguity (AUROC / F1)

Estimator	Kosmos-2	LLaVA	Qwen-VL	GPT-4o-mini

Bold: highest AUROC per model. “–”: no access to model-internal information.

RQ2 · Sensitivity to controlled perturbations is inconsistent

On VizWiz and VQAv2, UQ sensitivity varies substantially across datasets, models, and perturbation types. UQ methods are overall most effective at detecting SBJ textual perturbations, and worse at capturing cross-modal perturbations injected with minimal semantic change.

URR heatmap for Kosmos-2 on VizWiz: rows are UQ estimators, columns are cross-modal, visual, and textual perturbations at several intensities. — **Uncertainty reflection rate** under fine-grained image-, text-, and cross-modality perturbations for the selected dataset and model.Swipe to see the whole figure.

RQ3 · Modality-specific uncertainty induces hallucinations that UQ fails to detect

Modality-specific uncertainty induces distinct hallucination types: visual blur alone causes a 64.7% hallucination rate on attribute tasks for Qwen-VL.

Hallucination rate (%): items correct on the clean sample that become incorrect on the perturbed counterpart.

All hallucination rates (%)

Perturbation	LLaVA				Qwen-VL				GPT-4o-mini
Perturbation	Attr.	Count.	Rel.	Exist.	Attr.	Count.	Rel.	Exist.	Attr.	Count.	Rel.	Exist.

UQ methods fail to detect these perturbation-induced hallucinations: HCC(ΔU) is weak and model-dependent. HCC is low for visual blur across all models, despite it being a primary cause of hallucinations, and higher for low-impact perturbations such as typos.

HCC heatmap on CLEVR-Existence for Qwen-VL: rows are UQ estimators, columns are visual perturbations (blue labels) and textual perturbations (orange labels). — **HCC(ΔU) on CLEVR-Existence.** Rows list UQ estimators; columns are visual (blue) and textual (orange) perturbations. Cells show the point-biserial correlation between hallucination flips and uncertainty change; higher (warmer) is better. “–” (GPT-4o-mini): the API does not expose full token-level distributions.

Takeaways

Specialized, not general. UQ performance exhibits strong modality-specific specialization, and the optimal UQ method shifts for each VLM architecture.
Overt versus subtle ambiguity. UQ methods can serve as strong baselines for overt, group-level ambiguity, but break down on subtle, sample-wise AMB perturbations constructed with minimal semantic changes.
Weak link to hallucination risk. Modality-specific uncertainty drives hallucinations, yet the corresponding changes in UQ scores are weak and inconsistent across models.
A basis for better UQ. VLM-UQBench provides a basis for developing and validating more robust, modality-aware UQ methods.

Citation

@article{wang2026vlmuqbench,
  title   = {{VLM-UQBench}: A Benchmark for Modality-Specific and Cross-Modality
             Uncertainties in Vision Language Models},
  author  = {Wang, Chenyu and Chen, Tianle and Wu, Yuxin and He, Qijia and
             Chen, Mingyu and Ahmad, H. M. Sabbir and Batmanghelich, Kayhan and
             Li, Wenchao},
  journal = {arXiv preprint arXiv:2602.09214},
  year    = {2026}
}