2025USC · course project

Compositional reasoning failures in vision-language models

CLIP scores near-random on negation and counting. Adapting the vision side fixes most of it.

PyTorchHuggingFace TransformersCLIP ViT-B/32LoRA / PEFTContrastive learningCOCO

Negation: 60 → 91.5%
Spatial: 54 → 100%
Attribute binding: 58.7 → 88%

Overview

Vision–language models show affirmation and confirmation bias: they tend to agree with a caption whether or not it matches the image. This project reframes that as a compositional reasoning failure and tests it along four axes — negation, numeracy, attribute binding, and spatial relations.

Bias-Bench is the evaluation harness built for it: COCO images plus synthetic scenes, with rule-based generation of hard negative captions designed to stress each axis specifically.

Challenges

Baseline CLIP (ViT-B/32) performed near-random on numeracy and compositional tasks with negligible confidence margins. The model wasn't uncertain, it was confidently wrong — which rules out threshold tuning as a fix.

The task had to be reformulated as contrastive multi-choice classification, optimizing cosine similarity under a temperature-scaled cross-entropy objective, before adaptation could target the failure directly.

Locating the bias meant ablating text-only, vision-only, and multimodal adaptation separately rather than adapting everything at once.

Decisions

The calls that shaped this project, the alternatives they beat, and what each one cost.

Build Bias-Bench instead of using an existing benchmark
over evaluate on a standard VLM benchmark
Why
Existing benchmarks report one aggregate score, which can't tell you whether a model fails at negation, counting, attribute binding, or spatial relations. Rule-based hard negatives over COCO and synthetic scenes isolate each axis separately.
Trade-off
Results aren't directly comparable to published numbers on other benchmarks.
Ablate text-only, vision-only, and multimodal adaptation separately
over adapt both encoders and report the gain
Why
Adapting everything would have improved the score without locating the problem. Splitting the ablation is what showed the bias sits on the vision side — text-side adaptation alone doesn't resolve it.
Trade-off
Three training regimes to run instead of one.
Reformulate as contrastive multi-choice classification
over the stock image-text matching objective
Why
Baseline CLIP wasn't uncertain on these tasks, it was confidently wrong — so threshold tuning couldn't fix it. Optimizing cosine similarity under a temperature-scaled cross-entropy objective targets the failure directly.
Trade-off
Moves further from CLIP's pretraining objective, risking drift on general retrieval.
LoRA adapters over full fine-tuning
over fine-tune both encoders end to end
Why
The question was whether these failures are correctable at all, not how far a fully retrained model can get. Low-rank adapters answer that cheaply and keep the baseline weights intact for comparison.
Trade-off
Leaves capacity on the table — numeracy may need more than an adapter can give.

Outcomes

Substantial gains over baseline CLIP: negation 60.0% → 91.5%, numeracy 31.3% → 58.7%, attribute binding 58.7% → 88.0%, spatial reasoning 54.0% → 100.0%.

Vision-side fine-tuning is what resolves confirmation bias. Text-side adaptation alone does not.

Numeracy stays the weakest axis, which reads as a structural limit of contrastive VLM architectures rather than a training problem — counting likely needs object-centric or counting-aware representations.

Next projectMultimodal sarcasm detection