Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning
Image ablation separates visually grounded knowledge, directional priors, and representations deployed with the wrong sign.
The short version
This study examines a familiar interpretability signature: a probe decodes the answer, and steering improves performance. In VLM spatial reasoning, that combination can conceal a directional prior that persists when the image is replaced by a gray blank. The apparent recovery can therefore overstate visual grounding.
The vision-ablation arbiter separates three regimes: grounded and correct, vision-independent prior, and visually dependent but inverted deployment. In the last regime, information is decodable and behavior is controllable, yet the model uses the relevant direction with the wrong sign.
Decoding or steering a representation does not establish that the model used the image to make the decision.
Decodability and visual dependence can diverge
Each point relates probe accuracy to the behavioral effect of removing the image.
View in the paperWhy this matters
A model can expose a linearly decodable label without using the intended evidence in its deployed computation. Demonstrating that a representation exists and can affect output leaves a further question: does its behavioral effect depend on the visual input?
How it works
- Compare spatial-task behavior with probe decodability and controlled representation interventions.
- Replace or disrupt the visual input while retaining the task, using blank-image and additional ablation controls.
- Classify the resulting grounding regime and examine which minimal representation edit corrects inverted deployment.
Does steering recover visual evidence?
The plot compares steering with real and blank images to test whether recovery depends on vision.
View in the paperWhat the study finds
The study examines fourteen VLMs across six language-model families and separates grounding regimes across spatial axes. Additional tests compare several vision ablations and investigate decode-versus-deploy inversion across model families. Intervention results show that the correction needed can range from a training-free rotation to a learned low-rank edit, depending on representation geometry.
A cross-model grounding card
The original matrix summarizes spatial-axis behavior and grounding regimes across fourteen VLMs.
View in the paperWhat this means
The work separates four pieces of evidence that are easy to conflate: information can be decoded, an intervention can control output, the output can be correct, and the decision can depend on the image. A stronger account of latent knowledge needs to state which of these has actually been demonstrated.
This makes a simple counterfactual especially valuable. Removing visual evidence can challenge an otherwise convincing probe-and-steering story, while additional ablation controls test whether the result reflects an artifact of the blank input. The broader lesson is to make grounding a measured property of the deployed computation.
Paper and resources
This is an introduction to the linked paper. Refer to the original for method definitions, evaluation settings, and complete results.
Cite this work
@misc{decodable_not_grounded,
title = {Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning},
author = {Liao, Chih-Ting and Shen, Fei and Cao, Xin and Chua, Tat-Seng},
year = {2026},
eprint = {2606.31257},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.31257}
}