Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models
A controlled benchmark separates language coverage from image- and text-dominant risks in multimodal safety evaluation.
The short version
Lingua-SafetyBench contains 100,440 harmful image–text pairs across ten languages. It explicitly separates image-dominant from text-dominant risks, so evaluation can examine which channel carries the harmful intent instead of conflating all multimodal inputs.
This design reveals an asymmetric safety landscape: high-resource languages and non-high-resource languages do not share the same pattern of vulnerability across risk modalities. Analyses of model scaling and iteration further ask whether average improvements benefit all language groups equally.
Multimodal safety needs evaluation across the interaction of language and risk-bearing modality.
How the benchmark is composed
The source distributions are shown separately for image-dominant and text-dominant risks.
View in the paperWhy this matters
Text-only multilingual tests and monolingual multimodal tests leave a gap at their intersection. Image rendering alone also cannot capture the range of semantic interactions between a visual input and an instruction in another language.
How it works
- Construct image–text safety examples with an explicit distinction between image-dominant and text-dominant risks.
- Extend the examples across ten languages and evaluate models under a consistent multilingual protocol.
- Disaggregate attack success rates by language, risk modality, safety category, and model scale or iteration.
Safety varies across languages
The radar plots compare multilingual safety across evaluated models and model iterations.
View in the paperWhat the study finds
The benchmark analysis includes eleven readily deployable VLLMs and controlled comparisons within the Qwen family. Reported results show substantial variation across languages and an interaction between language-resource level and the dominant risk modality. Scaling improves average safety while leaving uneven gains, particularly for text-dominant non-high-resource settings.
Average progress can hide a widening gap
The controlled model-family comparison reports different scaling gains for high-resource and non-high-resource languages.
View in the paperWhat this means
A single aggregate safety score can hide which users and input conditions remain poorly protected. Explicitly controlling the language and the source of risk makes those differences measurable and gives alignment work a more specific target.
The broader opportunity is to connect evaluation coverage to mechanisms of access and understanding. A low attack success rate alone does not reveal whether a model recognized the risk or failed to process the input. The benchmark’s factorized design supports further studies that distinguish these explanations.
Paper and resources
This is an introduction to the linked paper. Refer to the original for method definitions, evaluation settings, and complete results.
Cite this work
@misc{lingua_safetybench,
title = {Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models},
author = {Shi, Enyi and Shao, Pengyang and Zhang, Yanxin and Cui, Chenhang and Lyu, Jiayi and Xia, Xiaobo and Shen, Fei and Chua, Tat-Seng},
year = {2026},
eprint = {2601.22737},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2601.22737}
}