Abstract
Vision Language Models (VLMs) are widely deployed in safety-critical scenarios, and understanding to which extent they can be controlled by adversarial perturbation is a prerequisite for evaluating their trustworthiness. Existing representation-alignment attacks, which make a VLM perceive a target image, achieve limited success at $\varepsilon \leq 4/255$. Therefore, VLMs seems robust to perturbations in this range. We show that this robustness does not hold, as targeted semantic substitution succeeds within the same range. Specifically, we align each stream of the source image with its counterpart in the target image in the victim VLM's post-merger token space, operating under a white-box threat model. We evaluate under a strict success criterion, requiring the model to simultaneously name the target, confirm its presence, and deny the source. In images, target semantics appear at $\varepsilon = 2/255$ and complete replacement reaches 38% at $\varepsilon = 4/255$. On video, complete replacement reaches 35.9% at $\varepsilon = 1/255$. We also observe a phenomenon of semantic fusion, where Large Language Model (LLM) rationalizes contradictory visual signals into a coherent narrative.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Zhang, B., Narayan, A., & Sarkar, A. (2026). It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at $ε\leq 4/255$. https://omanscience.com/en/articles/it-takes-little-to-rewrite-perception-targeted-semantic-substitution-in-vision-language-models-at-leq-4-255
MLA 9
Zhang, Binchi, et al. "It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at $ε\leq 4/255$." https://omanscience.com/en/articles/it-takes-little-to-rewrite-perception-targeted-semantic-substitution-in-vision-language-models-at-leq-4-255.
Chicago (author–date)
Zhang, Binchi, Apurva Narayan, and Atrisha Sarkar. 2026. "It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at $ε\leq 4/255$." https://omanscience.com/en/articles/it-takes-little-to-rewrite-perception-targeted-semantic-substitution-in-vision-language-models-at-leq-4-255.
Harvard
Zhang, B., Narayan, A. and Sarkar, A. (2026) 'It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at $ε\leq 4/255$', Available at: https://omanscience.com/en/articles/it-takes-little-to-rewrite-perception-targeted-semantic-substitution-in-vision-language-models-at-leq-4-255.
Vancouver
Zhang B, Narayan A, Sarkar A. It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at $ε\leq 4/255$. https://omanscience.com/en/articles/it-takes-little-to-rewrite-perception-targeted-semantic-substitution-in-vision-language-models-at-leq-4-255
IEEE
B. Zhang, A. Narayan, and A. Sarkar, "It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at $ε\leq 4/255$," https://omanscience.com/en/articles/it-takes-little-to-rewrite-perception-targeted-semantic-substitution-in-vision-language-models-at-leq-4-255.