Vision-Language Model Attacks

Comprehensive overview of the VLM attack surface, how vision encoders connect to language models, and why multimodal systems create new injection vectors.

What Are Vision-Language Models?

Vision-Language Models (VLMs) represent one of the most significant advances in AI capability -- and one of the largest expansions of the attack surface. Models like GPT-4o, Claude, and Gemini can see images, read documents, and reason about visual content alongside text instructions.

This capability introduces an entirely new class of vulnerabilities that do not exist in text-only language models.

The VLM Attack Surface

The attack surface of a VLM can be decomposed into four layers:

Input Encoding Attacks

The vision encoder transforms raw pixels into embedding vectors. This transformation is differentiable, meaning attackers can use gradient-based methods to craft images that produce specific embeddings -- even if the image looks innocuous to a human.

Cross-Modal Alignment Attacks

VLMs must bridge the gap between vision and language representations. The alignment layer (often a learned linear projection or cross-attention module) is trained on paired image-text data. Gaps in this training create exploitable misalignment -- situations where the model "reads" something from an image that a human would not.

Fusion-Level Attacks

Once image tokens enter the language model's context window, they are processed alongside text tokens. This creates the opportunity for image-based prompt injection -- hiding instructions in images that override or supplement the text prompt.

Output-Level Bypass

Many safety mechanisms were designed for text-only models. When harmful content is delivered through the visual modality, these text-centric filters may fail to detect it.

Why Multimodal Creates New Risks

Three structural factors make VLMs more vulnerable than text-only models:

1. Expanded Input Bandwidth

A single image can encode far more information than a typical text prompt. A 1024x1024 image contains roughly 3 million values (pixels x channels). This massive input space provides ample room to hide adversarial content that is imperceptible to humans.

2. Modality Asymmetry in Safety Training

Most RLHF and safety fine-tuning has focused on text. Visual safety training is newer, sparser, and harder to make comprehensive. The space of harmful images is vastly larger and more difficult to enumerate than harmful text patterns.

3. Cross-Modal Confusion

When the model must reconcile information from two modalities, conflicts between them can be exploited. An image showing one thing while the text says another can push the model into undefined behavior -- the security equivalent of a race condition.

Taxonomy of VLM Attacks

VLM Attacks
├── Image-Based Prompt Injection
│   ├── Visible text injection
│   ├── Steganographic injection
│   └── QR / barcode injection
├── Adversarial Image Examples
│   ├── Pixel-level perturbation (PGD, FGSM)
│   ├── Patch attacks
│   └── Transferable adversarial examples
├── OCR & Typographic Attacks
│   ├── Adversarial font rendering
│   ├── Text overlay manipulation
│   └── Unicode visual spoofing
├── VLM-Specific Jailbreaks
│   ├── Cross-modal inconsistency
│   ├── Visual safety bypass
│   └── Encoded instruction images
└── Architecture Exploits
    ├── Vision encoder targeting
    ├── Alignment gap exploitation
    └── Token embedding collision

Real-World Impact

VLM attacks are not theoretical. Deployed systems face concrete risks:

• Autonomous agents: A VLM-powered web browsing agent could be hijacked by adversarial images embedded in web pages, causing it to navigate to attacker-controlled sites or leak data.
• Document processing: Enterprise systems that use VLMs to process invoices, contracts, or forms are vulnerable to hidden instructions in document images.
• Accessibility tools: VLMs used to describe images for visually impaired users could be tricked into reading out attacker-chosen text.
• Content moderation: VLM-based moderation systems could be bypassed by encoding harmful content in ways the visual pipeline misclassifies.

Section Roadmap

This section covers VLM security in depth:

Related Topics

• Cross-Modal Attack Strategies -- attacks that bridge multiple modalities
• Audio Model Attack Surface -- parallel concepts in the audio domain
• Prompt Injection Fundamentals -- foundational injection techniques that extend to visual channels

References

• "Visual Adversarial Examples Jailbreak Aligned Large Language Models" - Qi et al. (2023) - Foundational work demonstrating adversarial image attacks against VLM safety alignment
• "On the Adversarial Robustness of Multi-Modal Foundation Models" - Schlarmann & Hein (2023) - Comprehensive analysis of adversarial vulnerabilities across VLM architectures
• "(Ab)using Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs" - Bagdasaryan et al. (2023) - Demonstrates image-based prompt injection in multimodal systems
• "FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts" - Gong et al. (2023) - Typographic attacks exploiting VLM text recognition