Purple Teaming for AI

Collaborative attack-defense exercises for AI systems: structuring purple team engagements, real-time knowledge transfer, joint attack simulation, and measuring defensive improvement through iterative testing.

Traditional red teaming is adversarial: the red team attacks, the blue team defends, and findings are shared after the engagement ends. Purple teaming collapses this timeline. Attack and defense happen simultaneously, with real-time knowledge transfer that accelerates both offensive discovery and defensive improvement.

For AI systems, purple teaming is particularly valuable because AI defenses are often tunable in real time: system prompts can be rewritten, guardrail classifiers can be retrained, and output filters can be updated without redeploying the underlying model. This enables a rapid iterate-and-test cycle that traditional infrastructure security cannot match.

Why Purple Teaming for AI

The Speed Advantage

AI defenses can often be modified and tested within minutes:

This rapid turnaround makes real-time collaborative testing practical. The red team demonstrates an attack, the blue team implements a defense, and the red team immediately tests whether the defense holds.

The Knowledge Transfer Advantage

In traditional red teaming, findings are delivered in a report weeks after discovery. By that time, context is lost, the system may have changed, and the defending team must reconstruct the attack scenario from documentation. Purple teaming eliminates this gap:

• Defenders see attacks in real time and understand the attacker's methodology
• Attackers see defenses deployed and can immediately assess their effectiveness
• Both teams develop intuition for the attack-defense dynamics specific to their system
• Novel defensive approaches can be tested immediately rather than theorized about

Exercise Structure

Format Options

Sprint Session Structure

A typical 3-hour sprint session:

Hour 1: Baseline Assessment (Red Team Leads)

1. Red team demonstrates 3-5 attack techniques against the current system
2. Blue team observes and takes notes on which attacks succeed and why
3. Both teams discuss the attack surface and prioritize which bypasses to address first

Hour 2: Defense Implementation (Blue Team Leads)

1. Blue team implements defenses for the highest-priority bypasses
2. Red team provides input on whether proposed defenses would actually block the attack or just force a minor variation
3. Defenses are deployed to the test environment

Hour 3: Validation and Iteration (Collaborative)

1. Red team re-tests with the defenses in place
2. For each defense: Does the original attack still work? Can simple variations bypass it?
3. Blue team adjusts defenses based on validation results
4. Final round of testing to confirm improvements
5. Both teams document findings, defensive improvements, and remaining gaps

Day Exercise Structure

A full-day exercise adds depth:

Morning: Reconnaissance and Baseline

• Red team performs reconnaissance and maps the current attack surface
• Blue team briefs on existing defenses, known limitations, and recent changes
• Joint threat modeling session to prioritize test areas

Midday: Attack-Defense Rounds

• Three iterative rounds of attack, defense implementation, and validation
• Each round focuses on a different attack category (e.g., round 1: direct injection, round 2: tool abuse, round 3: data exfiltration)
• Track metrics: initial bypass rate, post-defense bypass rate, defense implementation time

Afternoon: Advanced Scenarios and Wrap-Up

• Red team attempts combined/chained attacks using techniques from all categories
• Blue team evaluates whether defense interactions create new vulnerabilities
• Joint debrief documenting improvements, remaining gaps, and action items

AI-Specific Exercise Scenarios

Scenario 1: System Prompt Hardening

Objective: Iteratively harden a system prompt against injection attacks.

Round 1: Red team tests the baseline system prompt with five injection techniques. Record bypass rates.

Round 2: Blue team rewrites the system prompt with anti-override language, role confirmation, and instruction repetition. Red team re-tests.

Round 3: Red team adapts techniques (longer payloads, format mimicry, multi-turn). Blue team adds input classification. Re-test.

Round 4: Red team combines encoding with injection to evade the classifier. Blue team adds input normalization. Re-test.

Metric: Track bypass rate degradation across rounds. A successful exercise shows declining bypass rates with each iteration.

Scenario 2: Guardrail Bypass and Hardening

Objective: Test and improve input/output guardrail classifiers.

Round 1: Red team identifies guardrail bypass techniques
         (encoding, paraphrasing, indirect reference)
  ↓
Round 2: Blue team retrains classifier with red team's bypass
         examples as training data
  ↓
Round 3: Red team generates novel bypass variants not covered
         by the retraining
  ↓
Round 4: Blue team adds structural detection (not just content)
         to catch novel variants
  ↓
Repeat until bypass rate stabilizes at acceptable level

Scenario 3: Tool Authorization Testing

Objective: Validate that tool authorization controls prevent injection-driven abuse.

• Red team attempts to invoke unauthorized tools through prompt injection
• Blue team implements and tunes tool authorization rules
• Both teams map the complete set of tool actions available through injection
• Iterate until all unauthorized tool access paths are blocked or require human approval

Scenario 4: Multi-Agent Trust Boundary Exercise

Objective: Test trust boundaries in multi-agent architectures.

• Red team attempts to escalate privileges through agent handoffs
• Blue team implements context sanitization between agents
• Test whether sanitization blocks legitimate information transfer as well as attacks
• Iterate to find the balance between security and functionality

Measuring Improvement

Quantitative Metrics

Track these metrics across purple team exercises to measure progress:

Qualitative Metrics

• Knowledge transfer score: Can the blue team explain why each attack works, not just that it does?
• Defense creativity: Is the blue team developing novel defenses or just patching specific attack patterns?
• Attack diversity: Is the red team expanding technique categories or repeating known approaches?
• Collaboration quality: Are both teams sharing insights productively or operating in parallel?

Tracking Progress Over Time

Exercise 1 (January):
  Direct injection baseline:    85% bypass rate
  Post-exercise:                40% bypass rate
  Improvement:                  45 percentage points

Exercise 2 (February):
  Direct injection:             35% bypass rate (held from E1)
  Multi-turn injection:         70% bypass rate (new category)
  Post-exercise multi-turn:     30% bypass rate
  Improvement:                  40 percentage points

Exercise 3 (March):
  Direct injection:             30% bypass rate (continued improvement)
  Multi-turn injection:         25% bypass rate (held from E2)
  Encoding bypass:              60% bypass rate (new category)
  Post-exercise encoding:       20% bypass rate
  Overall trend:                Decreasing baseline bypass rates
                                across all categories

Team Composition and Roles

Purple Team Roles

Communication Protocols

Effective purple teaming requires structured communication:

Real-time narration. The red team narrates their approach: "I'm going to try a delimiter escape using XML tag closure. The hypothesis is that the system prompt uses XML tags to wrap user input."

Defense rationale. The blue team explains their defense logic: "I'm adding input sanitization that strips closing XML tags from user input. This should prevent the tag closure technique but may break legitimate XML content."

Honest assessment. Both teams must be candid. If a defense feels brittle, say so. If an attack relies on an unrealistic precondition, acknowledge it.

Common Anti-Patterns

Patch-specific defense. The blue team fixes only the exact payload the red team demonstrated. The fix should generalize to the attack category, not just the specific payload.

Red team tunnel vision. The red team keeps trying variations of one technique instead of switching to different attack categories. Breadth matters as much as depth.

Skipping the "why" discussion. Implementing a defense without discussing why the attack worked. Understanding the root cause produces more robust defenses.

No baseline measurement. Starting the exercise without measuring the initial bypass rate. Without a baseline, improvement cannot be quantified.

Over-filtering. Implementing such aggressive defenses that legitimate user requests are blocked. Always measure false positive rate alongside bypass rate.

Try It Yourself

Related Topics

• Red Team Methodology - The overall engagement framework
• Continuous Red Teaming - Purple teaming principles applied to ongoing programs
• Evidence Collection - Capturing findings during exercises
• Guardrails & Filtering - Defenses tested and improved in purple exercises

References

• Oakley, C. (2019). "Purple Team Field Manual" - Foundational purple team methodology
• MITRE (2024). ATLAS - Adversarial Threat Landscape for AI Systems
• Anthropic (2024). "Challenges in Red Teaming AI Systems"
• Microsoft (2024). "Planning Red Teaming for Large Language Models and Their Applications"
• OWASP (2025). OWASP Top 10 for LLM Applications