Scoping & Rules of Engagement

Defining scope, rules of engagement, authorization boundaries, and success criteria for AI red team engagements, with templates and checklists for common engagement types.

Scoping and rules of engagement (ROE) define what an AI red team is authorized to test, how they may test it, and what constraints apply. Poor scoping leads to either incomplete testing (too narrow) or operational incidents (too broad). For AI systems, scoping must address concerns that do not exist in traditional penetration testing: the stochastic nature of model behavior, the risk of generating genuinely harmful content during testing, rate limits on inference APIs, and the distinction between model-level and application-level testing.

Scope Definition

Scope Components

An AI red team scope must specify each of the following:

Scope Levels

AI red team engagements typically fall into one of three scope levels:

Level 1: Model Assessment. Testing the model's safety alignment and behavioral boundaries. Focuses on prompt injection, jailbreaking, and harmful content generation. Does not test application-layer controls.

Level 2: Application Assessment. Testing the complete application including the model, system prompt, input/output filters, tool integrations, and user interface. This is the most common engagement type.

Level 3: Full-Stack Assessment. Testing the entire AI system stack from infrastructure to model to application. Includes API security, authentication, infrastructure configuration, training pipeline, and deployment security in addition to model-level testing.

Rules of Engagement

AI-Specific ROE Considerations

Beyond standard penetration testing ROE, AI red teaming requires additional rules:

Rate limiting and cost controls. API inference calls cost money. Define:

• Maximum API calls per hour/day
• Maximum tokens consumed per engagement
• Who pays for inference costs (client or testing firm)
• Whether throttling or batching is required

Harmful content handling. Testing for harmful content generation creates a dilemma: the red team must demonstrate the harm but also handle the generated content responsibly. Define:

• How harmful outputs are captured and stored
• Who has access to evidence containing harmful content
• Retention and destruction schedules for harmful content
• Whether harmful content can be included in reports (redacted or full)
• Specific content categories that should not be generated even during testing

Model interaction boundaries. Define:

• Whether the team may attempt to modify model behavior (fine-tuning, RLHF manipulation)
• Whether adversarial training data or poisoning attempts are in scope
• Whether weight extraction or model stealing attempts are authorized
• Maximum conversation length and number of concurrent sessions

Production vs. staging. Define:

• Whether testing occurs on production or staging systems
• Whether real users may be affected during testing
• Time windows for testing (business hours, off-hours)
• Rollback procedures if testing causes unexpected behavior changes

Authorization Chain

AI red team authorization requires sign-off from additional stakeholders compared to traditional pen testing:

Escalation Procedures

Define clear escalation triggers:

Success Criteria

Defining "Success" for AI Red Teaming

Unlike traditional pen testing where success is often binary (did you compromise the target?), AI red teaming operates on a spectrum. Define success criteria along multiple dimensions:

Coverage metrics:

• Percentage of OWASP LLM Top 10 categories tested
• Number of attack vectors tested per scope component
• Number of unique techniques attempted per vector

Finding metrics:

• Number and severity of findings
• Bypass rates for each technique category
• Percentage of safety boundaries successfully breached

Quality metrics:

• All findings reproducible with documented evidence
• Remediation recommendations provided for each finding
• Severity ratings calibrated to business impact

Severity Framework

Agree on severity classification before testing begins:

Engagement Documentation

Test Plan Template

A test plan should include:

1. Engagement overview: Target system, scope level, timeline, team composition
2. Scope details: In-scope components, out-of-scope exclusions, authorized attack vectors
3. Rules of engagement: Rate limits, harmful content handling, escalation procedures
4. Test matrix: Specific test cases organized by attack vector and target component
5. Success criteria: Coverage, finding, and quality metrics
6. Evidence requirements: What must be captured for each finding
7. Reporting deliverables: Report format, audience, delivery timeline

Pre-Engagement Checklist

Before testing begins, verify:

• Authorization document signed by all required stakeholders
• API provider terms of service reviewed and compliant
• Test accounts and credentials provisioned
• Staging environment available (if applicable)
• Rate limits and cost caps configured
• Evidence storage with appropriate access controls provisioned
• Escalation contacts verified and reachable
• Communication channel established with system owner
• Harmful content handling procedures documented and acknowledged
• Team briefed on scope boundaries and prohibited actions

Try It Yourself

Related Topics

• Red Team Methodology - The overall engagement lifecycle
• AI-Specific Threat Modeling - Threat models that inform scope
• Evidence Collection - Standards for capturing and preserving findings
• Continuous Red Teaming - How scoping adapts for ongoing programs

References

• NIST (2024). AI Risk Management Framework (AI RMF 1.0)
• OWASP (2025). OWASP Top 10 for LLM Applications
• PTES (2024). Penetration Testing Execution Standard - Pre-Engagement
• CREST (2024). CREST Penetration Testing Guide