Writing Executive Summaries for AI Red Team Reports

# Tone Guide
 
## Too Alarming (avoid)
"Your AI system is fundamentally broken and can be trivially exploited
by any attacker to cause severe harm."
 
## Too Casual (avoid)
"We found a few things you might want to look at when you get a chance."
 
## Appropriate
"The assessment identified three critical vulnerabilities in the AI
system's safety controls that could allow authenticated users to bypass
content restrictions. These findings require prompt remediation given the
system's public-facing deployment and the potential for reputational impact."

# Executive Summary
 
## Engagement Overview
[1 paragraph: Who, what, when, scope]
 
## Overall Risk Rating
[Single rating with visual indicator: Critical / High / Medium / Low]
 
## Key Findings Summary
[3-5 bullet points, one per major finding category, business impact first]
 
## Risk Summary Table
[Table of findings by severity with count]
 
## Recommendation Priorities
[Numbered list of top 3-5 recommendations, highest impact first]
 
## Positive Observations
[2-3 things the system does well]

## Engagement Overview
 
[Red Team Organization] conducted an AI security assessment of [Client]'s
[System Name] between [Start Date] and [End Date]. The assessment evaluated
the security posture of the customer-facing AI chatbot, including its
resistance to prompt injection attacks, the effectiveness of content safety
controls, data isolation in the multi-tenant environment, and the security
of integrated function calling capabilities.
 
The assessment used a combination of manual expert testing and automated
vulnerability scanning across [N] test categories, generating [N] total
test interactions with the AI system. Testing was conducted against the
[production/staging] environment using [authenticated/unauthenticated]
access consistent with a [regular user/malicious insider] threat model.

## Overall Risk Rating: HIGH
 
The AI system's current security posture presents HIGH risk due to
the identification of [N] critical and [N] high-severity vulnerabilities
in safety-critical controls. The most significant findings demonstrate
that an authenticated user can bypass content safety restrictions and
extract confidential system configuration data. These findings are
reproducible, require no special tools or knowledge, and could be
exploited by any user of the system.

## Key Findings
 
1. **AI safety controls can be bypassed by any authenticated user.**
   Content restrictions designed to prevent the system from generating
   harmful instructions were bypassed using publicly known techniques.
   An attacker could use the chatbot to generate content that violates
   company policy and applicable regulations. *(Critical)*
 
2. **Confidential system configuration is exposed to users.**
   The AI system's internal instructions, which contain business logic
   and API integration details, can be extracted by asking the system
   to reveal its configuration. This information could enable more
   sophisticated attacks. *(High)*
 
3. **Customer data isolation has gaps in the knowledge base.**
   Under specific conditions, the AI system retrieved and presented
   documents belonging to other customer tenants. This represents a
   data breach scenario for affected customers. *(Critical)*
 
4. **Automated tools can be triggered to perform unintended actions.**
   The AI system's ability to call external tools (email, database
   queries) can be manipulated to perform actions not intended by the
   user, such as sending emails with attacker-controlled content. *(High)*
 
5. **Input rate limiting is insufficient for automated abuse.**
   The current rate limiting allows an attacker to send requests at
   a volume sufficient for automated exploitation, credential testing,
   or content generation abuse. *(Medium)*

## Finding Summary
 
| Severity | Count | Remediation Timeline |
|----------|-------|---------------------|
| Critical | 2 | Immediate (within 1 week) |
| High | 3 | Urgent (within 2 weeks) |
| Medium | 4 | Standard (within 30 days) |
| Low | 2 | Planned (within 90 days) |
| Informational | 3 | At discretion |
| **Total** | **14** | |

## Priority Recommendations
 
1. **Strengthen AI safety controls with defense-in-depth.**
   The current single-layer content filtering approach is insufficient.
   Implement multiple independent safety layers (input filtering, model
   safety training, output filtering) so that bypassing one layer does
   not compromise the entire safety posture.
   *Timeline: 1-2 weeks for initial hardening, ongoing for full implementation.*
 
2. **Separate confidential configuration from model-accessible context.**
   Move API keys, business logic, and sensitive instructions out of the
   system prompt into application-layer controls that the model cannot
   access or reveal. This eliminates the system prompt extraction risk
   entirely.
   *Timeline: 1 week.*
 
3. **Implement strict tenant isolation for knowledge base queries.**
   Add server-side tenant filtering at the database query level, not
   at the application level, to prevent cross-tenant data access
   regardless of how the query is constructed.
   *Timeline: 1-2 weeks.*
 
4. **Add parameter validation for all AI-triggered tool calls.**
   Implement strict allow-listing and validation for all parameters
   passed to external tools, independent of the model's own judgment.
   The model should not be the sole gatekeeper for tool call safety.
   *Timeline: 2 weeks.*
 
5. **Implement AI-specific monitoring and alerting.**
   Deploy monitoring that detects prompt injection attempts, unusual
   query patterns, and content policy violations in real time. Current
   monitoring does not distinguish adversarial interactions from normal
   usage.
   *Timeline: 2-4 weeks.*

## Positive Observations
 
- **Authentication and session management** are well-implemented.
  The assessment did not identify any vulnerabilities in user authentication,
  and session isolation between users is properly enforced.
 
- **API rate limiting** is present and correctly configured for normal
  usage patterns. While insufficient for targeted automated attacks,
  it provides protection against casual abuse.
 
- **Logging and audit trails** capture all AI interactions with sufficient
  detail for forensic analysis. This is a strong foundation for building
  the recommended AI-specific monitoring.

# Weak (subjective)
"Many prompt injection techniques were successful."
 
# Strong (quantified)
"Of 47 prompt injection techniques tested, 12 (25%) successfully
bypassed content safety controls. Successful techniques required
no special tools and could be attempted by any user."
 
# Weak
"The system prompt was extracted."
 
# Strong
"The system prompt (approximately 2,400 tokens of confidential
instructions including two API keys) was fully extracted using
a technique that requires a single user message."

# Executive Summary QA Checklist
 
## Content
- [ ] Engagement overview accurately describes scope and methodology
- [ ] Overall risk rating is justified and consistent with findings
- [ ] Key findings lead with business impact, not technical details
- [ ] All Critical and High findings are represented in the summary
- [ ] Recommendations are actionable and prioritized
- [ ] Positive observations are included
 
## Language
- [ ] No unexplained jargon or acronyms
- [ ] Severity language is calibrated (not too alarming, not too casual)
- [ ] Findings are quantified where possible
- [ ] Tone is professional and objective throughout
 
## Accuracy
- [ ] Severity counts match the full findings section
- [ ] Recommendation timelines are realistic
- [ ] No claims that are not supported by evidence
- [ ] Client name and system details are correct
 
## Format
- [ ] Length is appropriate for the audience (1-3 pages)
- [ ] Scannable structure with clear headings
- [ ] Risk summary table is present and correct
- [ ] No orphaned references to sections that do not exist

# AI Security Assessment Results
## [System Name] | [Date]
 
**Risk Rating: HIGH**
 
### Critical Findings
- Safety controls bypassable by any user (2 critical)
- Customer data isolation gaps (1 critical)
 
### Immediate Actions Required
1. Strengthen AI safety controls (1-2 weeks)
2. Fix tenant isolation in knowledge base (1-2 weeks)
3. Remove sensitive data from AI configuration (1 week)
 
### Positive: Auth, logging, and basic rate limiting are solid
 
---
14 total findings | 2 Critical | 3 High | 4 Medium | 2 Low | 3 Info

# Verbal Briefing Outline (5 minutes)
 
## Opening (30 seconds)
"We assessed your AI chatbot's security over two weeks using both
expert manual testing and automated scanning."
 
## Overall Picture (1 minute)
"Overall risk is HIGH. We found 14 vulnerabilities including two
critical issues that need immediate attention."
 
## Top 3 Findings (2 minutes)
"First, and most important: [Finding 1 in business terms]...
Second: [Finding 2 in business terms]...
Third: [Finding 3 in business terms]..."
 
## What to Do (1 minute)
"Three actions will address 80% of the risk: [Rec 1], [Rec 2], [Rec 3].
We estimate these can be completed in 2-3 weeks."
 
## Close (30 seconds)
"The detailed report has all findings with reproduction steps and
remediation guidance. We recommend a retest in [timeframe] after
remediation is complete."