SOC 2 for AI Systems
SOC 2 trust services criteria applied to AI systems, AI-specific controls, audit considerations, and how red teaming supports SOC 2 compliance for AI-powered services.
SOC 2 (System and Organization Controls 2) is the dominant audit framework for service organizations in the United States. As organizations increasingly embed AI into their service offerings, SOC 2 audits must evolve to address AI-specific risks. Red teamers who 理解 SOC 2 requirements can deliver findings that directly support audit evidence and help clients maintain their SOC 2 reports.
Trust Services Criteria Applied to AI
SOC 2 is built on five Trust Services Criteria (TSC). Each has specific implications for AI systems:
安全 (Common Criteria)
The 安全 category is always included in SOC 2 reports and forms the foundation for AI system controls:
| Criteria | Traditional Application | AI-Specific Extension |
|---|---|---|
| CC6.1 (Logical access) | User 認證 and 授權 | Model API access controls, 推論 endpoint 認證, prompt-level access restrictions |
| CC6.3 (Access removal) | Deprovisioning user accounts | Revoking API keys, removing model access, disabling fine-tuned model variants |
| CC6.6 (Boundary protection) | Network segmentation, firewalls | Model isolation, 提示詞注入 filtering, 輸入/輸出 boundary controls |
| CC6.7 (Data transmission) | Encryption in transit | Protecting prompts and completions in transit, securing model-to-model communication |
| CC6.8 (Malicious software) | Antivirus, endpoint protection | 對抗性 輸入 偵測, malicious prompt filtering, model integrity verification |
| CC7.2 (監控) | 安全 event 監控 | 監控 for 對抗性 attacks, unusual query patterns, data extraction attempts |
| CC7.3 (Anomaly 偵測) | Intrusion 偵測 systems | AI behavioral anomaly 偵測, 提示詞注入 偵測, 輸出 drift 監控 |
Availability
| Criteria | Traditional Application | AI-Specific Extension |
|---|---|---|
| A1.1 (Capacity management) | Server scaling, bandwidth planning | GPU capacity management, 推論 queue management, model serving scalability |
| A1.2 (Recovery procedures) | Backup and disaster recovery | Model rollback procedures, 訓練 checkpoint recovery, 推論 fallback paths |
Processing Integrity
Processing integrity is particularly relevant for AI systems, as their outputs directly affect business decisions:
| Criteria | Traditional Application | AI-Specific Extension |
|---|---|---|
| PI1.1 (Accurate processing) | Data validation, calculation verification | Model accuracy 監控, hallucination 偵測, 輸出 validation |
| PI1.2 (Complete processing) | Transaction completeness | Ensuring AI processes all inputs without silent failures or truncation |
| PI1.3 (Timely processing) | SLA compliance | Inference latency 監控, timeout handling for AI operations |
| PI1.4 (Authorized processing) | Approval workflows | Human-in-the-loop requirements for high-stakes AI decisions |
| PI1.5 (Error handling) | Exception processing | Graceful degradation when models fail, fallback behavior documentation |
Confidentiality
| Criteria | Traditional Application | AI-Specific Extension |
|---|---|---|
| C1.1 (Confidential data identification) | Data classification | 訓練資料 classification, prompt content classification, model weight protection |
| C1.2 (Confidential data disposal) | Secure deletion | Model unlearning, 訓練資料 removal, conversation data purging |
Privacy
| Criteria | Traditional Application | AI-Specific Extension |
|---|---|---|
| P1-P8 (Privacy criteria) | PII handling, consent, access | AI 訓練資料 privacy, prompt data handling, model memorization risks, user data in 微調 |
AI-Specific Control Objectives
Beyond mapping existing TSC to AI, organizations should 實作 additional AI-specific controls. These extend the SOC 2 framework to address risks unique to AI systems:
Model Governance Controls
| Control ID | Objective | Description | 紅隊 測試 |
|---|---|---|---|
| AI-GOV-01 | Model inventory | Maintain a complete inventory of all AI models in production | Verify completeness by discovering undocumented models |
| AI-GOV-02 | Model lifecycle management | Track models from development through retirement | Attempt to access deprecated or staging models |
| AI-GOV-03 | Model change control | Approve and document model changes before deployment | 測試 whether unapproved model versions can be deployed |
| AI-GOV-04 | Third-party model risk | 評估 and monitor risks from third-party AI providers | 測試 third-party model behavior, verify SLA compliance |
Model 安全 Controls
| Control ID | Objective | Description | 紅隊 測試 |
|---|---|---|---|
| AI-SEC-01 | Prompt injection prevention | Prevent unauthorized actions through prompt manipulation | Execute 提示詞注入 attack scenarios |
| AI-SEC-02 | 輸出 filtering | Prevent sensitive data in model outputs | Attempt data extraction through various 輸出 channels |
| AI-SEC-03 | Model access control | Restrict model capabilities based on user 授權 | 測試 privilege escalation through prompt manipulation |
| AI-SEC-04 | 對抗性 robustness | Maintain model behavior under 對抗性 conditions | 對抗性 測試 across 輸入 modalities |
Data Handling Controls
| Control ID | Objective | Description | 紅隊 測試 |
|---|---|---|---|
| AI-DATA-01 | 訓練資料 governance | Control what data is used for 訓練 and 微調 | Verify 訓練資料 provenance and 授權 |
| AI-DATA-02 | Prompt data isolation | Prevent cross-user data leakage through prompts | 測試 for conversation leakage between sessions |
| AI-DATA-03 | Data retention for AI | Define and enforce retention periods for AI interaction data | Verify that expired data is actually deleted |
| AI-DATA-04 | RAG data integrity | Ensure 檢索增強生成 uses authorized data | Attempt RAG 投毒 and unauthorized data injection |
Audit Considerations
SOC 2 Type I vs Type II for AI Systems
| Dimension | Type I | Type II |
|---|---|---|
| Scope | Design of controls at a point in time | Design and operating effectiveness over a period (typically 6-12 months) |
| AI relevance | Useful for initial AI system launches | Required to demonstrate sustained AI control effectiveness |
| Red team role | Point-in-time 評估 of AI controls | Periodic assessments throughout the audit period |
| Evidence needed | Control documentation and design review | 測試 results, 監控 logs, incident records over the period |
What Auditors Look For in AI Systems
Common auditor questions about AI controls:
| Question | What They Are Assessing | How 紅隊演練 Helps |
|---|---|---|
| "How do you prevent 提示詞注入?" | CC6.6 boundary protection for AI | Demonstrate whether 提示詞注入 controls actually work |
| "How do you monitor AI system behavior?" | CC7.2, CC7.3 監控 and anomaly 偵測 | Show whether 監控 detects 對抗性 activity |
| "How do you prevent data leakage through AI?" | C1.1 confidential data, P3 data collection | 測試 data extraction via model outputs |
| "How do you manage model changes?" | CC8.1 change management | Verify that model change procedures are followed |
| "How do you handle AI errors?" | PI1.5 error handling | 測試 failure modes and verify graceful degradation |
Evidence Collection for Auditors
Red team engagements supporting SOC 2 should produce evidence formatted for audit consumption:
| Evidence Type | Content | SOC 2 Relevance |
|---|---|---|
| 測試 plans | Scope, methodology, tools used, controls tested | Demonstrates systematic 評估 approach |
| 測試 results | Detailed findings with steps to reproduce | Proves control operating effectiveness (or failure) |
| Remediation verification | Re-測試 results after control improvements | Shows corrective action effectiveness |
| Continuous 監控 data | Automated 測試 results over the audit period | Supports Type II operating effectiveness |
| Exception logs | Documented control failures and responses | Shows management awareness and response capability |
紅隊 Engagement Structure for SOC 2
Pre-Engagement Alignment
Before conducting a 紅隊 engagement supporting SOC 2, align with the audit team:
識別 in-scope AI systems
Work with the client to 識別 which AI systems are included in their SOC 2 scope boundary. Only AI systems within the service organization's trust boundary require 測試.
Map controls to 測試 activities
Review the client's control matrix and map AI-specific controls to 紅隊 測試 scenarios. Each control should have at least one corresponding 測試.
Coordinate timing with auditors
For Type II reports, schedule 測試 at intervals throughout the audit period rather than all at once. This provides evidence of sustained control effectiveness.
Agree on evidence format
Confirm with the CPA firm what evidence format they require. Some auditors accept 紅隊 reports directly; others need findings formatted into their 測試 workpapers.
測試 Methodology by Trust Service Category
安全 測試 (CC criteria):
- Prompt injection attacks against all AI endpoints
- API 認證 and 授權 測試
- Data extraction attempts through model outputs
- 系統提示詞 extraction and abuse
- 對抗性 輸入 測試
Availability 測試 (A criteria):
- Model denial-of-service through resource exhaustion
- Inference pipeline stress 測試
- Failover and fallback behavior verification
- Recovery time objective validation
Processing integrity 測試 (PI criteria):
- Hallucination rate measurement under 對抗性 conditions
- 輸出 manipulation through carefully crafted inputs
- Verification that human oversight controls function correctly
- Error handling behavior under unexpected inputs
Confidentiality 測試 (C criteria):
- 訓練資料 extraction attempts
- Cross-tenant data leakage 測試
- Model weight and configuration extraction
- Conversation history isolation verification
Privacy 測試 (P criteria):
- PII extraction from model outputs
- Consent mechanism verification for AI data collection
- Data retention and deletion verification
- User data isolation in multi-tenant environments
Common Findings and Remediation
Findings That Affect SOC 2 Reports
| Finding | SOC 2 Impact | Severity for Auditors |
|---|---|---|
| Successful 提示詞注入 bypassing controls | CC6.6 control failure | High -- may result in qualified opinion |
| Data extraction through model outputs | C1.1, P3 control failure | High -- confidentiality and privacy impact |
| No 監控 for 對抗性 inputs | CC7.2, CC7.3 gap | Medium -- 偵測 deficiency |
| Model changes deployed without approval | CC8.1 control failure | Medium -- change management gap |
| No fallback behavior when model fails | PI1.5, A1.2 gap | Medium -- processing integrity and availability |
| Undocumented AI models in production | AI-GOV-01 gap | Low to Medium -- inventory completeness |
Remediation Priorities
Immediate priorities (fix before audit period ends):
- Any control failure that allows data extraction or unauthorized access
- Missing 監控 for AI-specific attack patterns
- Uncontrolled model deployment processes
Longer-term improvements:
- Automated 對抗性 測試 in CI/CD pipelines
- Enhanced AI behavioral 監控 dashboards
- Formal AI incident response procedures
- Regular model 安全 reviews integrated into change management
Integration with Other Frameworks
Organizations often maintain SOC 2 alongside other compliance frameworks. Red team findings should be mapped across all applicable frameworks:
| SOC 2 Criteria | ISO 42001 Control | NIST AI RMF | EU AI Act |
|---|---|---|---|
| CC6.6 (Boundary protection) | A.6.2.5 (Deployment) | MS-2.3 | Art. 15 (Cybersecurity) |
| CC7.2 (監控) | A.6.2.6 (監控) | MG-2.4 | Art. 9 (Risk management) |
| PI1.1 (Accurate processing) | A.6.2.4 (Verification) | MS-2.6 | Art. 15 (Accuracy) |
| C1.1 (Confidentiality) | A.7.4 (Data provenance) | MP-4.2 | Art. 10 (Data governance) |
| P3 (Collection) | A.10.2 (Fairness) | GV-6.1 | Art. 13 (Transparency) |
This cross-mapping allows a single 紅隊 engagement to produce findings relevant to multiple compliance requirements, maximizing the return on investment for clients managing complex compliance landscapes.