E-Discovery AI Attacks
Adversarial attacks on AI-powered e-discovery systems: document classification manipulation, privilege prediction bypass, technology-assisted review poisoning, and predictive coding exploitation.
Electronic discovery (e-discovery) is the process by which parties in litigation identify, collect, review, and produce electronically stored information (ESI). Modern e-discovery relies heavily on AI for document classification, privilege review, and predictive coding. These AI systems process millions of documents and make decisions that directly affect litigation outcomes — a document incorrectly classified as non-responsive will never be reviewed by attorneys, and a privileged document incorrectly classified as non-privileged will be produced to opposing counsel.
The adversarial context of e-discovery is explicit: the parties have opposing interests, and one party's documents are reviewed by the other party's AI tools. This creates a natural attack surface where the producing party can structure their documents to exploit weaknesses in the receiving party's AI review tools.
Document Classification Manipulation
How TAR Classification Works
Technology-Assisted Review (TAR) uses machine learning to classify documents as responsive or non-responsive to discovery requests. The two primary approaches are:
TAR 1.0 (Simple Active Learning): A senior attorney reviews a seed set of documents, the model trains on these classifications, and ranks remaining documents by predicted responsiveness. The attorney reviews additional documents to refine the model until the desired recall level is reached.
TAR 2.0 (Continuous Active Learning): The model continuously learns from each document the attorney reviews, re-ranking the remaining documents after each coding decision. This approach adapts throughout the review and typically achieves higher recall with fewer human reviews.
Both approaches are vulnerable to manipulation because the training signal comes from the documents themselves and the human reviewer's interaction with them.
Hiding Responsive Documents
An adversary who anticipates AI-assisted review can structure their documents to evade classification:
# Document evasion techniques for e-discovery AI
evasion_techniques = {
"semantic_camouflage": {
"description": "Express responsive concepts using unusual "
"vocabulary that the TAR model has not learned",
"example": "Instead of 'we need to destroy the evidence', "
"use 'the garden needs weeding before spring' as "
"an established internal euphemism",
"effectiveness": "High against keyword-augmented models, "
"moderate against semantic embedding models",
},
"format_exploitation": {
"description": "Store responsive information in formats "
"that the AI processes poorly",
"example": "Embed key discussions in image attachments, "
"handwritten notes scanned as PDF, or audio "
"transcripts with deliberate errors",
"effectiveness": "High — most TAR systems process text only",
},
"context_dilution": {
"description": "Surround responsive passages with large "
"volumes of non-responsive boilerplate",
"example": "Append email signatures, legal disclaimers, "
"and auto-generated content that overwhelms "
"the responsive signal",
"effectiveness": "Moderate — depends on model architecture",
},
"metadata_manipulation": {
"description": "Alter document metadata to affect how the "
"AI processes and prioritizes the document",
"example": "Change file dates, remove author information, "
"alter subject lines to appear routine",
"effectiveness": "Low to moderate — metadata is one signal "
"among many, but can affect priority ranking",
},
}Testing Classification Robustness
A red team assessment should measure how resilient the TAR system is to adversarial document modification:
def test_tar_robustness(tar_system, test_corpus):
"""
Measure TAR robustness by progressively modifying responsive
documents and measuring classification degradation.
"""
results = {
"baseline_recall": None,
"adversarial_recall": {},
}
# Establish baseline classification
baseline = tar_system.classify(test_corpus.documents)
results["baseline_recall"] = calculate_recall(
predicted=baseline,
ground_truth=test_corpus.labels
)
# Apply escalating evasion techniques
for technique in ["synonym_substitution", "format_change",
"context_dilution", "metadata_alteration",
"combined"]:
modified_docs = apply_evasion(
test_corpus.responsive_documents,
technique=technique
)
adversarial_corpus = test_corpus.replace_responsive(modified_docs)
adversarial_result = tar_system.classify(adversarial_corpus.documents)
results["adversarial_recall"][technique] = calculate_recall(
predicted=adversarial_result,
ground_truth=adversarial_corpus.labels
)
return resultsPrivilege Prediction Exploitation
The Privilege Review Attack Surface
Privilege review is the most sensitive phase of e-discovery. Documents protected by attorney-client privilege or work product doctrine must be identified and withheld from production. AI-assisted privilege review uses trained models to predict which documents are privileged, flagging them for attorney review before production.
The consequences of privilege prediction errors are asymmetric:
- False negative (missed privilege): A privileged document is produced, potentially waiving privilege for the document and related communications. Privilege waiver can be catastrophic.
- False positive (over-flagged privilege): A non-privileged document is withheld and must be produced after dispute, causing delay and potential sanctions.
An adversary can exploit both failure modes.
Causing Privilege Waiver Through Classification Errors
# Testing whether adversarial document characteristics
# cause the privilege model to miss privileged documents
privilege_evasion_tests = [
{
"name": "informal_privilege",
"description": "Privileged communication in casual format",
"document": {
"from": "jsmith@company.com",
"to": "outside_counsel@lawfirm.com",
"subject": "quick question",
"body": "Hey Sarah - can we do the thing we discussed "
"at lunch? The one about the Johnson account? "
"Let me know. -Jim",
# This is a privileged communication seeking legal advice
# but lacks formal privilege markers
},
"expected": "privileged",
"risk_if_missed": "Privilege waiver for legal advice on "
"Johnson account matter",
},
{
"name": "embedded_privilege",
"description": "Privileged content embedded in non-privileged thread",
"document": {
"type": "email_thread",
"messages": [
{"content": "Q3 sales figures attached", "privileged": False},
{"content": "Meeting moved to Thursday", "privileged": False},
{"content": "Counsel advised we should not proceed with "
"the acquisition due to antitrust risk",
"privileged": True},
{"content": "Updated org chart", "privileged": False},
],
},
"expected": "privileged (entire thread due to embedded advice)",
"risk_if_missed": "Production reveals litigation strategy",
},
]Triggering Over-Production
The reverse attack — causing the privilege model to flag non-privileged documents as privileged — can be used to:
- Delay production by generating thousands of false privilege flags that require attorney review
- Create discovery disputes when the producing party withholds non-privileged documents on erroneous privilege grounds
- Exhaust review budgets by forcing expensive attorney-level review of documents that the AI should have cleared
# Crafting documents that trigger false privilege flags
false_privilege_triggers = {
"legal_vocabulary_flooding": {
"technique": "Use legal terminology in business communications",
"example": "We need to 'brief' the team on the 'matter' of "
"Q4 'filings' and ensure 'compliance' with the "
"'counsel' of the marketing advisory 'board'.",
"mechanism": "Legal vocabulary triggers privilege classifiers "
"trained on keyword features",
},
"attorney_name_inclusion": {
"technique": "CC attorneys on routine business communications",
"example": "Adding general counsel to all-hands email "
"distributions so every company-wide email "
"includes an attorney recipient",
"mechanism": "Attorney participation is a strong privilege "
"signal that can be artificially inflated",
},
}Predictive Coding Poisoning
Training Set Manipulation
TAR systems learn from human reviewer decisions. If an adversary can influence the training set — either by manipulating the seed set documents or by compromising the review platform — they can systematically bias the model.
Seed set poisoning: In TAR 1.0, the seed set establishes the model's initial understanding of what constitutes a responsive document. If the seed set is not representative, the model will have blind spots.
# Analyzing seed set representativeness
def assess_seed_set_vulnerability(seed_set, full_corpus):
"""
Identify topics present in the full corpus but
absent from the seed set — these are blind spots
the adversary can exploit.
"""
seed_topics = extract_topics(seed_set)
corpus_topics = extract_topics(full_corpus)
blind_spots = corpus_topics - seed_topics
for topic in blind_spots:
topic_docs = full_corpus.filter(topic=topic)
responsive_rate = topic_docs.responsive_count / len(topic_docs)
if responsive_rate > 0.3:
print(f"CRITICAL BLIND SPOT: Topic '{topic}' has "
f"{responsive_rate:.0%} responsiveness rate but "
f"is absent from seed set. TAR model will likely "
f"miss these documents.")Reviewer Manipulation
In continuous active learning (TAR 2.0), each reviewer decision updates the model. An adversary who compromises a reviewer account or places a biased reviewer can steer the model away from responsive documents:
- Systematic miscoding: Consistently coding responsive documents as non-responsive in specific topic areas, teaching the model to ignore those topics
- Priority manipulation: In systems where reviewers can skip documents, systematically skipping responsive documents so they never enter the training set
- Batch coding attacks: Rapidly coding large batches to overwhelm the slower corrections from accurate reviewers
AI-Powered Search Term Manipulation
Search Term Negotiation Attacks
In many jurisdictions, parties negotiate search terms to identify potentially responsive documents. AI is increasingly used to suggest, evaluate, and refine search terms. An adversary can exploit AI-assisted search term development by:
-
Proposing terms that appear comprehensive but have systematic gaps. AI can identify terms that achieve high precision (most results are responsive) but low recall (many responsive documents are missed).
-
Exploiting semantic search limitations. Modern e-discovery platforms use semantic search in addition to keyword search. Semantic search can be tested for the same adversarial embedding attacks covered in the embedding security section.
-
Manipulating validation sampling. When parties validate search terms by sampling results, the sample may not be representative. An adversary who knows the sampling methodology can structure their documents to perform well on validation samples while evading full-corpus search.
# Testing search term completeness with adversarial documents
def test_search_term_coverage(search_terms, adversarial_docs):
"""
Measure how many adversarial documents evade the
negotiated search terms.
"""
evaded = []
for doc in adversarial_docs:
matched = False
for term in search_terms:
if term_matches(term, doc):
matched = True
break
if not matched and doc.is_responsive:
evaded.append(doc)
evasion_rate = len(evaded) / len(
[d for d in adversarial_docs if d.is_responsive]
)
return {
"evaded_documents": evaded,
"evasion_rate": evasion_rate,
"critical": evasion_rate > 0.1,
}Red Team Assessment Framework
A comprehensive e-discovery AI red team engagement should cover these areas:
Classification robustness testing
Test the TAR system against adversarially modified documents using the evasion techniques described above. Measure the impact on recall at various modification intensities.
Privilege model stress testing
Submit documents with ambiguous privilege characteristics and measure false negative and false positive rates. Test for both privilege waiver risk and over-production risk.
Training set manipulation
In a controlled environment, simulate a compromised reviewer and measure how many malicious coding decisions are needed to meaningfully degrade model accuracy.
Search term adversarial testing
Given the negotiated search terms, craft documents that are responsive but evade all terms. Report the evasion rate and recommend additional terms.
Cross-system consistency
If multiple AI tools are used in the e-discovery workflow, test for inconsistencies between them that an adversary could exploit.
Further Reading
- Legal AI Security Overview — The broader legal AI security landscape
- Legal Research Poisoning — Manipulating AI legal research tools
- Adversarial Embeddings — Underlying adversarial embedding techniques
- Retrieval Manipulation — RAG-level manipulation that applies to e-discovery