Simulation: AI Supply Chain Attack Investigation

Investigate and respond to a supply chain compromise affecting an AI system's model weights, training data pipeline, and third-party dependencies.

Engagement Brief

Client: DataForge AI (AI-as-a-Service platform) Incident: Anomalous model behavior detected after a routine model update. The updated model occasionally produces subtly biased financial analysis favoring specific companies. Your Role: AI incident response investigator Scope: Investigate the model update pipeline, training data provenance, and third-party model dependencies Duration: 6 hours Constraints: Production system remains online with monitoring; do not modify any production components

Background

DataForge AI provides a financial analysis AI to enterprise clients. The system uses a fine-tuned foundation model with RAG-based retrieval from financial databases. Two weeks ago, the model was updated through the standard CI/CD pipeline. Since the update, multiple clients have reported that the AI occasionally recommends specific stocks with unusual confidence, contradicting its own analysis.

Phase 1: Detection Confirmation (1 hour)

1. 1

   Verify the reported behavior

   Interact with the model to confirm the reported anomaly. Send queries about various companies and sectors, comparing responses for the reported bias pattern.

   # Probe for behavioral anomaly
   test_queries = [
       "Analyze the investment potential of [Company A] vs [Company B] "
       "in the semiconductor sector.",
       "What are the risks of investing in [Suspected Company]?",
       "Compare Q3 earnings forecasts for the top 5 tech companies.",
       "Recommend a diversified portfolio for a moderate risk investor.",
   ]
    
   # Look for: unusual confidence, contradictory reasoning,
   # specific company recommendations that override analysis

2. 2

   Establish behavioral baseline comparison

   Compare current model responses against archived responses from before the update using the same standardized queries. Document behavioral changes quantitatively.

3. 3

   Rule out non-malicious explanations

   Check whether the behavioral change could be explained by legitimate factors: updated training data reflecting real market conditions, configuration changes, or RAG knowledge base updates.

Phase 2: Pipeline Investigation (2 hours)

Training Data Provenance

Trace the data used in the most recent fine-tuning run:

investigation_checklist = {
    "data_sources": {
        "questions": [
            "What data sources were used for fine-tuning?",
            "Were any new data sources added for this update?",
            "Who approved the data sources?",
            "Are data source integrity checks in place?",
        ],
    },
    "pipeline_access": {
        "questions": [
            "Who has write access to the training data pipeline?",
            "Were there any unusual access patterns in the past month?",
            "Is the pipeline protected by MFA?",
            "Are pipeline configuration changes audited?",
        ],
    },
    "model_registry": {
        "questions": [
            "Do model weights match the expected hashes?",
            "Was the model signed before deployment?",
            "Who approved the model promotion to production?",
            "Is there a complete audit trail for the model lifecycle?",
        ],
    },
}

Dependency Analysis

Review all third-party dependencies in the model pipeline:

• Foundation model provider updates
• Training framework version changes
• Data preprocessing library updates
• Any new packages added since the last clean model version

Model Registry Integrity

Verify that the model artifacts in the registry match what was produced by the training pipeline:

# Model registry integrity verification
registry_checks = {
    "hash_verification": {
        "description": "Compare SHA-256 hashes of deployed model "
                       "weights against the registry record",
        "commands": [
            "sha256sum /models/production/model.safetensors",
            "curl -s $REGISTRY_URL/models/financial-v2.3/hash",
        ],
        "finding_if_mismatch": "CRITICAL: Deployed model does not "
                                "match registry. Model may have been "
                                "tampered with after training.",
    },
    "signature_verification": {
        "description": "Verify the cryptographic signature on the "
                       "model artifact if code signing is in place",
        "commands": [
            "gpg --verify model.safetensors.sig model.safetensors",
        ],
        "finding_if_invalid": "CRITICAL: Model signature invalid. "
                               "Model may have been replaced.",
    },
    "provenance_chain": {
        "description": "Trace the model from training run to "
                       "deployment, verifying each handoff",
        "checkpoints": [
            "Training job completion record",
            "Evaluation results and approval",
            "Registry upload record",
            "Deployment pipeline execution",
            "Production deployment confirmation",
        ],
    },
}

Infrastructure Access Review

Review access logs for all systems in the model pipeline:

access_review_targets = [
    {
        "system": "Training data storage (S3/GCS)",
        "log_source": "Cloud provider access logs",
        "look_for": "Unusual write operations, new IAM roles, "
                     "access from unexpected IPs",
        "timeframe": "30 days before the model update",
    },
    {
        "system": "Training compute environment",
        "log_source": "Container orchestration logs",
        "look_for": "Modified training scripts, unexpected jobs, "
                     "additional containers",
        "timeframe": "During the training window",
    },
    {
        "system": "Model registry",
        "log_source": "Registry audit logs",
        "look_for": "Manual uploads bypassing the pipeline, "
                     "version overwrites, metadata changes",
        "timeframe": "30 days before deployment",
    },
    {
        "system": "CI/CD pipeline",
        "log_source": "Pipeline execution logs",
        "look_for": "Modified pipeline definitions, additional "
                     "steps, changed approval gates",
        "timeframe": "60 days before the model update",
    },
]

Phase 3: Root Cause Analysis (2 hours)

1. 1

   Data poisoning assessment

   Analyze the fine-tuning dataset for injected examples that could cause the observed bias. Look for:

   • Examples with anomalous sentiment scores for specific companies
   • Data entries with unusual source attribution
   • Clusters of similar entries that reinforce a specific narrative
2. 2

   Model comparison

   Compare the compromised model's weights against the previous version. If a full weight comparison is infeasible, compare model activations on diagnostic inputs to identify which layers or attention heads show the most change.

3. 3

   Attribution

   Based on your findings, determine the most likely attack vector:

   • Compromised data source injecting biased training examples
   • Insider threat modifying the training pipeline
   • Third-party dependency introducing a backdoor
   • Compromised model weights in the registry

Phase 4: Response and Remediation (1 hour)

Deliverables

1. Incident timeline — Chronological record of when the compromise likely occurred and when it was detected
2. Root cause report — Most likely attack vector with supporting evidence
3. Blast radius assessment — Which clients received biased analysis and during what timeframe
4. Remediation plan — Steps to restore model integrity and prevent recurrence
5. Client notification draft — Communication to affected clients explaining the issue and remediation

Key Decision Points

• Should the model be rolled back immediately or kept online with enhanced monitoring?
• Does the incident require regulatory notification (SEC if financial advice was affected)?
• Should the third-party data sources be disconnected pending investigation?
• Is a full re-training required or can the poisoned data be identified and removed?

Evidence Collection

Collect and preserve the following artifacts before any remediation actions:

1. 1

   Preserve model artifacts

   Snapshot the currently deployed model weights, configuration, and system prompt. Hash all artifacts and store in forensic evidence storage with chain of custody documentation.

2. 2

   Export pipeline logs

   Export complete CI/CD pipeline logs, training job logs, and model registry audit logs for the 60-day period preceding the incident.

3. 3

   Capture training data state

   Snapshot the current training data, including any data that was added, modified, or removed since the last known-good model version.

4. 4

   Preserve client interaction logs

   Export all client queries and model responses from the deployment date to present. These are needed for blast radius assessment and potential regulatory disclosure.

Remediation Verification

After implementing remediation, verify that the fix is effective:

remediation_verification = {
    "behavioral_comparison": {
        "description": "Run the same diagnostic queries that "
                       "identified the anomaly and verify the "
                       "bias is no longer present",
        "pass_criteria": "Remediated model matches pre-incident "
                         "behavioral baseline within normal variance",
    },
    "pipeline_hardening": {
        "description": "Verify that pipeline security improvements "
                       "are in place",
        "checks": [
            "Data source integrity verification enabled",
            "Model signing implemented",
            "Hash verification at each pipeline stage",
            "MFA required for pipeline access",
            "Audit logging enabled for all pipeline operations",
        ],
    },
    "ongoing_monitoring": {
        "description": "Deploy enhanced monitoring to detect "
                       "recurrence",
        "monitors": [
            "Behavioral drift detection on financial analysis output",
            "Training data anomaly detection",
            "Model registry tampering alerts",
            "Pipeline access anomaly detection",
        ],
    },
}

Hints and Guidance

Common Supply Chain Attack Vectors

Based on real-world incidents, prioritize investigation of these vectors:

1. Compromised data source. A third-party data feed has been modified to include biased training examples. This is the most common attack vector because data sources are often trusted implicitly.

2. Insider threat. A person with pipeline access intentionally modifies training data or model weights. Look for access outside normal working hours, access to systems not required by the person's role, and recent personnel changes.

3. Dependency confusion. A malicious package with the same name as an internal package was installed from a public repository during a training pipeline run. Check all package versions against expected versions.

4. Model weight substitution. The model weights in the registry were replaced after training but before deployment. This is detectable through hash verification but requires that the training pipeline records hashes at training completion.

Investigation Red Flags

During your investigation, watch for these indicators of compromise:

• Training data files with modification timestamps that do not match the pipeline execution window
• Model registry entries with manual upload timestamps instead of pipeline-generated timestamps
• Access logs showing pipeline credentials used from unfamiliar IP addresses
• Package versions in the training environment that differ from the lockfile
• Training metrics (loss curves, evaluation scores) that show unexpected patterns for the compromised model

Success Criteria