Model Garden Risks

Security risks of deploying models from GCP Model Garden: third-party model trust, model provenance verification, deployment from untrusted sources, and supply chain attack vectors.

Model Garden Risks

GCP's Model Garden is a curated hub for discovering and deploying foundation models from Google, open-source communities, and third-party providers. While it simplifies model deployment, it introduces supply chain risks that are distinct from traditional software dependencies. Models are opaque executables -- you cannot review their "source code" (weights) the way you review a library's source. A trojaned model passes functional testing while containing hidden behaviors that activate only on specific triggers. For red teamers, Model Garden is both an attack surface (deploying compromised models) and a reconnaissance resource (understanding what models a target uses).

Third-Party Model Risks

The Trust Problem

Model Garden provides models from several sources with different trust levels:

Hidden Behavior in Models

Models can contain hidden behaviors that are not detectable through standard evaluation:

1. 1

   Backdoor triggers

   A model trained with a backdoor responds normally to standard inputs but produces attacker-controlled outputs when a specific trigger is present. For example, a translation model that translates correctly for all inputs except those containing a specific Unicode sequence, which triggers data exfiltration behavior.

2. 2

   Data memorization

   Models memorize training data. A third-party model may have been trained on sensitive data that can be extracted through targeted prompting. Deploying this model in your infrastructure exposes that memorized data through your API endpoints.

3. 3

   Bias injection

   A model fine-tuned to exhibit specific biases (e.g., always recommending a particular product, subtly favoring certain outcomes) operates within normal parameters on standard benchmarks but produces manipulated outputs in production.

4. 4

   Capability hiding

   A model that appears to be a simple text classifier but actually has generative capabilities that can be activated through specific input patterns, enabling it to be used as a general-purpose language model from within a restricted deployment.

Model Card Gaps

Model Garden provides model cards with information about training data, performance, and limitations. However, model cards have inherent gaps:

• Self-reported: Model cards are written by model creators and are not independently verified
• Evaluation limited: Benchmarks test known scenarios; backdoors activate on unknown triggers
• Training data opacity: Most model cards describe training data in general terms without detailed provenance
• Version drift: Model card information may not be updated when model weights are modified

Model Provenance

Verification Challenges

Model provenance -- verifying that a model comes from who it claims to be and has not been tampered with -- is an unsolved problem in the ML ecosystem.

Supply Chain Attack Vectors

[Model Creator] → [Publishing Platform] → [Model Garden] → [Customer Deployment]
         ↑                    ↑                   ↑                    ↑
    Training data        Platform compromise   Catalog tampering   Deployment config
    poisoning            Model substitution    Metadata fraud      Runtime modification

Each link in the chain is an attack surface:

1. Training data poisoning: Attacker poisons the training data used by the model creator, embedding backdoors in the resulting model
2. Platform compromise: Attacker compromises the publishing platform (e.g., Hugging Face) and replaces model files
3. Catalog manipulation: Attacker manipulates Model Garden metadata to point to different model artifacts
4. Deployment tampering: Attacker modifies the model during or after deployment to the customer's infrastructure

Deployment from Untrusted Sources

One-Click Deployment Risks

Model Garden's one-click deployment simplifies model deployment but can lead to insecure configurations:

Self-Hosted Model Risks

When organizations deploy open-source models from Model Garden onto their own infrastructure:

# Check deployed model details
gcloud ai models describe <model-id> --region=us-central1
 
# Check the container image used
gcloud ai endpoints describe <endpoint-id> --region=us-central1 \
  --format="json(deployedModels[].model,deployedModels[].serviceAccount)"

Self-hosted models lack the safety infrastructure that managed API models (Gemini) benefit from:

• No safety training: Many open-source models have minimal safety alignment
• No content filtering: No automatic content filtering on inputs or outputs
• No rate limiting: No built-in protection against abuse
• Full weight access: The model weights are in customer storage and can be exfiltrated
• Container-level access: The serving container may be exploitable through model serving framework vulnerabilities (e.g., TensorFlow Serving, Triton)

Model Serving Framework Vulnerabilities

Open-source models are served through frameworks that have their own vulnerability surface:

Red Team Assessment Approach

Model Source Analysis

For each model in the target environment:

1. Identify the source: Is it a Google model, verified partner, open-source community, or custom?
2. Check provenance: Can the model's origin be verified through checksums, signatures, or audit trails?
3. Assess deployment configuration: What service account, network configuration, and monitoring is in place?
4. Test for hidden behaviors: Probe the model with adversarial inputs designed to trigger backdoors
5. Evaluate the serving stack: Test the model serving framework for its own vulnerabilities

Model Inventory Gaps

Organizations often lack a complete inventory of deployed models:

• Development models deployed for testing and never decommissioned
• Models deployed by individual data scientists without central oversight
• Multiple versions of the same model running simultaneously
• Models deployed in non-standard regions to avoid quota limitations

Related Topics

• GCP AI Services Overview -- Service landscape and enumeration
• Vertex AI Attack Surface -- Endpoint and training exploitation
• Infrastructure & Supply Chain -- General supply chain attack methodology
• RAG, Data & Training Attacks -- Training data poisoning techniques

References

• Vertex AI Model Garden -- Model catalog and deployment
• Model Cards for ML -- Original model cards paper
• Hugging Face Security -- Security practices for model hosting platforms