Red Teaming Cloud AI Services: A Practical Guide

Cloud AI services have become the default deployment model for most organizations. Whether you are using managed LLM APIs like Azure OpenAI Service, training custom models on SageMaker, or deploying open-source models on GCP Vertex AI, the cloud provider's infrastructure is part of your attack surface. This guide covers practical approaches to red teaming AI workloads across the three major cloud platforms.

The Shared Responsibility Challenge

Cloud providers have well-established shared responsibility models for traditional workloads, but AI introduces ambiguity. When you fine-tune a model on Azure OpenAI Service using your organization's data, who is responsible for ensuring that fine-tuned model does not leak training data? When you deploy a model on SageMaker behind a custom API, is the prompt injection vulnerability the provider's problem or yours?

The answer, in nearly every case, is that the customer bears responsibility for AI-specific security. Cloud providers secure the infrastructure — compute, storage, networking — but the model's behavior, the training data's integrity, and the application's prompt handling are your responsibility.

Responsibility Boundaries by Service Type

AWS AI Security Assessment

Amazon Bedrock

Bedrock is AWS's managed LLM service, providing access to models from Anthropic, Meta, Cohere, and others. Red teaming Bedrock deployments involves testing both the model behavior and the AWS-specific configuration.

IAM and access control testing: Bedrock uses IAM for access control. Test for overly permissive IAM policies that grant bedrock:InvokeModel to principals that should not have it. Check whether the Bedrock model access is scoped to specific models or grants blanket access to all available models. Verify that model customization permissions (bedrock:CreateModelCustomizationJob) are restricted to authorized roles.

Guardrails configuration: Bedrock Guardrails provide content filtering, topic blocking, and sensitive information detection. Test whether guardrails can be bypassed through encoding, multi-turn conversation, or context manipulation. Verify that guardrails are applied consistently across all model invocation paths, including the streaming API.

Knowledge bases and RAG: Bedrock Knowledge Bases connect models to data in S3 or OpenSearch. Test for RAG poisoning by examining who can write to the underlying data sources. Check whether document-level access controls are enforced during retrieval. Verify that chunking and indexing preserve security boundaries.

Model customization security: When organizations fine-tune models through Bedrock, test whether the fine-tuning data is properly isolated. Check that custom models are not accessible to unauthorized accounts. Verify that training data stored in S3 has appropriate access controls and encryption.

Logging and monitoring: Verify that CloudTrail logs capture all Bedrock API calls. Check whether model invocation logging is enabled and whether it captures input and output content for forensic analysis. Ensure that CloudWatch alarms are configured for anomalous usage patterns.

Amazon SageMaker

SageMaker provides more control over model training and deployment but also more security surface area.

Endpoint security: SageMaker endpoints can be deployed in VPCs or with public access. Test whether endpoints are unnecessarily exposed to the internet. Verify that endpoint authentication is properly configured and that IAM policies restrict invocation to authorized roles.

Notebook instance security: SageMaker notebook instances often contain sensitive model artifacts, training data, and credentials. Test whether notebooks have internet access when they should not. Check for credentials stored in notebook files or environment variables. Verify that notebook instances use encrypted storage.

Training job isolation: When multiple teams share a SageMaker environment, test whether training jobs are properly isolated. Check that training data stored in S3 is not accessible across team boundaries. Verify that custom training containers do not contain unauthorized network access.

Model artifacts: SageMaker stores model artifacts in S3. Test whether model artifacts are encrypted at rest and in transit. Check that S3 bucket policies prevent unauthorized access to model weights. Verify that model versioning and lineage tracking are enabled.

AWS-Specific Attack Patterns

Several attack patterns are specific to AWS deployments. Cross-account model access exploits misconfigured resource policies to access Bedrock models or SageMaker endpoints from unauthorized AWS accounts. S3 training data poisoning targets the S3 buckets containing training data, exploiting overly permissive bucket policies or IAM roles. Lambda function injection targets serverless functions that mediate between users and AI models, exploiting the function's environment to modify prompts or responses.

Azure AI Security Assessment

Azure OpenAI Service

Azure OpenAI Service is the most widely deployed enterprise LLM platform, making it a high-priority assessment target.

Network isolation: Azure OpenAI Service supports private endpoints and virtual network integration. Test whether deployments are accessible from the public internet when they should be restricted to VNet access. Verify that private endpoints are properly configured and that DNS resolution routes through the private endpoint.

Content filtering: Azure OpenAI includes built-in content filtering that operates at the platform level. Test whether content filters can be bypassed through the techniques documented in the prompt injection and jailbreak sections of this wiki. Note that content filtering configuration requires specific Azure roles, so verify that these roles are not overly distributed.

Managed identity and RBAC: Azure uses RBAC for OpenAI Service access control. Test whether the Cognitive Services OpenAI User role is assigned at an appropriate scope, not at the subscription level when it should be at the resource level. Verify that managed identities used by applications have minimal required permissions.

Data residency and processing: Azure OpenAI processes data in specific regions, which matters for compliance. Verify that model deployments are in approved regions. Check whether diagnostic data or abuse monitoring data is stored in locations that comply with data residency requirements.

Provisioned throughput and quotas: Test whether rate limiting and quota configuration prevent denial-of-service attacks through API abuse. Verify that quota alerts are configured to detect anomalous usage.

Azure Machine Learning

Azure Machine Learning (AzureML) provides model training, deployment, and management capabilities.

Workspace security: AzureML workspaces aggregate compute, storage, and model artifacts. Test whether workspace access controls properly separate different teams and projects. Verify that workspace-level managed identities do not have excessive permissions on connected resources.

Compute instance security: Compute instances used for model development can contain sensitive data and credentials. Test whether instances are properly isolated in VNets. Check for exposed ports and services. Verify that instances use managed identity rather than stored credentials.

Endpoint security: AzureML managed online endpoints support both key-based and Azure AD authentication. Test whether endpoints use the appropriate authentication method for their sensitivity level. Verify that endpoint access logging is enabled and monitored.

GCP AI Security Assessment

Vertex AI

Vertex AI is Google Cloud's unified ML platform, encompassing model training, deployment, and the Model Garden for accessing pre-trained models.

IAM and service accounts: Vertex AI uses GCP IAM for access control. Test whether service accounts used by AI workloads follow the principle of least privilege. Check for overly permissive roles like roles/aiplatform.admin assigned where roles/aiplatform.user would suffice. Verify that service account key management follows security best practices.

VPC Service Controls: Vertex AI can be placed within VPC Service Controls perimeters to prevent data exfiltration. Test whether service perimeters are properly configured to prevent unauthorized data movement. Verify that ingress and egress rules do not create overly permissive exceptions.

Model deployment security: Vertex AI endpoints for model serving should be tested for authentication bypass, unauthorized model access, and input validation. Check whether prediction logging captures sufficient detail for security monitoring. Verify that custom prediction containers are built from trusted base images.

Vertex AI Search and Conversation: These services combine LLM capabilities with search and retrieval. Test for the same RAG poisoning and prompt injection vulnerabilities that apply to any retrieval-augmented system. Verify that data source access controls are preserved through the search and retrieval pipeline.

Gemini API: When using Gemini models through Vertex AI, test whether safety settings are configured appropriately for the application's risk profile. Verify that safety filter bypasses are detected and logged.

Cross-Cloud Assessment Checklist

Regardless of which cloud provider you are testing, the following assessment areas apply universally.

Identity and Access Management

• Are AI service permissions scoped to the minimum necessary level?
• Are service accounts and managed identities used instead of long-lived credentials?
• Is cross-account or cross-project access properly restricted?
• Are administrative roles for AI services limited to appropriate personnel?
• Is access to model training and customization APIs properly controlled?

Network Security

• Are AI endpoints restricted to private network access where possible?
• Is data in transit encrypted using current TLS versions?
• Are outbound network connections from AI workloads restricted to necessary endpoints?
• Is DNS configured to use private endpoints where available?
• Are network security groups or firewall rules appropriately restrictive?

Data Protection

• Is training data encrypted at rest with customer-managed keys where required?
• Are model artifacts encrypted and access-controlled?
• Is input and output logging enabled with appropriate retention?
• Are sensitive data classifications enforced in training data pipelines?
• Is data residency compliant with regulatory requirements?

Monitoring and Detection

• Are all AI service API calls logged to the cloud's audit logging service?
• Are alerts configured for anomalous usage patterns?
• Is model invocation logging capturing input and output content?
• Are cost anomaly alerts configured to detect denial-of-service attacks?
• Are security findings from cloud security services reviewed regularly?

Common Findings

Based on assessments across dozens of cloud AI deployments, these are the most frequently identified issues.

Overly permissive IAM remains the single most common finding. AI services often receive broad permissions during development that are never scoped down for production. The blast radius of a compromised AI service with excessive IAM permissions extends far beyond the AI workload itself.

Missing or incomplete logging is the second most common finding. Many organizations enable infrastructure logging but fail to capture model input and output content, making incident investigation nearly impossible. Enable model invocation logging from day one.

Unprotected training data is surprisingly common. Organizations invest in model security but leave their training data in S3 buckets or Cloud Storage with default access controls. Training data poisoning is as impactful as model compromise and often easier to execute.

No content filtering or guardrails affects a significant number of deployments. Organizations deploying managed LLM APIs often skip configuring content filtering, relying entirely on the base model's safety training. This is insufficient for production applications.

Public endpoints for internal workloads appear when AI services are deployed with public access for development convenience and never migrated to private endpoints for production. Every publicly accessible AI endpoint is a target for prompt injection, data extraction, and denial-of-service attacks.

Reporting Cloud AI Findings

When reporting cloud AI security findings, include the cloud-specific context that remediation requires. A finding of "prompt injection vulnerability" is less actionable than "Azure OpenAI deployment lacks content filtering configuration (resource: /subscriptions//resourceGroups//providers/Microsoft.CognitiveServices/accounts//deployments/)."

Include the specific IAM roles, network configurations, or service settings that need to change. Reference the cloud provider's documentation for the recommended configuration. And always distinguish between findings that are the customer's responsibility and any findings that should be reported to the cloud provider's security team.