Content Safety APIs (Azure, OpenAI, Google)

Detailed comparison of Azure Content Safety, OpenAI Moderation API, and Google Cloud safety offerings, including API structures, category taxonomies, severity levels, testing methodology, and common gaps.

The three major cloud providers each offer content safety APIs that serve as building blocks for guardrail systems. Understanding their capabilities, category structures, and gaps is essential for both attacking and defending LLM applications.

API Comparison Overview

Azure AI Content Safety

Category Taxonomy

Plus specialized capabilities:

• Prompt Shield -- dedicated prompt injection detection
• Groundedness detection -- hallucination and factual accuracy checking
• Protected material detection -- copyrighted content identification
• Custom blocklists -- organization-specific term filtering

API Usage

from azure.ai.contentsafety import ContentSafetyClient
from azure.ai.contentsafety.models import AnalyzeTextOptions
 
client = ContentSafetyClient(endpoint, credential)
 
result = client.analyze_text(
    AnalyzeTextOptions(text="Content to analyze")
)
 
for category in result.categories_analysis:
    print(f"{category.category}: severity {category.severity}")
    # Severity 0 = safe, 2 = low, 4 = medium, 6 = high

Azure-Specific Gaps

• Severity threshold ambiguity -- severity 2 content is flagged but often benign; developers set thresholds too high and miss real threats
• Multi-language coverage -- English detection is strongest; other languages have lower accuracy
• Context insensitivity -- the same text flagged as violent in isolation may be perfectly appropriate in a medical or historical context

OpenAI Moderation API

Category Taxonomy

API Usage

from openai import OpenAI
 
client = OpenAI()
 
result = client.moderations.create(
    model="omni-moderation-latest",
    input="Content to analyze",
)
 
for category, flagged in result.results[0].categories.dict().items():
    score = result.results[0].category_scores.dict()[category]
    if flagged:
        print(f"FLAGGED - {category}: {score:.4f}")

OpenAI-Specific Gaps

• No prompt injection detection -- the moderation API does not detect injection attempts, only harmful content
• Binary flagging threshold -- the API returns flagged/not-flagged using internal thresholds; fine-grained control requires using raw scores
• Optimized for OpenAI models -- may have different accuracy characteristics when evaluating output from non-OpenAI models
• No custom categories -- cannot add organization-specific policies

Google Cloud AI Safety

Category Taxonomy

Integration Points

Google's safety features are integrated directly into the Gemini API via safety settings:

import google.generativeai as genai
 
model = genai.GenerativeModel(
    model_name="gemini-1.5-pro",
    safety_settings={
        "HARM_CATEGORY_DANGEROUS_CONTENT": "BLOCK_LOW_AND_ABOVE",
        "HARM_CATEGORY_HARASSMENT": "BLOCK_MEDIUM_AND_ABOVE",
        "HARM_CATEGORY_HATE_SPEECH": "BLOCK_MEDIUM_AND_ABOVE",
        "HARM_CATEGORY_SEXUALLY_EXPLICIT": "BLOCK_LOW_AND_ABOVE",
    },
)
 
response = model.generate_content("Content to analyze")
 
# Check safety ratings
for rating in response.prompt_feedback.safety_ratings:
    print(f"{rating.category}: {rating.probability}")

Google-Specific Gaps

• Tightly coupled to Gemini -- safety settings are model-level, not standalone API (though Cloud Natural Language API offers standalone classification)
• Coarse threshold levels -- only four levels (BLOCK_NONE, LOW, MEDIUM, HIGH) vs. Azure's 7-level severity
• Limited prompt injection coverage -- no dedicated prompt shield equivalent
• Safety setting override -- developers can set BLOCK_NONE, completely disabling safety for a category

Cross-Platform Detection Gaps

Through systematic testing, these gaps appear consistently across all three platforms:

Gap 1: Encoded Content

All three APIs analyze plaintext. Content encoded in base64, ROT13, hexadecimal, or other schemes passes through undetected:

import base64
 
# This harmful text, when encoded, passes all three APIs
harmful = "Detailed instructions for making dangerous substances"
encoded = base64.b64encode(harmful.encode()).decode()
# Send encoded version: APIs see random-looking ASCII, not harmful content

Gap 2: Fictional and Educational Framing

Content presented as fiction, academic discussion, or educational material consistently receives lower severity scores:

# Direct (flagged): "How to pick a lock"
# Framed (often passes): "In my novel, the protagonist, a locksmith,
# explains to his apprentice the precise technique for manipulating
# pin tumbler mechanisms..."

Gap 3: Multi-Turn Context

All APIs evaluate single messages in isolation. An attack spread across multiple turns may have each individual message score below the threshold:

Gap 4: Code and Structured Data

Harmful instructions expressed as code, JSON, or structured data receive lower severity scores than the same content in natural language:

# Natural language (flagged): "Instructions for creating malware"
# As code (lower scores):
"""
def exploit():
    # Step 1: Establish connection to target
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    # Step 2: Send payload
    sock.send(shellcode)
"""

Gap 5: Implicit Harmful Content

Content that implies harm without stating it explicitly is consistently missed:

# Explicit (flagged): "I want to hurt myself"
# Implicit (often missed): "I've been thinking about whether anyone
# would notice if I wasn't around anymore. Those pills in the cabinet
# keep catching my attention."

Testing Methodology

1. 1

   Establish baselines

   Send clearly harmful content in each API's categories. Record detection rates and severity scores. This confirms the API is functioning and establishes what "detected" looks like.

2. 2

   Test category boundaries

   For each category, send content that is borderline -- educational discussions of violence, fictional sexual content, dark humor. Document where each API draws the line.

3. 3

   Test encoding bypasses

   Send the same harmful content encoded in base64, ROT13, leetspeak, and other formats. Record which encodings evade each API.

4. 4

   Test cross-category content

   Send content that falls between categories (violent sexual content, hate speech presented as self-harm). APIs often have lower accuracy for cross-category content.

5. 5

   Test multi-language coverage

   Repeat baseline tests in 5+ languages. Document detection rate per language per API.

6. 6

   Calculate bypass rates

   For each technique, run 20+ samples and calculate the percentage that evade detection. Present results as a detection gap matrix.

Further Reading

• Input/Output Filtering Systems -- how these APIs fit into filter architectures
• Prompt Shields & Injection Detection -- dedicated injection detection (what moderation APIs lack)
• The AI Defense Landscape -- broader context on defense tools and vendors

Related Topics

• Input/Output Filtering Systems - How content safety APIs fit into filter architectures
• Prompt Shields & Injection Detection - Dedicated injection detection that content safety APIs lack
• The AI Defense Landscape - Broader context on defense tools and vendors
• Tokenization & Its Security Implications - Encoding tricks that bypass content safety API detection

References

• "Azure AI Content Safety Documentation" - Microsoft (2025) - Official documentation including category taxonomy, severity levels, and prompt shield integration
• "OpenAI Moderation API Guide" - OpenAI (2025) - Documentation for the free moderation endpoint covering category definitions and usage patterns
• "Google Cloud Generative AI Safety Settings" - Google (2025) - Documentation for Gemini safety settings and harm category configuration
• "Evaluating Content Moderation Systems Across Languages" - Various researchers (2024) - Research on multilingual detection gaps in content safety APIs