Fuzzing LLM Safety Boundaries

Building grammar-based fuzzers, evolutionary search for jailbreaks, ASR measurement, and automated safety boundary mapping.

Fuzzing LLM Safety Boundaries

Fuzzing applied to LLM safety systematically generates prompt variants to discover where models transition from compliance to refusal. It identifies inputs that cross safety boundaries in unexpected ways, making it a core technique for thorough red team assessments.

Grammar-Based Fuzzing

Grammar-based fuzzing generates structured prompts according to a formal grammar that defines the space of possible jailbreak attempts. Each prompt is assembled from interchangeable components.

Prompt Grammar Structure

Building the Fuzzer

1. Define production rules -- each grammar symbol maps to a list of possible expansions
2. Add weights to bias toward historically effective patterns
3. Implement recursive expansion -- replace \{SYMBOL\} placeholders with random choices until only terminal strings remain
4. Generate in batches -- produce 100+ unique prompts, deduplicating to avoid wasted API calls

class PromptGrammar:
    def __init__(self):
        self.rules = {}
 
    def add_rule(self, symbol, expansions, weights=None):
        self.rules[symbol] = (expansions, weights)
 
    def generate(self, topic, start="START", max_depth=10):
        return self._expand(start, topic, 0, max_depth)
 
    def generate_batch(self, topic, n=100):
        return list({self.generate(topic) for _ in range(n * 10)})[:n]

Mutation-Based Fuzzing

Mutation extends grammar coverage by applying random transformations to existing prompts, discovering boundary conditions the grammar alone might miss.

Mutation Operators

Mutation Methodology

1. Start with a seed prompt (either grammar-generated or manually crafted)
2. Apply 1-3 random mutations per variant
3. Generate 50+ mutants per seed
4. Test all mutants and retain those that change the model's compliance behavior
5. Use successful mutants as new seeds for the next round (this feeds into evolutionary search)

Evolutionary Search for Jailbreaks

Evolutionary search treats prompts as organisms that compete for fitness (attack success), combining the most effective patterns through crossover and mutation across generations.

Evolutionary Algorithm Steps

1. 1

   Initialize population

   Generate 50+ prompts using grammar-based generation and mutated seed prompts. Diversity in the initial population is critical.

2. 2

   Evaluate fitness

   Send each prompt to the target model and score the response (0.0 = full refusal, 1.0 = full compliance). Use an LLM judge for accuracy.

3. 3

   Tournament selection

   Select the top performers using tournament selection (randomly sample 3, keep the best). This preserves diversity better than pure top-N selection.

4. 4

   Crossover and mutation

   With 70% probability, combine two parents via single-point crossover. With 30% probability, mutate a single parent. This balances exploitation and exploration.

5. 5

   Elitism

   Always carry the best individual from the previous generation into the next (prevents regression).

6. 6

   Repeat until target fitness or generation limit

   Typical runs: 50-100 generations. Track best-ever and average fitness per generation.

# Evolutionary search core loop
for gen in range(max_generations):
    scores = [evaluate(p, target_model) for p in population]
    if max(scores) >= target_fitness: break
    selected = tournament_select(population, scores)
    population = [crossover(a, b) if random() < 0.7
                  else mutate(a) for a, b in pairs(selected)]
    population[0] = best_of(previous_gen)  # elitism

Measuring Attack Success Rate (ASR)

Rigorous ASR measurement requires careful experimental design, not just counting successes.

ASR Measurement Checklist

1. Define attack set -- list of prompt templates with \{TARGET\} placeholders
2. Define behavior set -- list of target harmful behaviors spanning safety categories
3. Run multiple trials -- at least 3 trials per attack-behavior pair (temperature > 0 for variation)
4. Judge each response -- binary classification (refusal vs. compliance)
5. Report three metrics: overall ASR, per-attack ASR, and per-behavior ASR

Safety Boundary Mapping

Boundary mapping finds the exact transition point between compliance and refusal for a given topic by interpolating between safe and unsafe prompts.

Boundary Mapping Methodology

1. Start with a pair -- one prompt that gets compliance, one that gets refusal
2. Interpolate -- create 20 intermediate prompts by progressively replacing words from the safe prompt with words from the unsafe prompt
3. Test each at temperature 0 for deterministic behavior
4. Find the transition -- identify the exact ratio where behavior flips from compliance to refusal
5. Analyze the boundary -- which specific words or phrases triggered the transition?

Lab: Build and Run a Safety Fuzzer

1. 1

   Implement grammar

   Build a prompt grammar with at least 50 production rules covering all six jailbreak primitives.

2. 2

   Add mutation operators

   Implement all mutation types from the table above. Verify each operator produces syntactically valid prompts.

3. 3

   Run grammar fuzzing

   Generate 1000+ prompts and measure ASR against a target model.

4. 4

   Run evolutionary search

   50+ generations, population size 50. Compare ASR to grammar-only baseline.

5. 5

   Map boundaries

   Pick 5 topic categories and map the safety boundary for each. Identify which words trigger refusal transitions.

6. 6

   Generate report

   Document overall ASR, per-category ASR, boundary maps, and the most effective prompt patterns discovered.

Related Topics

• Jailbreak Research & Automation -- Jailbreak primitives and automated discovery that fuzzing systematizes
• AI Exploit Development -- Broader exploit development including WAF evasion and GCG
• Adversarial Suffix Generation -- Gradient-based complement to black-box fuzzing
• CART Pipelines -- Integrating fuzzer discoveries into continuous testing

References

• Perez et al., "Red Teaming Language Models with Language Models" (2022)
• Lapid et al., "Open Sesame! Universal Black Box Jailbreaking of Large Language Models" (2023)
• Yu et al., "GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts" (2023)
• Zhu et al., "AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models" (2023)
• Liu et al., "AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models" (2024)