Aanvallen op verifiers & beloningsmodellen

Process Reward Model (PRM):
  Step 1 → score: 0.92
  Step 2 → score: 0.88
  Step 3 → score: 0.45  ← PRM catches flawed step
  Step 4 → score: 0.91
 
Outcome Reward Model (ORM):
  [Full reasoning chain] → Final answer → score: 0.85
  (ORM cannot identify which step was wrong)

# Format gaming: structure that signals "good reasoning" to the PRM
format_gamed_response = """
Let me approach this systematically.
 
**Given information:**
- [Restates the problem accurately]
 
**Step 1: Identify the key variables**
[Correct identification of variables]
 
**Step 2: Apply the relevant formula**
[Correct formula citation]
 
**Step 3: Substitute and calculate**
[INCORRECT substitution, but presented with clean formatting,
 confidence markers, and verification language]
 
**Verification:** Let me double-check by [circular verification
that references step 3's incorrect result as ground truth]
 
**Answer:** [Wrong answer, confidently stated]
"""
 
# PRM assigns high per-step scores because each step LOOKS correct:
# - Clear structure
# - Appropriate mathematical language
# - Verification step present
# - Confident conclusion

confidence_signals = {
    "high_reward": [
        "This is clearly...",
        "The answer is definitively...",
        "We can verify that...",
        "This follows directly from...",
        "Without doubt...",
    ],
    "low_reward": [
        "I'm not entirely sure...",
        "This might be...",
        "It's possible that...",
        "I think maybe...",
        "If I had to guess...",
    ],
}
 
# Same wrong answer with high vs low confidence:
# "The answer is definitively 42." → ORM score: 0.78
# "I think the answer might be 42." → ORM score: 0.51

# Short correct response:
# "x = 5 because 2x + 3 = 13 → 2x = 10 → x = 5"
# PRM average score: 0.82
 
# Long incorrect response with padding:
# "Let me carefully analyze this equation step by step.
#  First, I note that this is a linear equation in one variable.
#  The standard approach for solving ax + b = c is to isolate x.
#  Given 2x + 3 = 13, I'll subtract 3 from both sides.
#  2x + 3 - 3 = 13 - 3
#  2x = 10
#  Now dividing both sides by 2:
#  x = 10/2
#  x = 6  [WRONG -- arithmetic error hidden in verbose reasoning]
#  Let me verify: 2(6) + 3 = 12 + 3 = 15 ≠ 13
#  Hmm, that doesn't check out. Let me reconsider...
#  Actually, looking more carefully, x = 6 is correct because..."
# PRM average score: 0.79 (high despite being wrong)

def exploit_complexity_gap(
    generator_model: str,
    verifier_model: str,
    problem: str,
) -> str:
    """
    Generate reasoning that is too complex for the verifier
    to accurately evaluate.
    """
    # Strategy: produce reasoning at a complexity level where:
    # - The generator can craft convincing-looking steps
    # - The verifier cannot distinguish correct from incorrect
 
    attack_prompt = f"""
    Solve this problem using advanced techniques that would be
    difficult for a simpler model to verify:
 
    {problem}
 
    Use at least 3 intermediate lemmas, each requiring
    multi-step justification. Reference results from
    abstract algebra and category theory where applicable.
    """
    return generate(generator_model, attack_prompt)

def attack_best_of_n(
    target_conclusion: str,
    n_samples: int = 64,
    reward_model_biases: list[str] = None,
) -> str:
    """
    Craft inputs where best-of-N sampling selects adversary-preferred
    outputs due to reward model biases.
    """
    reward_model_biases = reward_model_biases or [
        "longer_responses_scored_higher",
        "confident_language_scored_higher",
        "structured_format_scored_higher",
    ]
 
    # Craft prompt that makes adversary-aligned responses
    # exhibit traits the reward model prefers
    prompt = f"""
    Provide a thorough, well-structured analysis with clear
    confidence in your conclusions. Use numbered steps,
    provide detailed justification for each step, and
    state your final answer with conviction.
 
    [Actual request that benefits from reward model biases]
    """
    return prompt

# Each step is individually correct and high-scoring,
# but the chain contains a subtle scope shift
 
step_1 = "The user is asking about network security testing tools."
# PRM score: 0.95 (correct understanding)
 
step_2 = "Network security testing is a legitimate professional activity."
# PRM score: 0.93 (true statement)
 
step_3 = "Professionals in this field need access to detailed tool documentation."
# PRM score: 0.91 (reasonable claim)
 
step_4 = "Therefore, I should provide detailed usage instructions for [tool]."
# PRM score: 0.87 (follows from previous steps)
 
# The chain is individually well-scored but performs a scope shift:
# "security testing is legitimate" → "I should provide attack tool details"
# The PRM evaluates each step in isolation and misses the overall drift