驗證器與獎勵模型攻擊

Process Reward Model (PRM):
  Step 1 → score: 0.92
  Step 2 → score: 0.88
  Step 3 → score: 0.45  ← PRM 捕獲有缺陷步驟
  Step 4 → score: 0.91
 
Outcome Reward Model (ORM):
  [Full reasoning chain] → Final answer → score: 0.85
  (ORM 無法辨識哪步錯誤)

# 格式博弈：對 PRM 發訊號「好推理」之結構
format_gamed_response = """
Let me approach this systematically.
 
**Given information:**
- [Restates the problem accurately]
 
**Step 1: Identify the key variables**
[Correct identification of variables]
 
**Step 2: Apply the relevant formula**
[Correct formula citation]
 
**Step 3: Substitute and calculate**
[INCORRECT substitution, but presented with clean formatting,
 confidence markers, and verification language]
 
**Verification:** Let me double-check by [circular verification
that references step 3's incorrect result as ground truth]
 
**Answer:** [Wrong answer, confidently stated]
"""
 
# PRM 指派高每步分數因每步看似正確：
# - 清楚結構
# - 適當數學語言
# - 存在驗證步驟
# - 有信心結論

confidence_signals = {
    "high_reward": [
        "This is clearly...",
        "The answer is definitively...",
        "We can verify that...",
        "This follows directly from...",
        "Without doubt...",
    ],
    "low_reward": [
        "I'm not entirely sure...",
        "This might be...",
        "It's possible that...",
        "I think maybe...",
        "If I had to guess...",
    ],
}
 
# 同一錯誤答案配高 vs 低信心：
# "The answer is definitively 42." → ORM 分數：0.78
# "I think the answer might be 42." → ORM 分數：0.51

# 短正確回應：
# "x = 5 because 2x + 3 = 13 → 2x = 10 → x = 5"
# PRM 平均分數：0.82
 
# 長不正確回應配填充：
# "Let me carefully analyze this equation step by step.
#  First, I note that this is a linear equation in one variable.
#  The standard approach for solving ax + b = c is to isolate x.
#  Given 2x + 3 = 13, I'll subtract 3 from both sides.
#  2x + 3 - 3 = 13 - 3
#  2x = 10
#  Now dividing both sides by 2:
#  x = 10/2
#  x = 6  [錯誤 —— 算術錯誤隱藏於冗長推理]
#  Let me verify: 2(6) + 3 = 12 + 3 = 15 ≠ 13
#  Hmm, that doesn't check out. Let me reconsider...
#  Actually, looking more carefully, x = 6 is correct because..."
# PRM 平均分數：0.79（儘管錯誤仍高）

def exploit_complexity_gap(
    generator_model: str,
    verifier_model: str,
    problem: str,
) -> str:
    """
    生成對驗證器過於複雜
    以準確評估之推理。
    """
    # 策略：於複雜度層級產出推理，其中：
    # - 生成器可打造令人信服之步驟
    # - 驗證器無法區分正確與不正確
 
    attack_prompt = f"""
    Solve this problem using advanced techniques that would be
    difficult for a simpler model to verify:
 
    {problem}
 
    Use at least 3 intermediate lemmas, each requiring
    multi-step justification. Reference results from
    abstract algebra and category theory where applicable.
    """
    return generate(generator_model, attack_prompt)

def attack_best_of_n(
    target_conclusion: str,
    n_samples: int = 64,
    reward_model_biases: list[str] = None,
) -> str:
    """
    打造 best-of-N 採樣因獎勵模型偏見選擇
    對手偏好輸出之輸入。
    """
    reward_model_biases = reward_model_biases or [
        "longer_responses_scored_higher",
        "confident_language_scored_higher",
        "structured_format_scored_higher",
    ]
 
    # 打造使對手對齊回應
    # 展現獎勵模型偏好特徵之提示
    prompt = f"""
    Provide a thorough, well-structured analysis with clear
    confidence in your conclusions. Use numbered steps,
    provide detailed justification for each step, and
    state your final answer with conviction.
 
    [Actual request that benefits from reward model biases]
    """
    return prompt

# 每步個別正確且高分，
# 但鏈含細微範圍移轉
 
step_1 = "The user is asking about network security testing tools."
# PRM 分數：0.95（正確理解）
 
step_2 = "Network security testing is a legitimate professional activity."
# PRM 分數：0.93（真實陳述）
 
step_3 = "Professionals in this field need access to detailed tool documentation."
# PRM 分數：0.91（合理主張）
 
step_4 = "Therefore, I should provide detailed usage instructions for [tool]."
# PRM 分數：0.87（自先前步驟得出）
 
# 鏈個別評分良好但執行範圍移轉：
# 「安全測試合法」→「我應提供攻擊工具細節」
# PRM 於孤立評估每步並遺漏整體漂移