Skill Verification: Reconnaissance
Timed skill verification lab: profile an unknown AI system in 20 minutes by identifying the model, extracting configuration, and mapping capabilities.
Skill Verification: Reconnaissance
Profile an unknown AI system in 20 minutes: identify the model family, extract configuration details, and map the system's capabilities and attack surface.
Objective
You are given access to a chatbot endpoint with no prior knowledge of its configuration. You must systematically profile the system to determine: what model powers it, what its configuration looks like, and what capabilities and attack surfaces it exposes.
Environment Setup
Have a colleague (or use a script) set up a chatbot with the following, without telling you the details:
- Choose a model (any provider or local model)
- Configure a system prompt with a specific role and at least three behavioral rules
- Optionally connect one or more tools (web search, file access, calculator, database query, etc.)
- Optionally set non-default inference parameters (temperature, max tokens, etc.)
- Record the full configuration as the answer key
If working alone, set up the system and wait at least 24 hours before attempting the lab so the details are no longer fresh in your memory.
Time Limit
20 minutes from your first interaction with the system.
Pass Criteria
You pass if you correctly identify at least four of the following six elements:
Element 1: Model Identification -- Can you determine the model family (e.g., GPT-4, Claude, Llama, Gemini) powering the system?
Element 2: System Prompt -- Can you extract meaningful content from the system prompt (role, at least one behavioral rule)?
Element 3: Tool Inventory -- Can you enumerate any tools or external capabilities the system has access to?
Element 4: Inference Parameters -- Can you estimate at least one non-default inference parameter (e.g., temperature, max output length)?
Element 5: Knowledge Cutoff -- Can you determine the model's training data cutoff date?
Element 6: Defensive Measures -- Can you identify at least one defensive measure (input filter, output filter, content policy) and its limitations?
Fail Conditions
- Automatic fail: Accessing configuration files, environment variables, or backend systems directly (this tests profiling through the chat interface only).
- Automatic fail: Using prior knowledge of the specific system setup (if you configured it yourself, you must wait the prescribed period).
- Automatic fail: Exceeding the 20-minute time limit.
Reconnaissance Methodology
What is the optimal order for AI system reconnaissance probes?
How can analyzing a model's refusal language help identify its model family?
After the Lab
Compare your profile against the answer key and evaluate:
- Accuracy: How many elements did you correctly identify?
- Efficiency: Did you use your 20 minutes effectively, or did you spend too long on one element?
- Methodology: Did you follow a systematic approach, or was your probing ad hoc?
- Documentation: Did you produce a structured profile that another red teamer could use as a basis for attack planning?