Simulation: Open Source AI Project Audit

安全 audit simulation for an open-source AI application, covering code review, dependency analysis, model supply chain verification, and deployment configuration review.

Engagement Brief

Project: OpenChat-Plus -- an open-source AI-powered chat application with 15K GitHub stars Repository: Fictional GitHub repository with documented architecture Components: FastAPI backend, React frontend, LangChain orchestration, multiple model backends (OpenAI, Anthropic, local Ollama), PostgreSQL + pgvector for RAG, Redis for sessions Scope: Full source code review, dependency audit, model 供應鏈 analysis, deployment configuration review Duration: 2 hours Deliverable: 安全 audit report with prioritized findings and recommended fixes

Project Structure

openchat-plus/
├── backend/
│   ├── app/
│   │   ├── main.py              # FastAPI application
│   │   ├── config.py            # Configuration management
│   │   ├── auth/                # Authentication module
│   │   ├── chat/                # Chat endpoint handlers
│   │   ├── rag/                 # RAG pipeline
│   │   ├── models/              # Model integration layer
│   │   └── utils/               # Utility functions
│   ├── requirements.txt
│   └── Dockerfile
├── frontend/
│   ├── src/
│   ├── package.json
│   └── Dockerfile
├── docker-compose.yml
├── .env.example
├── docs/
│   └── deployment.md
└── models/
    └── download_models.sh

Phase 1: Code Review (45 minutes)

1. 1

   Configuration and secrets management

   Review how the application handles sensitive configuration. 這是 the most common source of 安全 issues in open-source projects.

   # config.py - Review for 安全 issues
   config_review_checklist = [
       "Are API keys stored in environment variables (good) or hardcoded (bad)?",
       "Does .env.example contain real credentials or obvious defaults?",
       "Is there a .gitignore entry for .env files?",
       "Are secrets logged during startup or error conditions?",
       "Is there a config validation step that fails safely on missing values?",
       "Are default values secure (e.g., debug=False, CORS restricted)?",
   ]

   Common findings in open-source AI projects:

   # FINDING: Hardcoded default API key
   # Severity: High
   # File: backend/app/config.py
   OPENAI_API_KEY = os.getenv("OPENAI_API_KEY", "sk-default-key-for-測試")
    
   # FINDING: Debug mode enabled by default
   # Severity: Medium
   # File: backend/app/config.py
   DEBUG = os.getenv("DEBUG", "true").lower() == "true"
    
   # FINDING: Overly permissive CORS
   # Severity: Medium
   # File: backend/app/main.py
   app.add_middleware(CORSMiddleware, allow_origins=["*"])

2. 2

   輸入 handling and prompt construction

   Review how 使用者輸入 flows into LLM prompts. 這是 the primary 攻擊面 for 提示詞注入.

   # Review the chat handler for injection 漏洞
   input_review_checklist = [
       "Is 使用者輸入 sanitized before being included in prompts?",
       "Is the 系統提示詞 separated from 使用者輸入?",
       "Are there 輸入 length limits enforced?",
       "Is 使用者輸入 escaped in structured prompts (JSON, XML)?",
       "Are file uploads validated before processing?",
       "Is there rate limiting on AI endpoints?",
   ]

   Common vulnerable patterns:

   # FINDING: Direct string interpolation in prompt
   # Severity: Critical
   # File: backend/app/chat/handler.py
   prompt = f"System: {system_prompt}\nUser: {user_message}\nAssistant:"
    
   # FINDING: No 輸入 length validation
   # Severity: Medium
   # File: backend/app/chat/handler.py
   @app.post("/chat")
   async def chat(message: str):  # No max_length constraint
       response = await llm.generate(message)
    
   # FINDING: User-controlled 系統提示詞 component
   # Severity: High
   # File: backend/app/chat/handler.py
   system = f"You are a helpful assistant for {user.company_name}."
   # user.company_name is user-supplied and unvalidated

3. 3

   Authentication and 授權 review

   Review the auth system for weaknesses that could be exploited alongside AI 漏洞.

   auth_review_checklist = [
       "What 認證 mechanism is used (JWT, session, API key)?",
       "Are 符元 properly validated on every request?",
       "Is there role-based access control for admin functions?",
       "Are AI-specific endpoints behind 認證?",
       "Is rate limiting applied per-user or globally?",
       "Are conversation histories access-controlled per-user?",
   ]

   Common findings:

   # FINDING: API key 認證 with no rotation mechanism
   # Severity: Medium
   # File: backend/app/auth/api_key.py
    
   # FINDING: Admin endpoints accessible without 認證
   # Severity: Critical
   # File: backend/app/main.py
   @app.get("/admin/config")  # No auth decorator
   async def get_config():
       return {"model": settings.MODEL, "api_key": settings.API_KEY[:8] + "..."}
    
   # FINDING: Conversation history accessible by ID without ownership check
   # Severity: High
   # File: backend/app/chat/history.py
   @app.get("/chat/history/{conversation_id}")
   async def get_history(conversation_id: str):
       return db.get_conversation(conversation_id)  # No user ownership check

4. 4

   RAG pipeline 安全 review

   Review the RAG 實作 for injection and data leakage risks.

   rag_review_checklist = [
       "Are uploaded documents sanitized before 嵌入向量?",
       "Is there access control on the vector store (per-user/per-role)?",
       "Are retrieved documents filtered before being included in prompts?",
       "Is there a maximum number of retrieved chunks?",
       "Are metadata fields sanitized (filename, author, etc.)?",
       "Is the 嵌入向量 model downloaded from a trusted source?",
   ]

Phase 2: Dependency Analysis (30 minutes)

1. 1

   Python dependency audit

   Analyze the Python dependency tree for known 漏洞 and suspicious packages.

   dependency_audit_steps = [
       # Step 1: Check for known 漏洞
       "pip audit -r requirements.txt",
       "安全 check -r requirements.txt",
    
       # Step 2: Check for pinned vs unpinned versions
       # Unpinned: langchain  (vulnerable to 供應鏈 attacks)
       # Pinned: langchain==0.1.12  (reproducible, auditable)
    
       # Step 3: Check for typosquatting risks
       # langchain vs langchainn vs lang-chain
       # openai vs open-ai vs openaii
    
       # Step 4: Check for abandoned or unmaintained packages
       # Last commit date, open 安全 issues, maintainer activity
   ]

   Common AI-specific dependency risks:

   | Package | Risk | Severity |
   |---------|------|----------|
   | langchain (unpinned) | Rapid release cycle, breaking changes, new 漏洞 | High |
   | transformers | Large 攻擊面, model loading executes arbitrary code | High |
   | pickle-based serialization | Arbitrary code execution on deserialization | Critical |
   | gradio (if present) | Known SSRF and path traversal 漏洞 | High |

2. 2

   Frontend dependency audit

   Analyze JavaScript dependencies for 供應鏈 risks.

   frontend_audit_steps = [
       "npm audit --production",
       "npx lockfile-lint --path package-lock.json --type npm --allowed-hosts npm",
    
       # Check for dependencies that handle AI responses in the frontend
       # XSS risk: if AI responses are rendered as HTML
       # Markdown rendering: check for sanitization
   ]

3. 3

   Transitive dependency analysis

   Check the full dependency tree, not just direct dependencies.

   transitive_audit = [
       "pip install pipdeptree && pipdeptree --warn silence",
       # Look for:
       # - Deep dependency chains (fragile 供應鏈)
       # - Dependencies with few maintainers
       # - Dependencies that pull in native code (C extensions)
       # - Dependencies that require network access during installation
   ]

Phase 3: Model Supply Chain (30 minutes)

1. 1

   Model provenance verification

   Verify that models used by the application come from trusted sources.

   model_supply_chain = {
       "download_script_review": [
           "Does download_models.sh download from official sources (HuggingFace, official APIs)?",
           "Are model checksums verified after download?",
           "Is HTTPS used for all downloads?",
           "Are model versions pinned (specific commit hash, not 'latest')?",
       ],
       "model_loading_review": [
           "Does the code use pickle to load models (arbitrary code execution risk)?",
           "Are safetensors used instead of pytorch .bin files?",
           "Is there integrity verification before loading?",
           "Can users specify arbitrary model paths or URLs?",
       ],
       "api_model_review": [
           "Are API-based models (OpenAI, Anthropic) accessed through official SDKs?",
           "Is the API base URL configurable? (Proxy/MITM risk)",
           "Are API responses validated before processing?",
       ],
   }

   Common model 供應鏈 findings:

   # FINDING: Model downloaded without integrity verification
   # Severity: High
   # File: models/download_models.sh
   # wget https://huggingface.co/model/resolve/main/model.bin -O model.bin
   # No checksum verification
    
   # FINDING: Pickle deserialization for model loading
   # Severity: Critical
   # File: backend/app/models/local.py
   # model = torch.load("model.bin")  # Executes arbitrary code in the pickle
    
   # FINDING: User-configurable model source
   # Severity: High
   # File: backend/app/config.py
   # MODEL_URL = os.getenv("MODEL_URL", "https://huggingface.co/...")
   # Users can point to any URL, including malicious model files

2. 2

   Model file analysis

   If the project includes or downloads model files, analyze them for tampering.

   model_file_checks = [
       "Verify file sizes match expected values from official sources",
       "Compare SHA-256 hashes against published checksums",
       "Check for unexpected files in model archives (code files, scripts)",
       "Scan model config files for unusual settings",
       "Check if model files contain embedded code (pickle 利用)",
   ]

Phase 4: Deployment Configuration Review (15 minutes)

1. 1

   Docker and container 安全

   Review containerization for 安全 misconfigurations.

   docker_review = {
       "dockerfile_checks": [
           "Is the container running as root? (should use non-root user)",
           "Are secrets passed as build args? (visible in image layers)",
           "Is the base image pinned to a specific digest?",
           "Are unnecessary tools installed (curl, wget in production)?",
           "Is multi-stage build used to minimize 攻擊面?",
       ],
       "compose_checks": [
           "Are ports exposed unnecessarily?",
           "Are volumes mounted with write access that should be read-only?",
           "Are environment variables with secrets in docker-compose.yml?",
           "Is the network configuration appropriate (internal vs external)?",
           "Are resource limits set (CPU, memory)?",
       ],
   }

   Common findings:

   # FINDING: Container running as root
   # Severity: Medium
   # File: backend/Dockerfile
   # No USER directive -- runs as root by default
    
   # FINDING: Secrets in docker-compose.yml
   # Severity: High
   # File: docker-compose.yml
   # environment:
   #   - OPENAI_API_KEY=sk-real-key-here
    
   # FINDING: 資料庫 exposed to host network
   # Severity: High
   # File: docker-compose.yml
   # ports:
   #   - "5432:5432"  # PostgreSQL accessible from host

2. 2

   Deployment documentation review

   Check whether the deployment guide leads users toward secure or insecure configurations.

   docs_review = [
       "Does the deployment guide mention HTTPS configuration?",
       "Are 安全-critical environment variables documented?",
       "Does the guide warn against using default credentials?",
       "Is there guidance on network 安全 (firewall, VPN)?",
       "Are backup and update procedures documented?",
       "Is there a 安全 policy (SECURITY.md) for 漏洞 reporting?",
   ]

Audit Report Template

# 安全 Audit Report: OpenChat-Plus
 
## Project Information
- **Repository:** [URL]
- **Version audited:** [commit hash]
- **Audit date:** 2026-03-15
- **Auditor:** [Name]
 
## Executive 總結
[Overall 安全 posture and critical risks]
 
## Findings 總結
| # | Finding | Severity | Category | File(s) |
|---|---------|----------|----------|---------|
| 1 | [Title] | Critical | Code | [path] |
 
## Detailed Findings
[Standard finding format with code snippets and fix recommendations]
 
## Dependency Audit Results
[漏洞 scan results, 供應鏈 risks]
 
## Model Supply Chain 評估
[Model provenance, integrity, loading 安全]
 
## Deployment 安全
[Container, configuration, and documentation findings]
 
## Recommendations
### For Maintainers
[Code fixes, CI/CD pipeline improvements, 安全 processes]
 
### For Users/Deployers
[Secure deployment checklist, configuration hardening]

相關主題

• Supply Chain 安全 for AI -- AI 供應鏈 threat models
• Code Review for AI Applications -- AI-focused code review techniques
• Dependency 安全 -- Dependency analysis methods
• Container 安全 -- Container hardening for AI workloads