Lab: Containerized Model Breakout

Explore techniques for escaping from containerized AI applications to the host system, testing container isolation boundaries in ML deployment environments.

Prerequisites

• Docker and container security fundamentals
• Linux system administration skills
• Understanding of namespace isolation and cgroups
• Familiarity with GPU passthrough and NVIDIA container toolkit

pip install docker python-dotenv

Background

Container breakout in AI environments has unique attack vectors. GPU passthrough requires elevated privileges, model files are often mounted from host filesystems, and inference servers frequently run with more capabilities than necessary. These ML-specific requirements weaken the container isolation that security teams expect.

Lab Exercises

1. 1

   Deploy a Vulnerable AI Container

   Create intentionally misconfigured containers that mirror common AI deployment anti-patterns.

   vulnerable-compose.ymlyamlShow code

   version: "3.8"
   services:
     # Anti-pattern 1: Privileged container for GPU access
     ai-privileged:
       image: python:3.11-slim
       privileged: true
       volumes:
         - /var/run/docker.sock:/var/run/docker.sock
         - ./models:/models
       command: sleep infinity
    
     # Anti-pattern 2: Excessive volume mounts
     ai-overmounted:
       image: python:3.11-slim
       volumes:
         - /:/host:ro
         - ./models:/models
       command: sleep infinity
    
     # Anti-pattern 3: Host network + capabilities
     ai-hostnet:
       image: python:3.11-slim
       network_mode: host
       cap_add:
         - SYS_PTRACE
         - SYS_ADMIN
       command: sleep infinity
    
     # Hardened comparison baseline
     ai-hardened:
       image: python:3.11-slim
       read_only: true
       security_opt:
         - no-new-privileges:true
       tmpfs:
         - /tmp
       volumes:
         - ./models:/models:ro
       command: sleep infinity

   docker compose -f vulnerable-compose.yml up -d

2. 2

   Enumerate Container Escape Vectors

   From inside each container, identify potential escape paths.

   container_recon.pypythonShow code

   #!/usr/bin/env python3
   """Container reconnaissance: identify escape vectors from inside."""
    
   import subprocess
   import os
    
   def run(cmd: str) -> str:
       try:
           result = subprocess.run(cmd, shell=True, capture_output=True,
                                   text=True, timeout=5)
           return result.stdout.strip() or result.stderr.strip()
       except Exception as e:
           return f"Error: {e}"
    
   def enumerate_escape_vectors():
       findings = []
    
       # Check if running as root
       uid = os.getuid()
       if uid == 0:
           findings.append(("CRITICAL", "Running as root (uid=0)"))
    
       # Check for Docker socket
       if os.path.exists("/var/run/docker.sock"):
           findings.append(("CRITICAL", "Docker socket mounted - full host access possible"))
    
       # Check for host filesystem mounts
       mounts = run("cat /proc/mounts")
       if "/host" in mounts:
           findings.append(("CRITICAL", "Host filesystem mounted"))
    
       # Check capabilities
       caps = run("cat /proc/self/status | grep Cap")
       if caps:
           findings.append(("INFO", f"Capabilities: {caps[:200]}"))
    
       # Check for privileged mode indicators
       devices = run("ls /dev/ | wc -l")
       if devices and int(devices) > 20:
           findings.append(("HIGH", f"Many devices visible ({devices}) - possible privileged mode"))
    
       # Check cgroup escape potential
       cgroup = run("cat /proc/1/cgroup")
       if "docker" not in cgroup and "containerd" not in cgroup:
           findings.append(("HIGH", "May not be in container cgroup namespace"))
    
       # Check for GPU device access
       gpu_devices = run("ls /dev/nvidia* 2>/dev/null")
       if gpu_devices:
           findings.append(("MEDIUM", f"GPU devices accessible: {gpu_devices}"))
    
       # Check network namespace
       hostname = run("hostname")
       if len(hostname) < 12:  # Container hostnames are usually 12+ char hashes
           findings.append(("MEDIUM", f"Hostname '{hostname}' suggests host network"))
    
       # Check for sensitive host files via proc
       host_info = run("cat /proc/version")
       findings.append(("INFO", f"Kernel: {host_info[:100]}"))
    
       return findings
    
   if __name__ == "__main__":
       print("=== Container Escape Vector Enumeration ===\n")
       findings = enumerate_escape_vectors()
       for severity, desc in findings:
           print(f"  [{severity}] {desc}")

   Run this inside each container:

   # Test in privileged container
   docker exec ai-privileged python3 container_recon.py
    
   # Test in overmounted container
   docker exec ai-overmounted python3 container_recon.py
    
   # Test in hardened container
   docker exec ai-hardened python3 container_recon.py

3. 3

   Demonstrate Docker Socket Escape

   Use a mounted Docker socket to escape to the host.

   socket_escape.pypythonShow code

   #!/usr/bin/env python3
   """Docker socket escape demonstration.
    
   This demonstrates why mounting the Docker socket inside containers
   is equivalent to giving full host access."""
    
   import subprocess
   import json
    
   def check_docker_socket():
       """Verify Docker socket is accessible."""
       import os
       socket_path = "/var/run/docker.sock"
       if not os.path.exists(socket_path):
           print("Docker socket not mounted. This escape vector is not available.")
           return False
       print(f"Docker socket found at {socket_path}")
       return True
    
   def list_host_containers():
       """List all containers on the host via the socket."""
       # Install Docker CLI if needed
       subprocess.run("apt-get update && apt-get install -y curl",
                       shell=True, capture_output=True)
    
       # Use curl to query Docker API directly
       result = subprocess.run(
           'curl -s --unix-socket /var/run/docker.sock '
           'http://localhost/containers/json',
           shell=True, capture_output=True, text=True,
       )
       if result.stdout:
           containers = json.loads(result.stdout)
           print(f"\nHost containers visible: {len(containers)}")
           for c in containers:
               print(f"  {c['Names'][0]}: {c['Image']} ({c['State']})")
    
   def demonstrate_host_access():
       """Show how socket access enables host filesystem reading."""
       print("\n=== Docker Socket Escape ===")
       print("With Docker socket access, an attacker can:")
       print("1. List all host containers and their configurations")
       print("2. Create a new privileged container with host root mounted")
       print("3. Execute commands on the host via the new container")
       print("4. Access secrets, credentials, and other container data")
       print("\nThis is why Docker socket mounting is a CRITICAL misconfiguration.")
    
   if __name__ == "__main__":
       if check_docker_socket():
           list_host_containers()
       demonstrate_host_access()

   docker exec ai-privileged python3 socket_escape.py

4. 4

   Implement Container Hardening

   Apply and verify hardening measures that prevent common escape vectors.

   hardening_checklist.pypythonShow code

   #!/usr/bin/env python3
   """Container hardening verification checklist for AI deployments."""
    
   import os
   import subprocess
    
   def run(cmd):
       try:
           r = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=5)
           return r.stdout.strip()
       except Exception:
           return ""
    
   CHECKS = [
       {
           "name": "Non-root user",
           "check": lambda: os.getuid() != 0,
           "fix": "Use USER directive in Dockerfile or --user flag",
       },
       {
           "name": "Read-only filesystem",
           "check": lambda: not os.access("/", os.W_OK),
           "fix": "Use --read-only flag with tmpfs for writable directories",
       },
       {
           "name": "No Docker socket",
           "check": lambda: not os.path.exists("/var/run/docker.sock"),
           "fix": "Never mount Docker socket in application containers",
       },
       {
           "name": "No host filesystem",
           "check": lambda: not os.path.exists("/host"),
           "fix": "Mount only specific directories needed, not entire host",
       },
       {
           "name": "Limited devices",
           "check": lambda: len(os.listdir("/dev")) < 20,
           "fix": "Avoid --privileged; use --device for specific GPU devices only",
       },
       {
           "name": "No new privileges",
           "check": lambda: "NoNewPrivs:\t1" in run("cat /proc/self/status"),
           "fix": "Use --security-opt no-new-privileges:true",
       },
       {
           "name": "Seccomp enabled",
           "check": lambda: "Seccomp:\t2" in run("cat /proc/self/status"),
           "fix": "Use default or custom seccomp profile",
       },
   ]
    
   if __name__ == "__main__":
       print("=== AI Container Hardening Checklist ===\n")
       passed = 0
       for check in CHECKS:
           try:
               result = check["check"]()
           except Exception:
               result = False
           status = "PASS" if result else "FAIL"
           passed += int(result)
           print(f"  [{status}] {check['name']}")
           if not result:
               print(f"         Fix: {check['fix']}")
    
       print(f"\nScore: {passed}/{len(CHECKS)}")

   # Compare hardened vs vulnerable
   docker exec ai-hardened python3 hardening_checklist.py
   docker exec ai-privileged python3 hardening_checklist.py

Troubleshooting

Related Topics

• Inference Server Exploitation - Attacking the services running inside containers
• GPU Side-Channel - Hardware-level attacks through GPU access
• Model Serving Attacks - Broader serving framework vulnerabilities
• Full-Stack Exploit - Chaining container escape with other vectors

References

• "Understanding Docker Container Security" - Docker Inc. (2024) - Container security architecture and best practices
• "Container Escape Techniques" - MITRE ATT&CK (2024) - Catalog of known container escape methods
• "Securing GPU Workloads in Containers" - NVIDIA (2024) - GPU-specific container security guidance
• "Threat Modeling ML Infrastructure" - Kumar et al. (2024) - ML-specific infrastructure threat modeling