Incident Response for Container Breaches: Playbooks That Actually Work

Step 2: Snapshot pod state before termination

# Capture pod YAML (includes labels, annotations, service account)
kubectl get pod  -n  -o yaml > forensic-pod-snapshot.yaml

# Extract container logs before they're lost
kubectl logs  -n  --tail=10000 > forensic-container-logs.txt

# If the container is still running, capture /proc and /sys
kubectl exec -it  -n  -- tar czf /tmp/proc-snapshot.tar.gz /proc/1/root /etc /var/log 2>/dev/null
kubectl cp /:/tmp/proc-snapshot.tar.gz ./proc-snapshot.tar.gz

Step 3: Scale down, don't delete

# Scale the deployment to 0 (preserves ReplicaSet for investigation)
kubectl scale deployment  -n  --replicas=0

# Or for DaemonSets
kubectl rollout pause daemonset  -n 

Phase 3: Forensic Collection — Playbook

Container forensics is a race against the orchestrator. The CIS Benchmark for Docker and Kubernetes recommends capturing these artifacts before any pod termination:

Container image forensics

# Save the running container as an image for offline analysis
docker commit  forensic-image:
docker save forensic-image: -o forensic-image-.tar

# Inspect image layers for tampered files
docker history --no-trunc forensic-image:
docker run --rm -it forensic-image: sh -c "find /etc -name '*.sh' -newer /etc/hostname -type f"

Runtime artifact capture

# Capture process list before the container terminates
kubectl exec -it  -n  -- ps aux > forensic-processes.txt

# Network connections from inside the container
kubectl exec -it  -n  -- ss -tulpn > forensic-network.txt

# Check for cron jobs, systemd timers, or modified binaries
kubectl exec -it  -n  -- bash -c "
  crontab -l 2>/dev/null
  ls -la /etc/cron* 2>/dev/null
  find /usr/bin /usr/local/bin -mmin -60 -type f 2>/dev/null
" > forensic-persistence.txt

Cluster-level forensics

# Capture events in the namespace for the past hour
kubectl get events -n  --sort-by=.lastTimestamp > forensic-events.txt

# Check for suspicious ConfigMap or Secret mounts
kubectl describe pod  -n  | grep -A5 "Mounts\|Volumes"

# Examine audit logs (if enabled)
kubectl logs -n kube-system -l component=kube-apiserver --since=1h > forensic-audit.log

Phase 4: Eradication and Recovery — Playbook

Once forensic data is secured, the recovery phase begins. The goal is not just to remove the attacker but to close the gap that enabled the breach.

Rebuild from trusted sources

• Never patch a compromised container.Rebuild from a verified base image with a new digest. Use your CI/CD pipeline to generate a fresh build with all security patches applied.
• Rotate ALL secrets.Every secret the compromised pod had access to is now attacker-known. Usekubectl create secret generic to regenerate, or integrate with Docker secrets managementand external vaults.
• Revoke and rotate service accounts.Delete the compromised service account and create a new one with scoped permissions. Apply least privilege using theKubernetes RBAC guide.

Image trust verification

# Verify image signatures with Cosign before redeploying
cosign verify /: --key 

# Check for known vulnerabilities in the replacement image
trivy image --severity CRITICAL,HIGH --ignore-unfixed /:

Real-World Breach Scenario: Step by Step

Let's simulate a realistic attack and walk through the response using the playbooks above.

The setup:A finance application running on Kubernetes with 6 microservices. A developer accidentally left a debug endpoint enabled in the payment-service container. An attacker discovered it through Shodan scanning and exploited an RCE vulnerability in the Node.js runtime.

This scenario mirrors theCIS Kubernetes Benchmarkrecommendations for incident readiness: have playbooks pre-written, test them in chaos engineering drills, and automate as much of the response as possible using admission controllers and runtime policies.

Incident Response Readiness Checklist

Use this checklist to assess your team's container IR readiness:

1. ⬜ Runtime detection deployed— Falco, Tracee, or eBPF-based tool monitoring all namespaces
2. ⬜ IR playbooks container-native— Updated in the last 90 days, tested in a staging cluster
3. ⬜ Kubernetes audit logging enabled— API server audit logs sent to SIEM with retention ≥ 90 days
4. ⬜ Network policies enforced— Default-deny ingress and egress on all namespaces
5. ⬜ Image signing pipeline in place— Cosign or Notary signing every production image
6. ⬜ Secrets externalized— No Secrets in Git; Vault or cloud KMS for all sensitive data
7. ⬜ Forensic toolkit staged— Pre-approved tool images for DFIR, accessible from the cluster
8. ⬜ Service account hygiene— No cluster-admin bindings on namespaced service accounts
9. ⬜ Immutable container strategy— No writeable filesystems in production;readOnlyRootFilesystem: true
10. ⬜ WAR room ready— Communication channels, runbooks, and escalation matrix pre-defined for container incidents

Related ShieldOps Reads

• Container Runtime Security: A Complete Guide to Falco, Seccomp, and AppArmor— Deploy the runtime detection tools that power Phase 1 of your IR playbook
• Security Chaos Engineering: Breaking Containers to Make Them Stronger— Test your IR playbooks before a real incident hits
• Kubernetes RBAC Deep Dive: Least Privilege Access Control Patterns— Prevent service account abuse that enables lateral movement
• Secrets Detection: 10 Critical Mistakes That Leak Credentials— Stop credential leaks before they lead to breaches
• ShieldOps Security Platform Overview— Scan your container images for vulnerabilities and misconfigurations
• Compliance Automation with ShieldOps— Map your incident response program to PCI-DSS and SOC 2 requirements

Frequently Asked Questions

What should I do first when I detect a container breach?

Apply network isolation immediately using a default-deny NetworkPolicy. Then capture forensic data (logs, process list, network connections) before terminating the pod. Premature pod deletion is the most common and costly IR mistake in container environments.

How is container forensics different from traditional forensics?

Containers are ephemeral—they can be terminated and rescheduled automatically, destroying all volatile evidence. You do not have hours to acquire a disk image. Forensic collection must happen while the container is still running, targeting memory, /proc, logs, and network state. The NIST SP 800-86 guide to computer forensics was updated in 2024 to include cloud-native artifact categories.

Can I use my existing SIEM for container incident detection?

Most SIEMs can ingest Kubernetes audit logs and container runtime alerts, but you need container-specific rules. Traditional rules based on user login times or SSH sessions do not apply. Focus on API server audit events, pod creation anomalies, and network flow logs from your CNI plugin (Cilium, Calico).

Should I patch the compromised container or rebuild it?

Never patch a compromised container. You cannot trust that the attacker has not modified system binaries, libraries, or configuration files. Always rebuild from a verified, signed base image with a known-good digest. Use your CI/CD pipeline to automate the rebuild and deploy the new image as a rolling update.

How often should we rehearse container incident response drills?

Run a tabletop exercise quarterly and a full technical drill (in a staging cluster) every six months. Use chaos engineering tools like LitmusChaos or Chaos Mesh to simulate realistic attack scenarios. The CIS Controls v8 recommends at least two full-scale incident response exercises per year for critical systems.

What compliance standards require container incident response plans?

PCI-DSS v4.0 Requirement 12.10, SOC 2 CC7, NIST SP 800-53 IR-4, and ISO 27001 A.16 all mandate incident response capabilities. For containerized environments specifically, theCIS Kubernetes Benchmarkincludes IR readiness checks, and NIST SP 800-190 provides container-specific incident response guidance.

Conclusion

Container breaches are inevitable—the question is whether your team will respond in minutes or days. By adopting container-native playbooks, pre-staging forensic tools, and automating detection with runtime security platforms, you cut containment time from hours to minutes.

Don't wait for a breach to build your playbook.Start scanning your container images with ShieldOps for free—identify vulnerabilities and misconfigurations before attackers do, and build the security posture that makes incident response manageable.