Kubernetes Debugging Secrets: 7 kubectl Commands Security Engineers Need

A single misconfigured pod in your Kubernetes cluster can expose databases, leak secrets, or provide an entry point for ransomware — yet most security engineers still debug blind. Kubectl isn't just a deployment tool; it's your primary forensic instrument when things go wrong in production.

Security incidents in Kubernetes rarely announce themselves with clear error messages. A pod that keeps crashing could be a resource constraint — or an active exploit consuming memory. A spike in API calls could be normal traffic — or a credential-stuffing attack. The difference is knowing which kubectl commands to run and how to interpret their output.

This guide covers 7 kubectl commands every security engineer needs for real-time debugging, forensic analysis, and threat detection in production Kubernetes clusters.

Why Kubectl Debugging Skills Matter for Security

Kubernetes hides complexity behind abstractions — Deployments, Services, Ingresses. But when something goes wrong, those abstractions become obstacles. You can'ttop a virtual machine or tcpdump a Service. You need kubectl.

Theofficial kubectl referencelists over 70 commands, but only a handful are critical for security work. The seven commands in this guide cover the full spectrum of security debugging: from runtime inspection to RBAC auditing to resource anomaly detection.

Security engineers who master these commands can:

• Detect active breacheswithin minutes of initial symptoms
• Forensic reconstructwhat happened before a crash or restart
• Audit permissionsquickly without external tools
• Identify resource abuselike crypto miners or data exfiltration

1. kubectl describe Pod — Deep Pod Introspection

Thekubectl describe command is the first stop for any pod-related issue. It surfaces events, conditions, container states, and restart counts — all critical signals for security analysis.

# Get detailed info about a specific pod
kubectl describe pod  -n 

# Describe all pods in a namespace to find anomalies
kubectl describe pods -n production | grep -A5 "State:.*Waiting\|State:.*Terminated\|Last State"

Security signals in describe output:

• Restart count > 5 in 1 hour— possible crash-loop exploit or OOM from resource exhaustion attack
• ImagePullBackOff or ErrImagePull— could indicate a compromised registry or tag mutation in supply chain
• Container status Waiting with CrashLoopBackOff— the pod started and died repeatedly; check previous logs
• Conditions with Ready=False and Reason=NodeAffinity— possible node isolation by attacker

The fix:Always checkkubectl describe pod before restarting or deleting a failing pod. The output contains the breadcrumb trail you need for root cause analysis.

2. kubectl logs --previous — Crash Forensics

When a pod crashes and restarts, its current logs only show the new session. The--previous flag retrieves logs from the terminated container — essential for understanding what caused the crash.

# Get logs from the previous (crashed) container instance
kubectl logs  --previous -n 

# Follow logs from all containers in a deployment
kubectl logs deployment/ --all-containers --since=2h -n 

Security signals in logs:

• Unexpected outbound connectionsin previous logs before crash — data exfiltration
• Segfaults or SIGKILL signals— possible exploit payload execution
• Repeated authentication failuresbefore a pod restarts — credential stuffing against a service
• Stack traces with suspicious imports— modified library or injected code

The fix:Makekubectl logs --previous the second command in any crash investigation (after describe). Pipe it to a file for forensic preservation: kubectl logs ... --previous > crash-forensics-$(date +%s).log.

3. kubectl exec — Runtime Container Inspection

Sometimes you need to get inside a running container to see what's actually happening.kubectl exec gives you a shell into any running container — the digital equivalent of a white-glove inspection.

# Get an interactive shell in a container
kubectl exec -it  -n  -- /bin/sh

# Run a single command inside a container without a shell
kubectl exec  -n  -- ps aux

# Check for unexpected processes
kubectl exec  -n  -- ls -la /proc/*/exe 2>/dev/null | head -20

Security signals from exec inspection:

• Unexpected processes— crypto miners (e.g.,xmrrig, xmrig), reverse shells (/dev/tcp), or port scanners
• Modified binaries— check/usr/bin, /usr/local/bin for files with suspicious modification dates
• Odd network connections—ss -tupn or netstat -tupn for connections to unknown IPs
• .ssh or .kube directoriesin unexpected places — credential harvesting
• Scheduled jobs (cron, anacron)inside containers — persistence mechanism

⚠️ Security note:kubectl exec itself is a privilege. Audit who has pods/exec access via RBAC. The command can be monitored by enabling audit logging on the API server. See the Kubernetes cluster debugging guidefor audit policy setup.

The fix:Restrictpods/exec to a dedicated security team role. Use kubectl auth can-i (command #4) to verify your RBAC posture.

4. kubectl auth can-i — RBAC Verification

Before you can debug, you need to know what you're allowed to do — and more importantly, what your service accounts can do.kubectl auth can-i checks whether an identity has a specific permission, both subjectively and directly against the authorization layer.

# Check if your current user can list secrets
kubectl auth can-i list secrets -n production

# List all permissions for the current user
kubectl auth can-i --list

# Check what a specific service account can do
kubectl auth can-i --list --as=system:serviceaccount:production:my-app

# Verify a specific verb-resource combination
kubectl auth can-i create pods/exec
kubectl auth can-i delete pods

Security signals from auth can-i:

• Wildcard access—* on any resource means excessive privilege
• Secrets access— service accounts shouldn't haveget secrets unless necessary
• Pod creation/deletionin namespaces where the account shouldn't deploy
• ClusterRole bindings— unintended cluster-wide access

The fix:Runkubectl auth can-i --list for every service account in your cluster quarterly. Integrate it into your IAM audit workflow. For a deeper dive into RBAC patterns, read Kubernetes RBAC Deep Dive: Least Privilege Access Control Patterns.

5. kubectl get events — Security Event Correlation

The Kubernetes events API records every significant state change in the cluster — pod creations, failures, scheduling decisions, and more. When investigated as a timeline, events reveal attack patterns.

# Get all events sorted chronologically
kubectl get events --sort-by=.lastTimestamp -A

# Events in the last hour for a specific namespace
kubectl get events -n production --since=1h

# Events related to a specific pod
kubectl get events --field-selector involvedObject.name= -A

# Watch events in real time
kubectl get events -A --watch

Security signals from events:

• Rapid pod creation/deletion in the same namespace— possible container escape attempts
• BackOff events across multiple pods simultaneously— widespread issue, possible DoS
• NodeNotReady events followed by pod migration— possible node compromise
• FailedMount events on secrets volumes— unauthorized secret access attempts
• Multiple FailedAttachVolume events— attempted persistent volume tampering

The fix:Set up automated event collection. Pipekubectl get events --sort-by=.lastTimestamp -A into your SIEM or logging pipeline. For compliance environments, this satisfies CIS benchmark recommendations for Kubernetes event auditing.

6. kubectl top pod — Resource Anomaly Detection

Sudden changes in resource consumption are strong indicators of compromise. Thekubectl top command shows real-time CPU and memory usage per pod and per node.

# Show resource usage for all pods
kubectl top pods -A

# Sort by memory usage to find anomalies
kubectl top pods -A --sort-by=memory

# Node-level resource overview
kubectl top nodes

Security signals from top output:

• CPU > 200% of request— possible crypto mining (miners are CPU-intensive)
• Memory growing linearly over time— memory leak from code execution exploit
• A pod using 10x the memory of peers— data exfiltration buffer or in-memory database extraction
• Sudden CPU spike with no deployment change— compromise payload execution
• Node memory > 85%— OOM risk that attackers could trigger for denail of service

The fix:Set up resource quotas and limit ranges on all namespaces. Configure HPA (Horizontal Pod Autoscaler) with security-aware thresholds. For production monitoring, comparekubectl top pods output against your baseline resource profiles.

7. kubectl get --raw /metrics — API Server Telemetry

The Kubernetes API server exposes raw Prometheus metrics that reveal cluster health and security posture at the control plane level. This is the most powerful command in a security engineer's arsenal — and the most overlooked.

# Get raw metrics from the API server
kubectl get --raw /metrics | head -100

# Filter for API request metrics
kubectl get --raw /metrics | grep "apiserver_request_"

# Check etcd latency (critical for performance forensics)
kubectl get --raw /metrics | grep "etcd_request_duration"

# Watch for 401/403 response codes
kubectl get --raw /metrics | grep "apiserver_request_total.*code=\"4[0-9][0-9]\""

Security signals from metrics:

• Spike in 401 (Unauthorized) responses— possible credential-stuffing attack
• Spike in 403 (Forbidden) responses— someone is probing permissions
• Unusual request rate to specific resources— data enumeration or reconnaissance
• Watch request count growing— someone may be monitoring changes for data exfiltration
• etcd latency spikes— possible DoS against the control plane

The fix:Regularly export/metrics to a monitoring stack (Prometheus + Grafana). Set up alerting rules for abnormal API request patterns. For sensitive clusters, consider enabling Pod Security Standardsat therestricted level.

Quick Comparison: Krew Plugin Extensions

While vanilla kubectl covers the 7 commands above, thekubectl plugin ecosystem (Krew)extends debugging capability further. Here are security-focused plugins worth installing:

Real-World Example: Detecting a Crypto Miner with kubectl

The Scenario:A DevOps engineer notices a pod in thestaging namespace has been restarting every 15 minutes. The service it runs is not critical, so they're about to delete it and move on.

The Security Response:

1. kubectl describe pod mining-suspicious-pod -n staging — Shows restart count of 87 in 4 hours and OOMKilled as the last reason.
2. kubectl logs mining-suspicious-pod --previous -n staging — Contains references to stratum+tcp:// (a mining pool protocol).
3. kubectl exec mining-suspicious-pod -n staging -- ps aux — Reveals ./xmrrig running with high CPU affinity.
4. kubectl top pods -n staging --sort-by=cpu — Confirms the pod is using 450% of its CPU request.

Result:The team identifies a crypto-mining container that entered the cluster through a compromised CI/CD pipeline. The miner kept getting OOMKilled because it ran inside a container with a 256MB memory limit, causing the repeated crash-loops that triggered the investigation.

For more on securing your CI/CD against these types of attacks, seeCI/CD Pipeline Security: 15 Best Practices.

7-Step Security Debugging Checklist

• ⬜Describe first— Runkubectl describe pod before any other command
• ⬜Check previous logs— Always retrievekubectl logs --previous for crashed pods
• ⬜Inspect runtime— Exec into the container and check processes, network, file system
• ⬜Audit permissions— Verify RBAC withkubectl auth can-i --list
• ⬜Correlate events— Sortkubectl get events chronologically and look for patterns
• ⬜Check resource usage— Runkubectl top pods sorted by CPU and memory
• ⬜Pull API metrics— Querykubectl get --raw /metrics for control plane anomalies

Frequently Asked Questions

Can kubectl debug commands be used without cluster-admin access?

Most debugging commands work with namespace-scoped access.kubectl describe pod, kubectl logs, and kubectl exec only need permissions on the specific namespace. Cluster-level commands like kubectl get nodes and kubectl get --raw /metrics require cluster-admin or equivalent ClusterRole bindings.

How can I log all kubectl exec commands for security auditing?

Enable the Kubernetes audit log with a policy that capturespods/exec subresource requests. Add this to your audit-policy.yaml: level: RequestResponse for resources: ["pods/exec"]. Forward audit logs to your SIEM or a dedicated logging pipeline.

What's the difference between kubectl logs --previous and kubectl logs -p?

They are the same.-p is the short form of --previous. Both retrieve logs from the terminated container instance. Use whichever you prefer — but use it consistently in your runbooks.

How often should I run kubectl auth can-i audits?

Quarterly manual audits for critical namespaces, plus automated audits in CI/CD pipelines. After any role or ClusterRole change, runkubectl auth can-i --list for affected service accounts. For production clusters, integrate RBAC auditing into your weekly security review.

Can kubectl metrics help detect zero-day exploits?

Indirectly, yes. While kubectl can't detect unknown CVEs, it can detect the behavioral side effects of an exploit — abnormal CPU/memory spikes, unexpected process execution, unusual API call patterns. Thekubectl top pods and kubectl get --raw /metrics commands are your best tools for behavioral detection.

Conclusion

Kubectl is not just a deployment tool — it's your primary incident response instrument for Kubernetes. The seven commands in this guide —describe, logs --previous, exec, auth can-i, get events, top pods, and get --raw /metrics — cover the complete spectrum of security debugging from runtime container inspection to control plane telemetry.

Master these commands, and you'll cut your mean time to detection (MTTD) from hours to minutes. Start by adding the 7-step checklist to your incident response runbook today.

Ready to go deeper?Create your ShieldOps accountto automatically scan your Kubernetes manifests for security misconfigurations. Try ourfree tier— no credit card required.

Related reads:Kubernetes RBAC Deep Dive·Kubernetes Network Policies·Container Runtime Security Guide·Kubernetes Supply Chain Security