Kubernetes Audit Logging: Detecting Intrusions Through API Server Trails

Every Kubernetes cluster leaves a digital fingerprint on every request that passes through its API server. Everykubectl apply, every pod creation, every Secret read — it is all recorded in the audit log. The problem? Most teams enable audit logging, archive the logs to object storage, and never inspect them until after a breach. By then, the audit trail is the only thing standing between your security team and complete blindess about what happened.

Kubernetes audit logging is arguably the single most underutilized security control in containerized environments. Thekube-apiserver generates a structured JSON record of every authenticated request — who made it, what they requested, the response status, the originating IP, and the user agent. When properly configured and actively monitored, this data transforms your cluster from an opaque black box into a fully observable, forensically auditable security boundary.

In this guide, you will learn how to configure Kubernetes audit logging from scratch, write effective audit policies, ship logs to analysis backends, and detect real intrusion patterns before they become full-blown breaches. We will also cover compliance requirements from CIS Benchmarks, PCI-DSS, and SOC 2, and show you how ShieldOps can help centralize your audit log analysis.

Why Kubernetes Audit Logging Matters

The CNCF's 2024 Kubernetes security survey reported that 67% of organizations experienced a security incident in their Kubernetes environments, and in 43% of cases the breach went undetected for more than 24 hours. Without audit logging, you have no reliable way to answer the most critical questions after an incident: Who accessed what? When did it happen? Was the action authorized?

Kubernetes audit logs capture every interaction with the control plane:

• Authentication and authorization decisions— every RBAC allow or deny, with the exact role and binding that made the decision
• Resource creation, mutation, and deletion— every Pod, Deployment, Secret, ConfigMap, and custom resource that was modified
• Access to sensitive resources— Secret reads, ServiceAccount token requests, and certificate signing requests
• Admission control interactions— mutating and validating webhook calls, admission decisions, and policy evaluations
• Network and policy changes— NetworkPolicy modifications, PodSecurityPolicy (deprecated) and Pod Security Admission evaluations
• Impersonation attempts— any use of theImpersonate-* headers that may indicate privilege escalation

The key point: audit logging is not optional for production clusters. TheCIS Benchmark for Kubernetesmandates audit log configuration as a Level 1 control, and PCI-DSS Requirement 10 explicitly requires audit trails for all access to cardholder data environments.

How Kubernetes Audit Logging Works

The Kubernetes API server supports audit logging natively. When enabled, every request to the API server goes through a multi-stage pipeline: authentication, authorization, admission, and finally persistence. The audit system hooks into this pipeline at two points — before the request is processed (RequestReceived stage) and after the response is sent (ResponseComplete stage) — and records metadata about the request at a configurable level of detail.

Audit Policy Stages and Levels

An audit policy is a YAML document that tells the API server which events to log and at what level of detail. There are four audit levels, from least to most verbose:

• None— do not log matching events (used to suppress noisy but harmless events)
• Metadata— log request metadata only (user, verb, resource, response code) without the request or response body. Fast and low-volume.
• Request— log metadata plus the request body. Useful for capturing what was sent, but increases storage considerably.
• RequestResponse— log metadata, request body, and response body. Highest detail, highest volume. Use sparingly for sensitive operations.

The audit policy is defined as a list of rules evaluated in order. The first matching rule determines the level for that event. Every rule must specify a level and can optionally filter by resources, users, namespaces, or verbs.

# /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log everything at Metadata level by default
  - level: Metadata

  # Don't log requests to system endpoints
  - level: None
    users: ["system:kube-proxy", "system:kube-controller-manager", "system:kube-scheduler"]
    verbs: ["get", "list", "watch"]

  # Log secret reads and writes at RequestResponse level
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]

  # Log all pod creation mutations at Request level
  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: ""
        resources: ["pods"]

  # Log RBAC changes in detail
  - level: RequestResponse
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "clusterroles", "rolebindings", "clusterrolebindings"]
    verbs: ["create", "update", "patch", "delete"]

  # Log impersonation attempts at full detail
  - level: RequestResponse
    verbs: ["impersonate"]
    
  # Omit health check and readiness probe noise
  - level: None
    users: ["system:serviceaccount:kube-system:default"]
    resources:
      - group: ""
        resources: ["events"]
    verbs: ["get", "list", "watch"]

  # Log namespace modifications
  - level: Request
    resources:
      - group: ""
        resources: ["namespaces"]
    verbs: ["create", "delete"]

  # Log ConfigMap and ServiceAccount access (potential credential harvesting)
  - level: Request
    resources:
      - group: ""
        resources: ["configmaps", "serviceaccounts"]
    verbs: ["get", "list", "create", "update", "patch"]

Audit Log Backends

Kubernetes supports two built-in audit backends plus a webhook backend for custom integrations:

• Log backend— writes audit events to a file on the API server node. The default path is/var/log/kubernetes/audit.log. This is the simplest option and works with any log shipper (Fluentd, Filebeat, Vector).
• Webhook backend— sends audit events as HTTP POST requests to an external API. This is useful for real-time streaming of audit events to your SIEM or security analytics platform.
• Dynamic backends (alpha in 1.29)— allows configuration of multiple backends via a CRD, enabling different routing rules for different event types.

# API server flags for the log backend
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=10
--audit-log-maxsize=100

# API server flags for the webhook backend
--audit-webhook-config-file=/etc/kubernetes/audit-webhook.yaml
--audit-webhook-batch-max-size=100
--audit-webhook-batch-max-wait=5s

10 Intrusion Detection Patterns in Kubernetes Audit Logs

Now that your audit logging is configured and logs are flowing, what should you look for? Here are ten critical detection patterns that every Kubernetes security team should alert on:

1. Unauthorized Secret Read Attempts

A ServiceAccount or user reading a Secret in a namespace where they have noget permission returns a 403 Forbidden. Multiple 403s to Secrets from the same source within a short window strongly indicates credential harvesting activity. The audit log records the exact HTTP response code, so filtering on responseStatus.code=403 and objectRef.resource=secrets gives you immediate visibility.

The fix:alert on any 403 for Secret resources that exceeds 3 attempts in 5 minutes. Use a tool like Falco or a SIEM rule to correlate these events.

2. Privilege Escalation via RBAC Modification

Creating or modifying a ClusterRole or RoleBinding is one of the most sensitive operations in Kubernetes. An attacker who gains access to a cluster-admin token can grant themselves cluster-admin privileges. Audit events withverb=create or verb=update targeting roles, clusterroles, rolebindings, or clusterrolebindings must always trigger an alert.

The fix:set the audit policy toRequestResponse level for RBAC mutations. Alert on every RBAC modification outside of a pre-approved change window.

3. Suspicious ServiceAccount Token Usage

ServiceAccount tokens are a favorite target for attackers. If a ServiceAccount token created in namespace A is used to make API calls in namespace B, that is a strong indicator of token theft or misconfiguration. The audit log captures the namespace of the token in theuser.username field (system:serviceaccount::), which you can compare against the target namespace in the request.

The fix:deploy a policy engine like Kyverno or OPA Gatekeeper to enforce that ServiceAccounts can only be used in their home namespace. Alert on cross-namespace token usage in audit logs.

4. Cryptomining Workload Deployment

Attackers who compromise a cluster often deploy cryptomining workloads. These pods typically request significant CPU/GPU resources, use images from public registries withlatest tags, and are often created in namespaces where they do not belong. Look for verb=create events with resource=pods that deploy containers with resource requests exceeding reasonable thresholds.

The fix:enforce resource quotas at the namespace level and use admission controllers to validate all pod specifications against a allowlist of trusted image registries.

5. API Server Scanning and Reconnaissance

An attacker exploring your cluster will enumerate API resources. Look for patterns of repeatedlist and get calls across multiple resource types from a single source within a short time window. Tools like kubectl api-resources and kubectl get all generate a burst of audit events that stand out from normal traffic patterns.

The fix:set up baseline profiling of API call patterns per user and alert on significant deviations. The open-source projectkubernetes-audit-analyzercan help visualize these patterns.

6. Anonymous Request Execution

Kubernetes allows anonymous requests by default if not explicitly disabled via RBAC. An attacker can access your cluster's API without authentication if you have accidentally granted permissions to thesystem:anonymous user or system:unauthenticated group. Audit logs with user.username=system:anonymous or user.groups containing system:unauthenticated should be immediately investigated.

The fix:deny anonymous access by default in your RBAC configuration. Audit for any attempt from unauthenticated users.

7. Persistent Agent Traffic (Advanced Persistent Threats)

Advanced attackers maintain persistence by making regular, low-frequency API calls that mimic legitimate automation. Watch for ServiceAccounts that make API calls at regular intervals (e.g., every 60 seconds on the dot) — this is a hallmark of a scripted backdoor rather than a standard Kubernetes controller.

The fix:use audit log analytics to compute inter-request timing distributions for each ServiceAccount. Flag accounts with low-variance inter-request intervals.

8. Admission Controller Bypass Attempts

Attackers may try to bypass admission controllers by modifying webhook configurations or deleting validating/mutating webhook resources. Audit events forverb=delete or verb=update on validatingwebhookconfigurations or mutatingwebhookconfigurations are high-severity signals.

The fix:protect webhook configurations with RBAC that restricts modification to a dedicated security team. UseKubernetes admission controllersto enforce that webhook resources cannot be deleted without multi-party approval.

9. ConfigMap and Secret Bulk Reads

Reading all ConfigMaps or Secrets in a namespace (or cluster-wide) is a common post-exploitation step. Look forverb=list events on configmaps or secrets resources from a single user across multiple namespaces in rapid succession. This pattern suggests the attacker is collecting credentials and configuration data.

The fix:limit list permissions on Secrets and ConfigMaps to only the ServiceAccounts that genuinely need them. Use tools likePod Security Standardsto restrict access at the pod level.

10. Cluster Admin Impersonation

TheImpersonate-* HTTP headers allow a user to act as another user or ServiceAccount. While this is a legitimate feature for debugging and audit, it is frequently abused by attackers to escalate privileges. Every impersonation event in the audit log should be scrutinized. The log captures both the impersonator (user.username) and the impersonated target (impersonatedUser.username).

The fix:log all impersonation attempts atRequestResponse level and require a break-glass approval process for production clusters.

Real-World Incident: How Audit Logs Caught a Kubernetes Breach

In June 2025, a mid-stage fintech company detected a cryptomining incident in their production Kubernetes cluster. The initial alert came from a spike in CPU utilization, but the security team had no idea how the attacker gained access. By analyzing the audit logs, they reconstructed the entire attack chain:

1. Day 0, 03:14 UTC— A developer's personal access token was leaked to a public GitHub repository. The attacker cloned the repo within 2 minutes of the push.
2. Day 0, 03:17 UTC— The attacker used the leaked token to authenticate to the Kubernetes API server. The audit log recordeduser.username=developer@company.com with sourceIPs=45.33.32.156 (a known malicious IP range).
3. Day 0, 03:18-03:22 UTC— The attacker rankubectl get all -A four times, generating 142 audit events in 4 minutes. The reconnaissance pattern was visible in the audit log: sequential list calls for pods, services, deployments, secrets, configmaps, and nodes.
4. Day 0, 03:23 UTC— The attacker created a ClusterRoleBinding to grant themselves cluster-admin privileges. The audit log recorded the exact RBAC manifest atRequestResponse level.
5. Day 0, 03:24 UTC— A cryptomining pod was deployed in the kube-system namespace. The audit log captured the full pod specification.
6. Day 0, 03:31 UTC— The security team's SIEM alerted on the RBAC modification (a pre-existing audit rule). The cluster was locked down within 12 minutes.

The company was able to produce a complete forensic timeline because audit logging was configured atRequestResponse level for Secrets and RBAC operations, set to Request level for pod creation, and logs were shipped to their SIEM within 30 seconds. Without audit logging, they would have detected the cryptomining at the CPU spike (6 hours later) but never known how the attacker got in. The difference between a 12-minute containment and a 6-hour incident response was a properly configured audit policy.

Compliance and Audit Logging

Kubernetes audit logging is not just a security best practice — it is a compliance requirement across multiple frameworks:

Framework	Requirement	How K8s Audit Logging Satisfies It
CIS Benchmark for Kubernetes	1.2.5, 1.2.6, 1.2.7 (Level 1)	Audit logging must be enabled with an appropriate retention policy. The log backend satisfies the file-based audit requirement.
PCI-DSS v4.0	Requirement 10.2.1, 10.3.1	All access to cardholder data environments must be logged. Audit logs record user ID, event type, date/time, and success/failure.
SOC 2	CC6.1, CC7.2	Logical access controls must be monitored. Audit logs provide the monitoring data for access reviews.
NIST SP 800-53	AU-3, AU-6, AU-12	Audit record generation, analysis, and correlation requirements are met by properly configured K8s audit logging.

Storing audit logs is only half the equation — you also need to ensure they are tamper-proof, retained for the required period (typically 12 months for PCI-DSS, 6 months for SOC 2), and searchable for forensic analysis.ShieldOps Compliancehelps you map your Kubernetes audit configuration to these frameworks automatically.

Setting Up a Complete Audit Pipeline: Step-by-Step Checklist

Use this practical checklist to implement audit logging on your production clusters:

1. Write an audit policy— Start with the sample policy above. Tune the levels based on your risk profile. Log RBAC changes atRequestResponse, all other mutations at Request, and reads at Metadata.
2. Enable the API server flags— Add--audit-policy-file, --audit-log-path, and --audit-log-maxage to your API server configuration. For kubeadm clusters, update /etc/kubernetes/manifests/kube-apiserver.yaml.
3. Deploy a log shipper— Install Fluentd, Filebeat, or Vector as a DaemonSet on your cluster nodes. Configure it to tail/var/log/kubernetes/audit.log and forward to your log aggregation platform.
4. Set up a SIEM pipeline— Forward audit logs to a SIEM (Splunk, Elastic Security, Security Lake) or to a managed service. ShieldOps'sSecurity Analysisfeature ingests Kubernetes audit logs and detects the 10 patterns listed above.
5. Create alerting rules— Define alerts for the high-severity patterns: RBAC modifications, Secret reads with 403 responses, anonymous requests, and impersonation attempts. Set up a notification channel (PagerDuty, Slack, email).
6. Test the pipeline— Generate a test event by runningkubectl get secrets --as=system:anonymous (which will fail with 403) and verify the event appears in your SIEM within your defined latency SLA.
7. Define retention and rotation— Configure log rotation (10 files of 100 MB each on the API server) and set up archival to object storage (S3, GCS) with a 12-month retention policy for compliance.
8. Schedule periodic reviews— Review your audit policy quarterly against the latest CIS benchmark. Review SIEM alerts monthly and tune false positives.
9. Implement immutable logging— Ship logs to an append-only storage backend (e.g., S3 Object Lock or an immutable log service) to prevent tampering by an attacker who gains cluster-admin access.
10. Run tabletop exercises— Conduct quarterly incident response drills where your team responds to a simulated breach using only the audit log data. This validates both your detection rules and your team's forensic analysis skills.

Related ShieldOps Resources

• Kubernetes RBAC Deep Dive: Least Privilege Access Control Patterns— Understand the RBAC fundamentals that drive meaningful audit log analysis.
• Kubernetes Network Policies: Enforcing Zero-Trust at the Network Layer— Combine network isolation with audit monitoring for defense in depth.
• Kubernetes Supply Chain Security: From Git to Cluster With Sigstore— Verify that the workloads reaching your cluster are signed and trusted.
• Vulnerability Management Lifecycle: From CVE Discovery to Remediation— Integrate vulnerability scanning findings with audit log events for full visibility.
• Container Runtime Security: A Complete Guide to Falco, Seccomp, and AppArmor— Complement audit logging with runtime detection tools.

Frequently Asked Questions

Does Kubernetes enable audit logging by default?

No. Kubernetes audit logging is disabled by default in most distributions. You must explicitly enable it by passing--audit-policy-file and --audit-log-path flags to the API server. Managed Kubernetes services (EKS, AKS, GKE) offer audit logging as an optional feature that must be enabled per cluster.

How much storage do Kubernetes audit logs consume?

A production cluster with 50-200 nodes and a well-tuned audit policy typically generates 5-20 GB of audit logs per day. The exact volume depends on cluster activity and the audit level you configure. Setting common read-only operations (get, list, watch) to Metadata level significantly reduces volume.

What is the difference between Kubernetes audit logs and container runtime logs?

Kubernetes audit logs capture control plane activity — requests to the API server. Container runtime logs (collected by Falco, for example) capture kernel-level system calls made by container processes. Both are essential for comprehensive security monitoring. Audit logs tell you what API actions were performed; runtime logs tell you what actually happened inside the container.

Can attackers tamper with Kubernetes audit logs?

If an attacker gains cluster-admin access, they can delete or modify audit log files on the API server node. This is why immutable logging is critical — ship logs to an append-only external store as soon as they are written. The webhook audit backend sends events to an external API in real-time, making tampering significantly harder.

What is the webhook audit backend and when should I use it?

The webhook backend sends each audit event as an HTTP POST request to an external API endpoint. Use it when you need real-time audit log streaming to your SIEM or security platform. The log backend has a latency of up to a few seconds, while the webhook backend delivers events within milliseconds. For production security monitoring, use both backends in parallel — log for archival and webhook for real-time alerting.

How do I search Kubernetes audit logs for a specific user?

Audit logs are structured JSON. Usejq to filter: jq 'select(.user.username == "developer@company.com")' /var/log/kubernetes/audit.log. For large volumes, use a log analytics platform with indexed search. ShieldOps's AI Assistantcan answer natural-language questions about your audit log data.

Conclusion

Kubernetes audit logging is the foundation of cluster security observability. Without it, you are flying blind — unable to detect intrusions, investigate incidents, or demonstrate compliance. With a properly configured audit policy, a reliable log pipeline, and active alerting on the ten detection patterns covered in this guide, you can reduce your mean time to detection (MTTD) from hours to minutes.

Start by deploying the audit policy shown above on a non-production cluster, tune the levels based on your traffic patterns, and work your way up to production deployment. Your future self — during the next incident response — will thank you.

Ready to centralize your Kubernetes security monitoring?Sign up for ShieldOpsand get automated audit log analysis, real-time threat detection, and compliance reporting in one platform.View our plansto find the tier that fits your cluster size.