- Synacktiv's incident responders say an Amazon EKS investigation is largely decided in advance, by the logging and monitoring already in place.
- EKS control plane logs are off by default, and removing or restarting a pod erases most of the evidence inside it.
- Leaders should confirm which logs are on, how long they are kept, and whether responders can reach the nodes.
A security camera helps only if it was installed before the incident. Cloud clusters work the same way. When a Kubernetes cluster is compromised, what investigators can prove depends on what was already recording.
That is the argument in a new guide from Synacktiv, a French security firm. Its incident response team (CSIRT) says Kubernetes clusters now appear regularly in its cases. The guide covers Amazon EKS, AWS's managed Kubernetes service. Its central point is plain. In the firm's words, the outcome of an EKS investigation is "largely decided before the incident."
The default is silence
Synacktiv states that none of the EKS control plane logs are on by default. Five log types exist. Each cluster must enable them through the control plane logging API. They then flow into one CloudWatch log group per cluster, as separate streams.
The audit log is the most useful. Each entry names the account involved, the action taken and the resource it touched. It also shows the network address and the client software used.
For a new pod, the entry holds the full manifest. That includes the image, the command, environment variables and mounted volumes. Synacktiv calls this "a goldmine" for an analyst.
Container and host logs need a second step. The amazon-cloudwatch-observability add-on runs Fluent Bit, a log shipper, on each node. It sends data plane logs to three log groups. It also tags each entry with the pod and namespace. Responders then avoid searching node by node.
What suspicious activity looks like in the logs
The guide gives concrete patterns. A burst of SelfSubjectAccessReview entries shows an account listing its own privileges. The kubectl auth can-i --list command does the same thing.
Edits to access-control roles also matter. The escalate verb lets an account widen its own permissions.
Synacktiv calls pod creations rejected by Pod Security Admission (an HTTP 403 response) a reliable sign of an evasion attempt. A node account checking its own permissions is another flag. A legitimate kubelet does not do that.
Persistence leaves a trace too. A static pod is a workload the node's kubelet starts from a file on disk, outside the normal API workflow. In the audit log it appears as a pod created by a node identity. It carries the annotation kubernetes.io/config.source set to "file". Synacktiv calls that conclusive evidence of a pod placed directly on the node's disk.
Part of the evidence sits outside the cluster
Nodes and some pods carry an AWS identity. If someone misuses it, other AWS resources are affected.
Synacktiv says the node role is especially sensitive. Whoever holds it may be able to hand out the credentials that pods use on that cluster.
CloudTrail records this activity. A GuardDuty finding named UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS reports node-role credentials used from an outside IP address. A review limited to the cluster would miss it. The proof sits in AWS's own account logs.
Where the tools stop
GuardDuty is often the first alert. Synacktiv says its detection logic is hidden. Teams cannot read the rules or add their own.
GuardDuty's optional Runtime Monitoring watches system calls inside containers. It does not support EKS clusters on Fargate. Synacktiv treats a finding as a lead to confirm in the audit log, not a verdict.
Access matters as well. On standard clusters, responders can reach worker nodes over SSH or SSM. Under Auto Mode or Fargate, AWS manages the host. Direct node access is not available.
A last-resort command, kubectl debug node, gets around this. Synacktiv warns that it adds a workload to the node under examination. It also leaves new pod-creation entries in the audit log.
The guide also leaves out third-party runtime tools such as Falco, Sysdig and Wiz.
Questions to put to your cloud team
This guide describes one firm's method. It does not measure how many organisations leave these logs off. Still, the questions it raises are easy to ask.
Which of the five control plane log types are on for each cluster? How long are they kept? Is the Container Insights add-on installed? Is GuardDuty EKS Protection on? Do we run standard nodes, Auto Mode or Fargate, and can responders reach the nodes?
One more question concerns network records. Network flow logs carry no pod context. Teams therefore need a time-stamped inventory of which pod held which IP address. After a pod is gone, can someone match a network address to it?
Once a pod is removed or restarted, most of the evidence inside it is gone, Synacktiv says. Retention is therefore not paperwork. It decides whether a team can reconstruct an incident or only guess.
Forensics is not a service you call afterwards. It is a setting you choose beforehand.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Synacktiv.
