Agent Foskett Case File • Azure Kubernetes Service • Defender for Containers • KQL
The Container Was Rebuilt — But the Suspicious Behaviour Returned
The team deleted a suspicious pod in Azure Kubernetes Service.
Kubernetes immediately created another one.
Minutes later, the same unexpected outbound connection appeared.
“We rebuilt the container, Agent Foskett.”
“Excellent. Did you remove the problem?”
“We removed the pod.”
Agent Foskett looked at the deployment manifest. “Ah. You deleted the actor. The script is still running.” ☕
Case file: The behaviour that survived the rebuild
A fictional investigation into AKS pod recreation, Kubernetes controllers, container image integrity, runtime detection and workload identity.
✓ Reconstruct pod and deployment lineage
✓ Correlate runtime alerts with container logs
✓ Identify the persistence mechanism before remediation
The alert came back
At 02:14 UTC, Microsoft Defender for Containers raised a runtime alert about unexpected process execution in a production AKS workload. The pod also appeared to initiate outbound traffic to a destination not used by the application.
An operator deleted the affected pod. The deployment controller restored the desired replica count, and the replacement pod initially looked healthy. At 02:27 UTC, similar runtime behaviour was reported again.
This is a fictional training case. Times, names, indicators and findings are illustrative. Real alert names, log fields and telemetry coverage vary by enabled Defender plans, AKS configuration, agent versions and data collection settings.
An operator deleted the affected pod. The deployment controller restored the desired replica count, and the replacement pod initially looked healthy. At 02:27 UTC, similar runtime behaviour was reported again.
This is a fictional training case. Times, names, indicators and findings are illustrative. Real alert names, log fields and telemetry coverage vary by enabled Defender plans, AKS configuration, agent versions and data collection settings.
02:14 UTCA runtime alert identifies suspicious process activity in an AKS pod.
02:19 UTCThe operator deletes the pod. Its controller immediately creates a replacement.
02:27 UTCA similar alert appears against the replacement pod. The original cause remains unresolved.
A pod is disposable. Its configuration is not
In Kubernetes, deleting a pod managed by a Deployment or ReplicaSet does not remove the controller, image reference, startup command, environment variables, mounted volumes or service account settings. The controller attempts to recreate the workload from its declared specification.
A recurring alert could reflect a compromised image, a malicious or misconfigured startup script, a mounted persistent volume, an injected sidecar, an abused init container, a compromised application dependency or an external actor repeatedly exploiting the workload. Recurrence alone does not identify which explanation is correct.
A recurring alert could reflect a compromised image, a malicious or misconfigured startup script, a mounted persistent volume, an injected sidecar, an abused init container, a compromised application dependency or an external actor repeatedly exploiting the workload. Recurrence alone does not identify which explanation is correct.
Preserve evidence before deleting anything else
Record the AKS cluster, namespace, workload name, pod UID, container ID, image digest, node, Kubernetes service account, workload identity annotations and exact UTC alert times. Capture the current Deployment, ReplicaSet, pod specification and relevant events using your approved incident-response process. Preserve runtime alerts and container logs before log rotation or further pod replacement removes short-lived evidence.
Avoid relying on a pod name alone: replacement pods often have different names and UIDs. Compare owner references and immutable image digests. A mutable tag such as
Avoid relying on a pod name alone: replacement pods often have different names and UIDs. Compare owner references and immutable image digests. A mutable tag such as
latest cannot establish that two pods ran the same image bytes.KQL: Identify replacement pods and their controller lineage
In a Log Analytics workspace collecting Azure Monitor Container Insights inventory,
KubePodInventory can help reconstruct pod transitions. Column availability and ingestion frequency depend on the environment. This query looks at one namespace and pod-name prefix; confirm the prefix and time range before running. Inventory is sampled state, not a complete Kubernetes API audit trail.- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
// Log Analytics / Sentinel: requires KubePodInventory ingestion
let lookback = 24h;
let targetNamespace = "production";
let podPrefix = "orders-api-";
KubePodInventory
| where TimeGenerated > ago(lookback)
| where Namespace == targetNamespace
| where Name startswith podPrefix
| summarize FirstSeen=min(TimeGenerated), LastSeen=max(TimeGenerated),
States=make_set(PodStatus, 10),
Nodes=make_set(Computer, 10)
by ClusterName, Namespace, Name, PodUid
| order by FirstSeen ascKQL: Look for suspicious container output
Container stdout and stderr can provide useful context around an alert, but they do not constitute a full process execution trace. In environments ingesting
Defender for Containers runtime alerts and, where deployed, endpoint/runtime sensor telemetry should be used to validate process and network claims.
ContainerLogV2, start with the affected namespace and a tight time window. Search terms below are triage hints, not proof of compromise: legitimate diagnostic tools and application messages can match them.Defender for Containers runtime alerts and, where deployed, endpoint/runtime sensor telemetry should be used to validate process and network claims.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
// Log Analytics / Sentinel: requires ContainerLogV2 ingestion
let startTime = datetime(2026-10-22T02:00:00Z);
let endTime = datetime(2026-10-22T03:00:00Z);
ContainerLogV2
| where TimeGenerated between (startTime .. endTime)
| where PodNamespace == "production"
| where PodName startswith "orders-api-"
| extend Message = tostring(LogMessage)
| where Message has_any ("curl", "wget", "sh", "token", "metadata")
| project TimeGenerated, PodNamespace, PodName,
ContainerName, LogSource, Message
| order by TimeGenerated ascKQL: Correlate Defender alerts with the affected workload
When Microsoft Defender for Cloud alerts are connected to Microsoft Sentinel, alert information may appear in
The example deliberately leaves the entity payload visible rather than inventing a universal Kubernetes schema.
SecurityAlert. The Entities payload varies by alert type; do not assume a stable pod-ID property. Start with the alert time and inspect the alert name, provider and raw entities, then correlate confirmed identifiers back to the cluster and workload.The example deliberately leaves the entity payload visible rather than inventing a universal Kubernetes schema.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
// Microsoft Sentinel: requires relevant SecurityAlert ingestion
let startTime = datetime(2026-10-22T02:00:00Z);
let endTime = datetime(2026-10-22T03:00:00Z);
SecurityAlert
| where TimeGenerated between (startTime .. endTime)
| where ProviderName has "Azure" or ProductName has "Defender"
| project TimeGenerated, AlertName, AlertSeverity,
ProviderName, ProductName, CompromisedEntity,
Entities, ExtendedProperties, SystemAlertId
| order by TimeGenerated ascWhat did the Kubernetes audit trail say?
The Kubernetes API audit trail is the best place to test whether someone changed a Deployment, created a new pod, patched a service account or modified a workload specification. On AKS, audit records can be sent to Log Analytics through diagnostic settings, but they are not automatically present in every workspace.
Check the configured destination and table format first. Resource-specific tables and the legacy
Check the configured destination and table format first. Resource-specific tables and the legacy
AzureDiagnostics table have different schemas. In the AKS control plane, inspect create, update, patch and delete events for the Deployment, ReplicaSet, ConfigMaps, Secrets and service accounts. Compare the event identity, request source, timestamp and actual spec changes. A successful API call does not, by itself, prove malicious intent.The clue was in the Deployment, not the replacement pod
The team compared the live Deployment specification with its last approved GitOps commit. The image digest matched the approved release, but a startup command had been added to the Deployment template. The command launched a helper process before the application started. That helper generated the same suspicious activity each time Kubernetes recreated the pod.
The fictional audit trail showed that the Deployment template had been patched through an overprivileged automation identity shortly before the first alert. The replacement pod was faithfully executing the modified specification. The investigation then focused on how that identity was used and whether its permissions extended beyond this namespace.
The fictional audit trail showed that the Deployment template had been patched through an overprivileged automation identity shortly before the first alert. The replacement pod was faithfully executing the modified specification. The investigation then focused on how that identity was used and whether its permissions extended beyond this namespace.
Was workload identity involved?
The pod used a Kubernetes service account. Investigators checked whether that service account was mapped to a Microsoft Entra workload identity and whether the application could request Azure access tokens. The presence of a projected service account token or federated identity credential is not inherently suspicious; it may be required for normal application access.
In this case, no evidence showed that an Azure token had been used to access another resource. The confirmed issue was unauthorised modification of the Kubernetes Deployment via an automation identity with excessive Kubernetes permissions. The team treated potential Azure identity abuse as a separate hypothesis until corroborating sign-in and resource-access evidence could be collected.
In this case, no evidence showed that an Azure token had been used to access another resource. The confirmed issue was unauthorised modification of the Kubernetes Deployment via an automation identity with excessive Kubernetes permissions. The team treated potential Azure identity abuse as a separate hypothesis until corroborating sign-in and resource-access evidence could be collected.
A six-stage evidence timeline
This timeline is an illustrative reconstruction. The important technique is to align runtime alerts, controller actions, API changes, image digests and identity evidence on a common UTC clock.
01:58 UTC — Configuration changeAn automation identity patches the Deployment startup command.
02:14 UTC — Runtime alertDefender detects suspicious process behaviour in the existing pod.
02:19 UTC — Pod deletedAn operator removes the pod without changing the Deployment template.
02:20 UTC — Pod recreatedThe ReplicaSet starts a replacement using the same modified specification.
02:27 UTC — Alert returnsThe helper process runs again and triggers similar detection.
03:06 UTC — Root cause isolatedDeployment diff and audit records connect the repeated activity to the unauthorised template change.
Containment: fix the source of truth
Contain the affected workload according to business criticality and the incident-response plan. Preserve manifests, audit records, container evidence and alert details. Revoke or restrict the automation identity if its use is unauthorised, rotate exposed credentials where indicated, and review its cluster roles and bindings.
Restore the Deployment from a verified approved manifest, pin and validate the intended image digest, inspect referenced ConfigMaps, Secrets, init containers and volumes, and redeploy through a trusted pipeline. Monitor the replacement workload for the original indicators and new unexpected behaviour. Deleting pods repeatedly is not remediation if the controlling specification remains compromised.
Restore the Deployment from a verified approved manifest, pin and validate the intended image digest, inspect referenced ConfigMaps, Secrets, init containers and volumes, and redeploy through a trusted pipeline. Monitor the replacement workload for the original indicators and new unexpected behaviour. Deleting pods repeatedly is not remediation if the controlling specification remains compromised.
Defensive lessons for AKS teams
Use GitOps or another controlled deployment process with peer review and drift detection. Limit who can patch workloads, especially in production namespaces. Apply Kubernetes RBAC least privilege, admission policies, image signing or verification where appropriate, and Defender for Containers runtime monitoring. Preserve Kubernetes audit logs and verify that alerts, inventory and container logs are actually ingested before an incident.
Separate the evidence sources: inventory shows what existed, audit logs show control-plane changes, container logs show application output, and runtime sensors may show process or network activity. No single data source tells the whole story.
Separate the evidence sources: inventory shows what existed, audit logs show control-plane changes, container logs show application output, and runtime sensors may show process or network activity. No single data source tells the whole story.
Agent Foskett closes the case
“We rebuilt it twice, Agent Foskett.”
“And Kubernetes rebuilt it exactly as instructed.”
“So the container wasn’t ignoring us?”
“No. It was obeying the wrong instructions.”
Agent Foskett closed the incident report.
“Next time, investigate the recipe before ordering another cake.” ☕😂
The Logs Already Knew!
“And Kubernetes rebuilt it exactly as instructed.”
“So the container wasn’t ignoring us?”
“No. It was obeying the wrong instructions.”
Agent Foskett closed the incident report.
“Next time, investigate the recipe before ordering another cake.” ☕😂
The Logs Already Knew!
Related investigations
The Managed Identity Had No Password — So How Was It Misused?Investigate Azure workload permissions and unexpected resource access.The Device Was Compliant Yesterday — Today It Wasn't Even ManagedFollow device identity, compliance and Conditional Access evidence.The Certificate Was Valid — But Nobody Knew Who Was Using ItExamine workload identity credentials, permissions and service principal sign-ins.
