Agent Foskett Case File • Microsoft Sentinel • Automation Rules • SecurityIncident • SentinelAudit • SOAR • KQL
The Automation Rule Closed the Incident Before Anyone Investigated It
The incident appeared in Microsoft Sentinel.
A suspicious sign-in.
A privileged account.
Enough evidence to deserve investigation.
Minutes later, the incident was closed.
The resolution looked reassuring.
False Positive.
There was only one problem.
No analyst had investigated it.
Agent Foskett followed the incident history and found the decision had been made automatically.
The Incident Was Closed. The Investigation Never Started.
Automation can accelerate triage and response. If its conditions are too broad, it can also make a dangerous decision faster.
✓ Reconstruct the incident lifecycle
✓ Identify the automation rule
✓ Reopen the evidence
Closed did not mean investigated
The first clue was the timing.
The incident was created and closed within minutes.
There were no analyst notes explaining the evidence. No documented triage. No meaningful investigation trail.
Agent Foskett stopped treating the incident status as a conclusion and started treating it as another event on the timeline.
An incident status tells you what happened to the case. It does not prove what happened in the attack.
The incident was created and closed within minutes.
There were no analyst notes explaining the evidence. No documented triage. No meaningful investigation trail.
Agent Foskett stopped treating the incident status as a conclusion and started treating it as another event on the timeline.
An incident status tells you what happened to the case. It does not prove what happened in the attack.
Start with the incident record
Where Microsoft Sentinel incident data was available in Log Analytics, Agent Foskett used
The table can contain multiple records for the same incident as it is updated, making it useful for comparing creation, status and closure-related changes.
SecurityIncident to reconstruct the incident's recorded state over time.The table can contain multiple records for the same incident as it is updated, making it useful for comparing creation, status and closure-related changes.
Reconstruct the incident lifecycle
Agent Foskett filtered on the incident number and ordered the records chronologically. The objective was simple: determine when the incident appeared, when its status changed and what metadata accompanied that change.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
let TargetIncidentNumber = 4281;
SecurityIncident
| where TimeGenerated > ago(30d)
| where IncidentNumber == TargetIncidentNumber
| project TimeGenerated,
IncidentNumber,
Title,
Status,
Classification,
ClassificationReason,
ClassificationComment,
Owner,
ModifiedBy,
IncidentUrl
| order by TimeGenerated ascThe closure happened too quickly
The timeline showed the incident moving from active to closed almost immediately.
The classification said false positive, but the evidence supporting that classification was missing.
That did not prove the incident was malicious.
It proved the closure decision itself needed investigation.
A classification is a recorded decision. It is not a substitute for the evidence behind the decision.
The classification said false positive, but the evidence supporting that classification was missing.
That did not prove the incident was malicious.
It proved the closure decision itself needed investigation.
A classification is a recorded decision. It is not a substitute for the evidence behind the decision.
Automation rules can change incident state
Microsoft Sentinel automation rules can perform actions on incidents when configured trigger conditions are met. Those actions can include changing status, assigning owners, changing severity, adding tags and running playbooks.
That capability is powerful because repetitive triage work can happen immediately.
It also means automation becomes part of the security decision path.
If automation can close an incident, its conditions deserve the same scrutiny as detection logic.
That capability is powerful because repetitive triage work can happen immediately.
It also means automation becomes part of the security decision path.
If automation can close an incident, its conditions deserve the same scrutiny as detection logic.
Look for Sentinel incident and automation activity
Where Microsoft Sentinel audit telemetry was available, Agent Foskett searched
SentinelAudit around the incident's creation and closure window. He preserved the operation, resource, correlation and extended properties rather than assuming one field would explain the complete workflow.- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
SentinelAudit
| where TimeGenerated > ago(7d)
| where SentinelResourceType has_any
("Incidents", "Automation")
| project TimeGenerated,
OperationName,
Status,
SentinelResourceName,
SentinelResourceId,
SentinelResourceType,
SentinelResourceKind,
ExtendedProperties,
CorrelationId
| order by TimeGenerated ascFind the rule that matched
The organisation's automation configuration revealed a rule intended to suppress a known class of benign sign-in incidents.
The idea was reasonable.
The condition was not.
Instead of narrowly matching the known benign scenario, the rule relied on broad attributes shared by unrelated incidents.
When those conditions matched, the incident was automatically closed and classified.
The automation did exactly what it had been configured to do.
The idea was reasonable.
The condition was not.
Instead of narrowly matching the known benign scenario, the rule relied on broad attributes shared by unrelated incidents.
When those conditions matched, the incident was automatically closed and classified.
The automation did exactly what it had been configured to do.
The problem was the condition boundary
Agent Foskett compared the incident with the intended use case for the automation rule.
Some attributes matched.
The actual security context did not.
The account was privileged. The source IP was unusual. The sign-in deserved human review.
But the automation rule could not understand that wider context because nobody had encoded it into the conditions.
Automation can only reason over the conditions you give it.
Some attributes matched.
The actual security context did not.
The account was privileged. The source IP was unusual. The sign-in deserved human review.
But the automation rule could not understand that wider context because nobody had encoded it into the conditions.
Automation can only reason over the conditions you give it.
Find other incidents closed by the same pattern
One incorrectly closed incident raised a larger question: how many others had been processed the same way? Agent Foskett searched for recently closed incidents with the same classification and reviewed the time between creation and modification as a triage lead.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
SecurityIncident
| where TimeGenerated > ago(30d)
| where Status =~ "Closed"
| where Classification =~ "FalsePositive"
| summarize arg_max(TimeGenerated, *)
by IncidentNumber
| project IncidentNumber,
Title,
CreatedTime,
LastModifiedTime,
Classification,
ClassificationReason,
ClassificationComment,
ModifiedBy,
IncidentUrl
| order by LastModifiedTime descRapid closure was a lead — not proof of automation
Agent Foskett did not label every quickly closed incident as automatically mishandled.
An analyst can legitimately investigate and close a simple case quickly. Different workflows can also update incidents in different ways.
The query created a review set.
Each candidate still needed its own history and automation evidence.
Speed can identify cases worth reviewing. It does not establish why they were closed.
An analyst can legitimately investigate and close a simple case quickly. Different workflows can also update incidents in different ways.
The query created a review set.
Each candidate still needed its own history and automation evidence.
Speed can identify cases worth reviewing. It does not establish why they were closed.
Reopen the original evidence
The incident's false-positive classification had influenced how people perceived the case.
Agent Foskett ignored it and returned to the underlying security evidence.
The sign-in was reviewed on its own merits: account, source IP, location, authentication details, Conditional Access context and surrounding activity.
The automation result was removed from the reasoning until the evidence could independently support it.
Agent Foskett ignored it and returned to the underlying security evidence.
The sign-in was reviewed on its own merits: account, source IP, location, authentication details, Conditional Access context and surrounding activity.
The automation result was removed from the reasoning until the evidence could independently support it.
Return to the sign-in evidence
The exact evidence depends on the incident. In this case, Agent Foskett returned to
SigninLogs and reconstructed the activity around the user and source address instead of accepting the automated classification.- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
let TargetUser = "admin@contoso.com";
let StartTime = datetime(2026-10-04 01:00:00);
let EndTime = datetime(2026-10-04 04:00:00);
SigninLogs
| where TimeGenerated between (StartTime .. EndTime)
| where UserPrincipalName =~ TargetUser
| project TimeGenerated,
UserPrincipalName,
IPAddress,
AppDisplayName,
Location,
ResultType,
AuthenticationRequirement,
ConditionalAccessStatus,
CorrelationId
| order by TimeGenerated ascAutomation had changed the SOC's behaviour
The technical problem was not merely that one incident had the wrong status.
Once an incident was automatically labelled false positive and closed, analysts were less likely to revisit it.
The automation had therefore influenced both the Sentinel object and the human workflow around it.
A bad automation decision can become a bad analyst assumption.
Once an incident was automatically labelled false positive and closed, analysts were less likely to revisit it.
The automation had therefore influenced both the Sentinel object and the human workflow around it.
A bad automation decision can become a bad analyst assumption.
Fix the automation — then review its history
Agent Foskett narrowed the rule conditions so that only the intended benign scenario could match.
But changing the rule only protected future incidents.
The historical question remained.
Every incident previously closed by the automation during the affected period became a review candidate.
Correcting automation stops the next mistake. It does not investigate the previous ones.
But changing the rule only protected future incidents.
The historical question remained.
Every incident previously closed by the automation during the affected period became a review candidate.
Correcting automation stops the next mistake. It does not investigate the previous ones.
Build the automation-failure timeline
Incident createdSentinel generates an incident from suspicious security evidence.
Automation conditions matchA broadly configured automation rule treats the incident as a known benign case.
Incident closedThe status and classification change before meaningful analyst review occurs.
Closure questionedThe timing and missing investigation trail expose the automated decision.
Underlying evidence reopenedThe original activity is investigated independently of the false-positive label.
Historical scope reviewedOther incidents processed by the same automation become candidates for reassessment.
What Agent Foskett checked
Incident historyHow quickly did the incident move from creation to closure?
ClassificationWhat decision was recorded, and was supporting evidence documented?
Automation conditionsExactly what caused the rule to match this incident?
Automation actionsDid the rule close, classify, tag, assign or otherwise alter the incident?
Underlying telemetryWhat does the original security evidence show without the automated conclusion?
Historical impactWhich other incidents may have been processed by the same faulty logic?
What the evidence can and cannot prove
SecurityIncident can help reconstruct recorded incident state and updates where the relevant data is available. Sentinel audit and configuration evidence can help establish automation and management activity.A quickly closed incident does not automatically prove bad automation.
An automation rule firing does not automatically mean the underlying incident was malicious either.
The investigation must separate the incident's security evidence from the workflow that classified and closed it.
The lesson
Automation is supposed to make the SOC faster.
In this case, it did.
It made the wrong decision faster.
The incident was closed before anyone had established whether the activity was benign.
Automation should accelerate investigation — not replace the evidence required to justify a security decision.
In this case, it did.
It made the wrong decision faster.
The incident was closed before anyone had established whether the activity was benign.
Automation should accelerate investigation — not replace the evidence required to justify a security decision.
Investigation principle: When an incident is automatically closed or classified, verify that the automation conditions represent the complete security context and periodically review the cases those rules remove from analyst queues.
Related investigations
Agent Foskett InvestigationsContinue through the complete collection of real-world Microsoft security investigations.KQL Threat HuntingUse KQL to investigate identity, endpoint, cloud and detection-engineering evidence.SOC Analyst AcademyBuild practical investigation, triage and automation skills across Microsoft security platforms.
Final thought
Automation is one of the SOC's greatest force multipliers.
But every automated action encodes an assumption.
When the assumption is wrong, the system can consistently make the wrong decision at machine speed.
Do not confuse automated closure with investigated closure.
Agent Foskett followed the status change backwards, found the rule that made the decision and returned to the evidence everyone else had stopped looking at.
The Logs Already Knew! 🔎
But every automated action encodes an assumption.
When the assumption is wrong, the system can consistently make the wrong decision at machine speed.
Do not confuse automated closure with investigated closure.
Agent Foskett followed the status change backwards, found the rule that made the decision and returned to the evidence everyone else had stopped looking at.
The Logs Already Knew! 🔎
