Lesson 88 — The Environment Changed After Deployment
The detection had been reliable for months.
Then its alert volume suddenly tripled.
The rule had not changed.
The environment had.
Production is not permanent.
New software, workflows, identities and infrastructure can change what normal looks like after deployment.
Case briefing
Investigation objective
Recognise detection drift, identify the environmental change that invalidated an original assumption and decide whether the rule, its context or its baseline needs to be updated.
Investigator's rule
When a stable detection changes behaviour, investigate the environment before assuming the detection is broken.
Stage 1 — prove that something changed
DeviceProcessEvents
| where Timestamp > ago(90d)
| where FileName in~ ("powershell.exe", "pwsh.exe")
| where InitiatingProcessFileName in~ (
"msedge.exe", "chrome.exe", "firefox.exe"
)
| summarize Events=count() by bin(Timestamp, 1d)
| order by Timestamp ascLook for the change point
Do not compare only “before” and “after” totals. Trend the underlying behaviour so you can identify approximately when the environment shifted.
Alert drift or telemetry drift?
Confirm whether the underlying activity changed, the telemetry changed, or only the alerting layer changed. Those lead to different investigations.
Stage 2 — list the original assumptions
| Original assumption | Why it mattered |
|---|---|
| Browsers rarely launch PowerShell | Made the behaviour distinctive |
| Only a small support population uses the workflow | Kept expected volume low |
| Developer devices are handled separately | Reduced benign overlap |
| Endpoint telemetry is consistently available | Made the rule observable |
Detections contain hidden dependencies
A rule may depend on business processes, software versions, device roles, identity patterns and telemetry quality even when none of those appear directly in the query.
Write assumptions down
If the assumptions were documented when the rule was built, drift review becomes much faster. If they were not, reconstruct them now.
Stage 3 — identify what changed
Environmental drift can be legitimate
A new deployment tool, browser extension, automation platform or support workflow may make previously unusual behaviour common.
But drift can also be malicious
A sudden increase is not automatically “the business changed.” Establish the cause with evidence before retuning the detection.
Stage 4 — find the new population
DeviceProcessEvents
| where Timestamp > ago(14d)
| where FileName in~ ("powershell.exe", "pwsh.exe")
| where InitiatingProcessFileName in~ (
"msedge.exe", "chrome.exe", "firefox.exe"
)
| summarize Events=count(),
FirstSeen=min(Timestamp),
LastSeen=max(Timestamp)
by DeviceId, DeviceName, AccountName
| order by Events descConcentration gives you a lead
If most new volume comes from a small set of devices or accounts, investigate what changed on those entities rather than immediately changing the global rule.
Compare first seen dates
Entities that appear only after the change point can connect detection drift to a deployment, migration or new operational process.
Stage 5 — correlate with known change
| Evidence | Finding |
|---|---|
| Detection increase began Monday 08:00 | Clear change point |
| Most new events came from support laptops | Population concentrated |
| New remote-support workflow deployed Monday | Timing correlates |
| Workflow launches PowerShell through a browser-based console | Explains behavioural overlap |
| Known malicious test still matches | Security hypothesis remains relevant |
Correlation is not enough by itself
Validate that the changed workflow actually produces the observed process chain. A change ticket with the right date is useful context, not proof.
The rule did not suddenly become wrong
The environment changed the meaning and frequency of one of its signals. That requires a new engineering decision.
Stage 6 — decide what should change
Prefer the smallest justified change
If one verified support workflow causes the drift, a narrow contextual treatment may be safer than changing the threshold for every device in the organisation.
Sometimes the hypothesis must evolve
If browser-launched PowerShell has become common everywhere, the original signal may no longer be distinctive enough. The detection may need stronger behavioural context.
Stage 7 — retest against the new environment
| Test | Question |
|---|---|
| Recent backtest | Is volume operationally sustainable now? |
| Old historical backtest | Did the change unexpectedly alter earlier coverage? |
| Known malicious cases | Do confirmed attacks still match? |
| New legitimate workflow | Is expected activity handled as intended? |
| Edge populations | Did the tuning affect unrelated devices or users? |
Use both old and new history
Testing only recent telemetry may optimise the rule for today's environment while accidentally losing behaviour it was originally designed to detect.
Preserve positive controls
Known malicious cases remain valuable regression tests. A detection should continue to reproduce the security outcomes it was built to protect.
Stage 8 — monitor for detection drift
Drift monitoring is maintenance
Production detections should not be treated as “finished.” Their assumptions need periodic review as the organisation and telemetry evolve.
Review changes proactively
Major software rollouts, migrations and identity changes can be useful triggers for targeted detection review rather than waiting for the SOC queue to reveal the problem.
Stage 9 — document the revised assumption
Update the detection record
Record what changed, which assumption expired, what evidence justified the new logic and when the revised decision should be reviewed again.
Give the change an expiry point
Temporary workflows become permanent, systems are retired and device populations move. Review dates help prevent old exceptions from surviving long after their justification disappears.
Stage 10 — treat detections as living controls
Deployment is not the finish line
A detection can be correct on the day it ships and wrong six months later without a single line of query logic changing.
Drift should be explainable
When detection behaviour changes, the team should be able to identify whether the cause is the threat, the environment, the data or the rule.
Lesson 88 key takeaways
- Production detections are built on assumptions about the environment.
- Those assumptions can become invalid even when rule logic does not change.
- Trend underlying behaviour to identify the change point.
- Distinguish alert drift from telemetry and behavioural drift.
- Reconstruct the assumptions that made the original signal useful.
- Investigate new devices, accounts, software and workflows contributing to volume.
- Do not assume a sudden change is benign.
- Correlate environmental changes with telemetry evidence.
- Prefer narrow, evidence-based changes over broad exclusions.
- Retest against both the new environment and historical malicious cases.
- Monitor volume, entity concentration, coverage and data-source health after deployment.
- Document revised assumptions and review dates.
- Detection engineering continues after production deployment.
Module 9 — Detection Engineering: Turning Findings into Protection
Lesson 88 showed how environmental drift can invalidate a detection's original assumptions and why deployed rules need ongoing monitoring and retesting. Lesson 89 turns to another production requirement: can another analyst clearly explain why the alert fired?
Continue your SOC Analyst training
🔎 SOC Analyst Academy — Module 9: Detection Engineering: Turning Findings into Protection
What is detection drift in security operations?
Lesson 88 of the Agent Foskett SOC Analyst Academy teaches analysts how changes in software, workflows, identities, device populations and telemetry can invalidate assumptions behind a production security detection.
Monitoring and maintaining deployed detections
Learn how to identify detection drift, locate environmental change points, review original assumptions, retest known malicious cases and safely update detection logic without creating new coverage gaps.
