Lesson 84 — The Detection Looked Good Until We Backtested It
The logic looked clean.
The threshold was evidence-based.
Then Agent Foskett ran it against the last 30 days.
It would have generated 4,286 alerts.
The detection was not ready.
Test before production.
A sensible query can behave very differently when exposed to real historical telemetry.
Case briefing
Investigation objective
Test proposed detection logic against historical telemetry to measure likely volume, identify unexpected benign patterns, confirm known malicious cases and expose weaknesses before production.
Investigator's rule
A detection is not ready because the query runs. It is ready when you understand what the query returns.
Stage 1 — what backtesting actually proves
| Development check | Backtest question |
|---|---|
| Does the query execute? | How would the logic have behaved historically? |
| Does it find the known example? | What else would it have found? |
| Does the syntax look correct? | What workload would it have created? |
| Does the idea sound reasonable? | Does real telemetry support the idea? |
History becomes a test environment
Past telemetry exposes the candidate to real devices, users and operational workflows without immediately filling the production alert queue.
Backtesting is not prediction
History cannot guarantee how a future attacker will behave. It can show how the proposed logic interacts with behaviour already observed.
Stage 2 — replay the candidate logic
DeviceProcessEvents
| where Timestamp > ago(30d)
| where FileName in~ ("powershell.exe", "pwsh.exe")
| summarize EventCount=count()
by DeviceId, DeviceName, AccountName,
bin(Timestamp, 5m)
| where EventCount > 10
| order by EventCount descTest the logic you intend to deploy
A simplified approximation can hide the behaviour of the real production rule. Keep the tested time range and candidate version explicit.
Measure more than rows
Total matches, unique devices, unique accounts and repeated entities each describe a different part of the likely SOC workload.
Stage 3 — the first result
Concentration matters
If a few known systems create most matches, the rule may need population-aware logic rather than a completely different security hypothesis.
Translate results into analyst work
Ask how historical matches would become alerts or incidents. A technically accurate rule can still be operationally unusable.
Stage 4 — find the dominant entities
DeviceProcessEvents
| where Timestamp > ago(30d)
| where FileName in~ ("powershell.exe", "pwsh.exe")
| summarize EventCount=count()
by DeviceId, DeviceName, AccountName,
bin(Timestamp, 5m)
| where EventCount > 10
| summarize Matches=count(), Peak=max(EventCount)
by DeviceName, AccountName
| order by Matches descInvestigate before excluding
Understand why high-frequency systems behave differently before deciding whether segmentation, contextual logic or a narrow exclusion is justified.
Sample different result types
Review common patterns, rare matches, different device populations and activity around known incidents — not only the largest results.
Stage 5 — classify what the backtest found
| Historical pattern | Possible interpretation |
|---|---|
| Same device every night | Recurring automation or administration |
| Many developer devices | Different population baseline |
| One user across many devices | Admin activity or possible credential misuse |
| Rare isolated workstation | Higher-value investigation candidate |
| Known incident timestamp | Opportunity to validate malicious coverage |
A backtest can uncover an investigation
An unexplained historical result should not be dismissed because the exercise began as engineering. The underlying activity may still matter.
Known benign needs evidence
As in false-positive tuning, classify historical matches from context rather than assuming that frequent behaviour must be safe.
Stage 6 — confirm the known malicious case
Use positive controls
Known malicious examples provide reference points. If the candidate misses the behaviour that motivated it, understand why before proceeding.
Do not overfit one incident
An attacker may change volume, timing or sequence. Avoid building confidence around one historical example that perfectly matches the rule.
Stage 7 — compare candidate versions
| Measure | Candidate A | Candidate B |
|---|---|---|
| 30-day matches | 4,286 | 312 |
| Unique devices | 147 | 41 |
| Known malicious case | Matched | Matched |
| Dominant benign workflow | Included | Contextually reduced |
| Coverage trade-off | Broad noise | Requires documented review |
Lower is not automatically better
A reduced match count is useful only when the team can explain what disappeared, why it disappeared and which attack coverage remains.
Keep the evidence between versions
Comparing candidate versions makes tuning measurable rather than relying on memory or intuition.
Stage 8 — look for time-based surprises
Totals can hide bursts
A rule may look moderately noisy over a month while producing an unacceptable spike during deployment, maintenance or another recurring event.
Test meaningful operating periods
Where relevant, compare weekdays, weekends, maintenance windows and business cycles. The environment may not behave uniformly.
Stage 9 — production-readiness decision
| Question | Finding |
|---|---|
| Does the logic execute? | Yes |
| Does the known attack match? | Yes |
| Is historical volume acceptable? | No |
| Is the source of noise understood? | Mostly — high-frequency populations dominate |
| Can the hypothesis be preserved? | Yes — refine context and population |
| Ready for production? | No — retest first |
“Not ready” is a good result
Finding a weakness before deployment is exactly what backtesting is supposed to achieve.
Production should not be the first real test
The SOC queue is an expensive place to discover that a rule behaves differently from what the engineer expected.
Stage 10 — write the engineering decision
Backtesting creates evidence
Instead of saying “this should work,” the team can explain how the candidate behaved against real telemetry and where uncertainty remains.
The next problem is operational
Even a valid detection can create repeated work if every matching event becomes another alert. That is where Lesson 85 takes us.
Lesson 84 key takeaways
- Backtest detection logic before trusting it in production.
- A query running successfully does not prove operational usefulness.
- Test the actual candidate logic and record the historical period.
- Measure total matches, unique entities and result concentration.
- Translate historical matches into likely analyst workload.
- Investigate the behaviours dominating the backtest.
- Confirm known malicious cases still match.
- Do not overfit the detection to one attack example.
- Compare candidate versions using measurable evidence.
- Review time-based spikes hidden by aggregate totals.
- A lower match count is useful only when the lost matches are understood.
- “Not ready” is a successful engineering outcome when discovered before deployment.
- Backtesting reduces production surprises but cannot predict every future behaviour.
Module 9 — Detection Engineering: Turning Findings into Protection
Lesson 84 backtested the candidate and exposed the workload hidden behind apparently sensible logic. Lesson 85 will examine what happens when a production detection fires hundreds of times every day.
Continue your SOC Analyst training
🔎 SOC Analyst Academy — Module 9: Detection Engineering: Turning Findings into Protection
How do you backtest a security detection?
Lesson 84 of the Agent Foskett SOC Analyst Academy teaches analysts how to test proposed detection logic against historical telemetry before production deployment, measure expected volume and investigate unexpected matches.
Backtesting detections in Microsoft Defender XDR
Use historical Advanced Hunting telemetry to validate known malicious cases, identify dominant benign patterns, compare candidate versions and expose operational weaknesses before a rule reaches the SOC queue.
