Agent Foskett Academy • KQL Academy • Module 14 • Lesson 168 • Detection Engineering & Proactive Threat Hunting

Lesson 168 — The Detection Looked Good Until We Backtested It

The query was clean. The logic made sense. The signals correlated around the same process, device and time window. On paper, the detection looked ready.

Then we ran it against historical telemetry. Hundreds of results appeared — and most belonged to software deployment, administrator tooling and legitimate automation. Backtesting had done exactly what it was supposed to do: expose the difference between a good detection idea and a production-ready detection.

A detection is not ready because the KQL runs successfully. It is ready when historical testing shows that it finds useful behaviour at a level analysts can realistically investigate.

Backtest the Detection Back to KQL Academy

Agent Foskett KQL Academy detection backtesting

Your detection problem

A promising multi-signal rule performs badly when exposed to real historical data. Measure the noise, find recurring legitimate patterns, test known suspicious examples and tune without destroying the behaviour the detection was designed to find.

✓ Run against historical telemetry

✓ Measure prevalence and noise

✓ Identify false-positive clusters

✓ Retest after every tuning change

Detection briefing

DETECTION CANDIDATE Looks strong in development ↓ RUN AGAINST 30 DAYS OF TELEMETRY ↓ 847 MATCHES ↓ GROUP + MEASURE ↓ MOST RESULTS = LEGITIMATE ADMIN / AUTOMATION ↓ IDENTIFY THE NOISE ↓ TUNE CAREFULLY ↓ RETEST KNOWN SUSPICIOUS BEHAVIOUR ↓ BACKTEST AGAIN ↓ PRODUCTION CANDIDATE

Detection objective

Use KQL to evaluate a candidate detection against historical telemetry, measure how frequently it fires, identify the users, devices and command patterns creating noise, validate whether known suspicious activity remains detectable and document the tuning decisions made before production deployment.

Detection engineer's rule

Never tune only to make the result count smaller. A quiet rule that no longer detects the behaviour it was designed to find is not an improvement. Every exclusion must preserve the security hypothesis.

Stage 1 — run the candidate across a meaningful historical window

The first test is deliberately simple. Take the core behaviour from Lesson 167 and extend the time range. A one-hour or one-day development window may hide recurring legitimate activity. Thirty days gives us a much better view of prevalence across normal business cycles.

1234567891011121314151617181920212223242526272829

DeviceProcessEvents
| where Timestamp > ago(30d)
| where FileName in~ ("powershell.exe", "pwsh.exe")
| extend BrowserParent =
    InitiatingProcessFileName in~ (
        "chrome.exe",
        "msedge.exe",
        "firefox.exe"
    )
| extend DownloadBehaviour =
    ProcessCommandLine has_any (
        "Invoke-WebRequest",
        "DownloadString",
        "WebClient",
        "Start-BitsTransfer"
    )
| extend EncodedCommand =
    ProcessCommandLine has_any (
        "-enc",
        "-encodedcommand"
    )
| extend SignalCount =
    toint(BrowserParent)
    + toint(DownloadBehaviour)
    + toint(EncodedCommand)
| where SignalCount >= 2
| project Timestamp, DeviceName, AccountName,
          ProcessCommandLine, InitiatingProcessFileName,
          SignalCount

The first surprise

The query may return far more events than expected. That is useful information. Backtesting is not a demonstration that your query works syntactically; it is an experiment designed to discover how the logic behaves in the real environment.

Choose the window deliberately

Include enough history to capture patching, software deployment, month-end activity, administrator maintenance and other recurring patterns. A detection tested only during a quiet afternoon has barely been tested at all.

Stage 2 — measure before reading individual events

Do not manually open hundreds of rows. Summarise the backtest first. We want to know whether the matches are distributed across the estate or concentrated around a few users, devices or parent processes.

12345678910111213141516171819202122232425262728

let Backtest =
    DeviceProcessEvents
    | where Timestamp > ago(30d)
    | where FileName in~ ("powershell.exe", "pwsh.exe")
    | extend BrowserParent =
        InitiatingProcessFileName in~ (
            "chrome.exe", "msedge.exe", "firefox.exe"
        )
    | extend DownloadBehaviour =
        ProcessCommandLine has_any (
            "Invoke-WebRequest", "DownloadString",
            "WebClient", "Start-BitsTransfer"
        )
    | extend EncodedCommand =
        ProcessCommandLine has_any (
            "-enc", "-encodedcommand"
        )
    | extend SignalCount =
        toint(BrowserParent)
        + toint(DownloadBehaviour)
        + toint(EncodedCommand)
    | where SignalCount >= 2;
Backtest
| summarize Matches=count(),
            Devices=dcount(DeviceId),
            Users=dcount(AccountName)
          by InitiatingProcessFileName
| order by Matches desc

Concentration tells a story

If 80 percent of the results come from three deployment servers or one administrator workflow, you have learned something actionable. The detection may not be fundamentally wrong; it may simply be missing environmental context.

Volume is not the only metric

Record unique devices, users, recurring command patterns and time distribution as well as total matches. Ten events across ten sensitive endpoints may matter more than one hundred repetitions of the same approved automation.

Stage 3 — find the false-positive clusters

Now group the command lines and process context. Normalise command-line values so repeated legitimate workflows become visible instead of appearing as hundreds of unrelated rows.

123456789101112131415161718192021222324252627

DeviceProcessEvents
| where Timestamp > ago(30d)
| where FileName in~ ("powershell.exe", "pwsh.exe")
| extend BrowserParent =
    InitiatingProcessFileName in~ (
        "chrome.exe", "msedge.exe", "firefox.exe"
    )
| extend DownloadBehaviour =
    ProcessCommandLine has_any (
        "Invoke-WebRequest", "DownloadString",
        "WebClient", "Start-BitsTransfer"
    )
| extend EncodedCommand =
    ProcessCommandLine has_any (
        "-enc", "-encodedcommand"
    )
| extend SignalCount =
    toint(BrowserParent)
    + toint(DownloadBehaviour)
    + toint(EncodedCommand)
| where SignalCount >= 2
| summarize Matches=count(),
            Devices=make_set(DeviceName, 10),
            Users=make_set(AccountName, 10)
          by InitiatingProcessFileName,
             ProcessCommandLine
| order by Matches desc

Investigate before excluding

A frequently occurring command is not automatically benign. Confirm the owner, purpose, deployment mechanism, expected devices and change history before creating an exclusion.

Prefer narrow exclusions

Exclude the verified workflow rather than an entire technology. Removing all PowerShell, all administrator accounts or all management servers may make the rule quiet while creating a large blind spot.

Stage 4 — tune the verified noise without erasing the hypothesis

Suppose investigation confirms that a signed internal deployment script creates a recurring legitimate pattern. Add the narrowest reliable environmental condition available and document why it exists.

123456789101112131415161718192021222324252627282930313233

let ApprovedDeploymentDevices =
    dynamic([
        "DEPLOY-01",
        "DEPLOY-02"
    ]);
DeviceProcessEvents
| where Timestamp > ago(30d)
| where FileName in~ ("powershell.exe", "pwsh.exe")
| extend BrowserParent =
    InitiatingProcessFileName in~ (
        "chrome.exe", "msedge.exe", "firefox.exe"
    )
| extend DownloadBehaviour =
    ProcessCommandLine has_any (
        "Invoke-WebRequest", "DownloadString",
        "WebClient", "Start-BitsTransfer"
    )
| extend EncodedCommand =
    ProcessCommandLine has_any (
        "-enc", "-encodedcommand"
    )
| extend SignalCount =
    toint(BrowserParent)
    + toint(DownloadBehaviour)
    + toint(EncodedCommand)
| where SignalCount >= 2
| where not(
    DeviceName in~ (ApprovedDeploymentDevices)
    and ProcessCommandLine has
        "Approved-Software-Deployment.ps1"
)
| project Timestamp, DeviceName, AccountName,
          ProcessCommandLine, SignalCount

Context makes exclusions safer

The example requires both an approved device and a specific verified workflow. That is much safer than excluding every event containing a generic word such as “deployment”. Production allowlists should be governed and reviewed as environments change.

Every exclusion is a security decision

Record who validated the behaviour, why it is trusted, when the exclusion was added and what would cause it to be reviewed. An undocumented exclusion can become tomorrow's detection blind spot.

Stage 5 — test whether the detection still catches what matters

Tuning is only half the test. Now verify that suspicious examples still satisfy the detection logic. If you have labelled historical incidents, red-team activity or controlled test events, compare expected detections with actual detections.

1234567891011121314151617181920212223242526272829303132

let KnownSuspiciousDevices =
    dynamic([
        "LAB-WIN11-07",
        "REDTEAM-CLIENT-02"
    ]);
DeviceProcessEvents
| where Timestamp > ago(30d)
| where DeviceName in~ (KnownSuspiciousDevices)
| where FileName in~ ("powershell.exe", "pwsh.exe")
| extend BrowserParent =
    InitiatingProcessFileName in~ (
        "chrome.exe", "msedge.exe", "firefox.exe"
    )
| extend DownloadBehaviour =
    ProcessCommandLine has_any (
        "Invoke-WebRequest", "DownloadString",
        "WebClient", "Start-BitsTransfer"
    )
| extend EncodedCommand =
    ProcessCommandLine has_any (
        "-enc", "-encodedcommand"
    )
| extend SignalCount =
    toint(BrowserParent)
    + toint(DownloadBehaviour)
    + toint(EncodedCommand)
| project Timestamp, DeviceName,
          AccountName, ProcessCommandLine,
          BrowserParent, DownloadBehaviour,
          EncodedCommand, SignalCount,
          WouldDetect = SignalCount >= 2
| order by Timestamp desc

False negatives matter too

If a known suspicious event now produces WouldDetect = false, stop and investigate the tuning. Reducing false positives at the cost of obvious false negatives is not successful optimisation.

Known examples are valuable regression tests

Keep a small set of representative test cases where your process allows it. Each time the rule changes, rerun those cases. Detection engineering benefits from regression testing just as software engineering does.

Stage 6 — compare before and after

Finally, measure the effect of tuning. The goal is not zero results. The goal is a meaningful reduction in verified noise while retaining suspicious coverage and enough context for analysts to make decisions.

1234567891011121314151617181920212223242526272829303132

let BaseCandidates =
    DeviceProcessEvents
    | where Timestamp > ago(30d)
    | where FileName in~ ("powershell.exe", "pwsh.exe")
    | extend BrowserParent =
        InitiatingProcessFileName in~ (
            "chrome.exe", "msedge.exe", "firefox.exe"
        )
    | extend DownloadBehaviour =
        ProcessCommandLine has_any (
            "Invoke-WebRequest", "DownloadString",
            "WebClient", "Start-BitsTransfer"
        )
    | extend EncodedCommand =
        ProcessCommandLine has_any (
            "-enc", "-encodedcommand"
        )
    | extend SignalCount =
        toint(BrowserParent)
        + toint(DownloadBehaviour)
        + toint(EncodedCommand)
    | where SignalCount >= 2;
BaseCandidates
| summarize BeforeTuning=count(),
            VerifiedNoise=countif(
                ProcessCommandLine has
                "Approved-Software-Deployment.ps1"
            ),
            RemainingCandidates=countif(
                ProcessCommandLine !has
                "Approved-Software-Deployment.ps1"
            )

Measure the trade-off

Record candidate volume before and after tuning, the verified noise removed and the suspicious test cases retained. Those measurements make the change reviewable instead of relying on “the results look better now”.

Backtesting never really ends

Software changes, administrators introduce new workflows and attackers change techniques. A production detection should be reviewed when its result profile changes significantly, not frozen forever after its first successful test.

Agent Foskett's detection backtesting workflow

WRITE THE SECURITY HYPOTHESIS ↓ BUILD THE CANDIDATE DETECTION ↓ RUN AGAINST HISTORICAL DATA ↓ MEASURE VOLUME + PREVALENCE ↓ CLUSTER RECURRING RESULTS ↓ INVESTIGATE THE NOISE ↓ VERIFY LEGITIMATE BEHAVIOUR ↓ ADD NARROW, DOCUMENTED TUNING ↓ RETEST KNOWN SUSPICIOUS EXAMPLES ↓ COMPARE BEFORE + AFTER ↓ PROMOTE / REWORK / REJECT ↓ MONITOR IN PRODUCTION

Backtesting is where detection engineering stops being a clever query and starts becoming an evidence-based security control.

What a backtest can reveal

Backtest finding	What it may mean	Detection response
Very high result volume	The behaviour is common or the logic is too broad.	Group results and identify dominant legitimate patterns.
Results concentrated on a few devices	Management or deployment infrastructure may explain the activity.	Investigate the workflow before considering a narrow exclusion.
Known suspicious example is missed	The threshold or tuning may be too restrictive.	Review the logic before production deployment.
One command dominates results	A repeatable legitimate workflow or widespread suspicious activity may exist.	Validate ownership and purpose; never assume frequency means benign.
Noise falls but suspicious tests still match	Tuning may be improving precision without obvious coverage loss.	Document the evidence and continue validation.

Write the backtest like a detection engineer

Example: The candidate detection was evaluated against 30 days of historical endpoint telemetry. Initial testing produced a high volume of matches concentrated around a small number of approved software deployment workflows. Those patterns were investigated and verified before narrowly scoped exclusions were introduced. The tuned logic was then rerun against the same historical period and tested against known suspicious examples to confirm that the reduction in false positives did not remove the behaviour the detection was designed to identify. Result volume, exclusions and validation outcomes were documented before production promotion.

Lesson 168 key takeaways

A syntactically correct KQL query is not automatically a production-ready detection.
Backtest candidate detections across a meaningful historical period.
Measure result volume, prevalence, affected devices, users and recurring patterns.
Investigate frequent behaviour before deciding that it is benign.
Use the narrowest reliable exclusion for verified legitimate activity.
Never tune solely to reduce the number of alerts.
Retest known suspicious examples after every material tuning change.
False negatives are as important to understand as false positives.
Document exclusions, assumptions, validation evidence and tuning decisions.
Detection backtesting should continue as the environment and threat landscape change.

Module 14 — now measure whether analysts can live with it

Lesson 168 showed that historical testing can expose noise and blind spots before a detection reaches production. But even a technically accurate detection can fail operationally if it fires too often, duplicates other analytics or consumes more analyst time than the risk justifies. Next we move from query quality to operational quality.

Next: Lesson 169 — The Detection Worked — But It Fired 600 Times a Day.

Continue your KQL investigation training

Module 14 turns investigation knowledge into proactive hunts, reusable analytics and higher-confidence detections.

KQL Academy Lesson 167 — Three Weak Signals Became One Strong Detection Microsoft Sentinel Academy Agent Foskett Academy

🔎 KQL Academy — Module 14: Advanced Detection Engineering & Proactive Threat Hunting

Turn investigation knowledge into proactive hunts, reusable analytics and higher-confidence detections.

⬅ Previous lesson

Lesson 167 — Three Weak Signals Became One Strong DetectionCombine multiple low-confidence behavioural indicators into a higher-confidence detection candidate.

✅ Current lesson

Lesson 168 — The Detection Looked Good Until We Backtested ItTest detection logic against historical telemetry to expose noise, blind spots and assumptions before production deployment.

Next lesson

Lesson 169 — The Detection Worked — But It Fired 600 Times a DayMeasure operational alert volume and tune a working detection so analysts can realistically investigate what it produces.

Backtest KQL detections against historical telemetry

Lesson 168 of the Agent Foskett KQL Academy shows how to test candidate Microsoft Defender XDR detection logic across historical endpoint telemetry before production deployment.

Measure false positives and preserve detection coverage

Learn how to measure result prevalence, cluster recurring legitimate activity, create narrow exclusions, validate known suspicious examples and document detection tuning decisions.