Lesson 165 — Building a Baseline Before Setting the Threshold

Detection briefing

Stage 1 — start with history, not the threshold

Suppose we want to detect unusually frequent PowerShell execution. Instead of immediately deciding that 20 executions per hour is suspicious, first measure hourly activity over a useful historical period.

DeviceProcessEvents
| where Timestamp > ago(30d)
| where FileName in~ ("powershell.exe", "pwsh.exe")
| summarize Executions=count(),
            Devices=dcount(DeviceName),
            Users=dcount(AccountName)
          by bin(Timestamp, 1h)
| order by Timestamp asc

Stage 2 — build baselines per user

An environment-wide average can hide important differences. Administrative users may execute PowerShell regularly while most business users rarely do. Measure the behaviour at the entity level.

let HourlyActivity =
    DeviceProcessEvents
    | where Timestamp between (ago(30d) .. ago(1d))
    | where FileName in~ ("powershell.exe", "pwsh.exe")
    | summarize HourlyExecutions=count()
              by AccountName, bin(Timestamp, 1h);
HourlyActivity
| summarize Average=avg(HourlyExecutions),
            Median=percentile(HourlyExecutions, 50),
            P95=percentile(HourlyExecutions, 95),
            Maximum=max(HourlyExecutions),
            ActiveHours=count()
          by AccountName
| order by P95 desc

Stage 3 — compare device baselines too

The same account can behave differently depending on the endpoint. Servers, jump hosts, developer workstations and standard user devices may have very different legitimate execution patterns.

let HourlyDeviceActivity =
    DeviceProcessEvents
    | where Timestamp between (ago(30d) .. ago(1d))
    | where FileName in~ ("powershell.exe", "pwsh.exe")
    | summarize HourlyExecutions=count()
              by DeviceName, bin(Timestamp, 1h);
HourlyDeviceActivity
| summarize Average=avg(HourlyExecutions),
            P95=percentile(HourlyExecutions, 95),
            Maximum=max(HourlyExecutions)
          by DeviceName
| order by P95 desc

Stage 4 — compare current behaviour with historical expectation

Now bring recent activity together with the historical baseline. This makes the detection relative to what each account normally does instead of relying on one arbitrary global number.

let Baseline =
    DeviceProcessEvents
    | where Timestamp between (ago(30d) .. ago(1d))
    | where FileName in~ ("powershell.exe", "pwsh.exe")
    | summarize HourlyExecutions=count()
              by AccountName, bin(Timestamp, 1h)
    | summarize P95=percentile(HourlyExecutions, 95),
                HistoricalMax=max(HourlyExecutions)
              by AccountName;
let Current =
    DeviceProcessEvents
    | where Timestamp > ago(1h)
    | where FileName in~ ("powershell.exe", "pwsh.exe")
    | summarize CurrentExecutions=count(),
                Devices=dcount(DeviceName)
              by AccountName;
Current
| join kind=leftouter Baseline on AccountName
| extend P95=coalesce(P95, 0.0)
| extend AboveBaseline =
    CurrentExecutions > P95
| project AccountName, CurrentExecutions,
          P95, HistoricalMax, Devices,
          AboveBaseline
| order by CurrentExecutions desc

Stage 5 — test a candidate threshold before alerting

Before converting the baseline into production detection logic, measure how often the candidate threshold would have fired historically. A threshold that would generate hundreds of daily alerts has already told you something important.

let Hourly =
    DeviceProcessEvents
    | where Timestamp > ago(30d)
    | where FileName in~ ("powershell.exe", "pwsh.exe")
    | summarize Executions=count()
              by AccountName, bin(Timestamp, 1h);
let UserBaseline =
    Hourly
    | summarize P95=percentile(Executions, 95)
              by AccountName;
Hourly
| join kind=inner UserBaseline on AccountName
| where Executions > P95
| summarize CandidateAlerts=count(),
            AlertDays=dcount(startofday(Timestamp)),
            PeakExecutions=max(Executions)
          by AccountName
| order by CandidateAlerts desc

Agent Foskett's baseline-to-threshold workflow

Weak thresholds versus evidence-based thresholds

Write the threshold decision like a detection engineer

Example: Thirty days of historical PowerShell telemetry was analysed by account and device. Standard business users showed very low hourly execution volumes, while administrative and automation identities displayed materially higher and more variable activity. Rather than applying a single global threshold, the detection was evaluated against entity-level historical distributions. Candidate thresholds were backtested to measure expected alert volume, and deviations above normal ranges were retained as investigation leads rather than treated as proof of compromise. The resulting threshold can therefore be traced to observed environmental behaviour instead of an arbitrary number.

Lesson 165 key takeaways

• Do not choose a detection threshold before measuring normal behaviour.
• Use a historical period that represents legitimate environmental activity.
• Remember that historical data can contain abnormal or malicious behaviour.
• Build baselines at the user, device or role level when behaviour differs materially.
• Average alone does not describe a behavioural distribution.
• Median, percentiles and maximum values provide useful additional context.
• A percentile is a reference point, not an automatic security threshold.
• Handle entities with little or no history separately.
• Backtest candidate thresholds before production deployment.
• Measure expected analyst workload as well as detection coverage.
• Review and recalibrate baselines as the environment changes.

Module 14 — now make the baseline adapt

Lesson 165 replaced arbitrary thresholds with evidence from historical behaviour. But environments change. Static baselines can become stale as users change roles, systems are upgraded and normal workloads shift. Next we look at how to recognise those changes without allowing every new pattern to silently redefine normal.