Agent Foskett Academy • SOC Analyst Academy • Module 9 • Lesson 83 • Detection Engineering: Turning Findings into Protection

Lesson 83 — Build a Baseline Before Setting the Threshold

The detection needed a threshold.

Someone suggested 10 events in five minutes.

Agent Foskett asked one question:

“Why ten?”

Nobody knew.

Before choosing the number, the team needed to know what normal actually looked like.

A threshold should be justified by observed behaviour — not chosen because the number looks sensible.

Build the Baseline Back to SOC Analyst Academy

Agent Foskett building a historical baseline before setting a detection threshold

Thresholds need evidence.

Measure normal frequency, distribution and variation before deciding where suspicious behaviour begins.

✓ Measure

✓ Compare

✓ Justify

✓ Validate

Case briefing

DETECTION IDEA Multiple suspicious events from the same account within a short period PROPOSED THRESHOLD: 10 EVENTS IN 5 MINUTES QUESTION: WHY 10? ANSWER: "IT SEEMED REASONABLE." STOP. FIRST FIND OUT: WHAT DOES NORMAL LOOK LIKE?

Investigation objective

Use historical telemetry to understand normal event frequency and variation before choosing a threshold for a production detection.

Investigator's rule

Do not choose the threshold first and search for evidence to justify it afterwards.

Stage 1 — define exactly what you are measuring

Question	Example decision
What event?	Suspicious PowerShell executions
Grouped by what?	Device and account
Over what interval?	Five-minute windows
Across which population?	Managed workstations
Over what history?	Previous 30 days

A baseline needs a unit

“Normal PowerShell usage” is too vague. Define what is counted, how it is grouped and over what period before interpreting the numbers.

Population matters

A threshold that makes sense for ordinary workstations may be useless on developer devices, management servers or automation hosts.

Stage 2 — measure historical frequency

1234567891011

DeviceProcessEvents
| where Timestamp > ago(30d)
| where FileName in~ ("powershell.exe", "pwsh.exe")
| summarize EventCount=count()
  by DeviceId, DeviceName, AccountName,
     bin(Timestamp, 5m)
| summarize Windows=count(),
            Average=avg(EventCount),
            Maximum=max(EventCount),
            P95=percentile(EventCount, 95)

Look beyond the average

An average can hide occasional legitimate bursts. Maximums and percentiles help reveal how far normal behaviour sometimes stretches.

Historical range matters

One quiet day is not a baseline. Use enough history to capture ordinary operational variation, maintenance activity and recurring business processes.

Stage 3 — inspect the distribution

30-DAY BASELINE EVENTS PER 5-MIN WINDOW 1–2 events ████████████████████ 3–4 events ███████ 5–6 events ███ 7–8 events █ 9–10 events ▏ 11+ events rare OBSERVATION: MOST NORMAL ACTIVITY IS BELOW 5. BUT OCCASIONAL LEGITIMATE BURSTS REACH 10. A THRESHOLD OF 10 MAY NOT MEAN WHAT WE FIRST ASSUMED.

Normal is a distribution

There may not be one magic value separating benign from malicious. The baseline shows how common or unusual a value is within the observed environment.

Rare does not equal malicious

A rare event deserves attention, but rarity alone does not establish intent. Investigate what produced the outliers.

Stage 4 — investigate the outliers

Outlier	Context	Interpretation
14 events / 5 min	Monthly deployment server	Expected recurring administration
12 events / 5 min	Developer workstation	Known build workflow
9 events / 5 min	Finance workstation	No established explanation
27 events / 5 min	Known incident device	Confirmed malicious activity

Context explains the tails

High values may represent legitimate specialised workloads. A single global threshold can therefore create noise or miss suspicious behaviour in quieter populations.

Segment when justified

If different populations genuinely behave differently, consider separate baselines or contextual logic rather than forcing every device through the same number.

Stage 5 — compare the malicious case to normal

NORMAL WORKSTATIONS Typical: 1–4 95th pct: 6 Observed max: 10 KNOWN ATTACK 27 events in 5 minutes SEPARATION: NORMAL ────────┐ │ 1 2 3 4 5 6 ...10 27 ▲ ATTACK THIS GIVES THE TEAM EVIDENCE TO REASON ABOUT A THRESHOLD.

Use known attacks as reference points

A baseline tells you what normal looks like. Known malicious examples help show where attack behaviour sits relative to that normal range.

Do not optimise for one attack sample

If you set the threshold just below one known incident, a slightly quieter variation may disappear. Consider plausible attacker variation and other historical malicious cases.

Stage 6 — test candidate thresholds

Candidate	Expected result	Trade-off
> 5 events / 5 min	High sensitivity	Likely substantial benign volume
> 10 events / 5 min	Lower benign volume	May still catch specialised legitimate workloads
> 15 events / 5 min	Much quieter	Greater chance of missing lower-volume malicious activity
Threshold + suspicious context	Potentially stronger precision	Requires reliable contextual telemetry

Every threshold is a trade-off

Moving the number upward generally reduces alert volume but can also reduce sensitivity. Moving it downward can improve sensitivity while increasing analyst workload.

There is no universally correct number

The right threshold depends on the hypothesis, telemetry, environment, expected attacker behaviour and operational capacity of the SOC.

Stage 7 — test by population

1234567891011

DeviceProcessEvents
| where Timestamp > ago(30d)
| where FileName in~ ("powershell.exe", "pwsh.exe")
| summarize EventCount=count()
  by DeviceId, DeviceName, AccountName,
     bin(Timestamp, 5m)
| where EventCount > 10
| summarize Matches=count(),
            Peak=max(EventCount)
  by DeviceName, AccountName
| order by Matches desc

Find who would trigger the rule

Before deployment, identify which devices and accounts repeatedly exceed the candidate threshold. Those entities often reveal legitimate workflows the baseline needs to explain.

Repeated outliers deserve understanding

A server exceeding the threshold every night is probably telling you something different from a finance workstation exceeding it for the first time.

Stage 8 — document the threshold rationale

THRESHOLD DECISION MEASURE: PowerShell events per device/user per 5-minute window BASELINE: 30 days NORMAL WORKSTATIONS: 95th percentile = 6 observed maximum = 10 KNOWN MALICIOUS CASE: 27 events CANDIDATE THRESHOLD: > 10 events RATIONALE: Values above 10 are uncommon for the target workstation population and separate the known incident from most historical normal activity. LIMITATION: Lower-volume malicious activity may not trigger this condition. REVIEW: Backtest before production.

Write down why the number exists

A future analyst should be able to trace the threshold back to evidence rather than finding an unexplained constant buried in a query.

Document the blind spot too

If the threshold is greater than 10, activity at 10 or below will not satisfy that condition. State the limitation explicitly.

Stage 9 — baseline drift is inevitable

Environmental change	Possible baseline effect
New management platform	PowerShell frequency increases
Application migration	Old benign pattern disappears
Developer team expansion	More high-frequency endpoints appear
Security control change	Telemetry volume or fields change

A baseline has a date

Historical behaviour is not permanent truth. Record the observation period and revisit assumptions when the environment changes.

Thresholds need maintenance

A justified threshold can become poorly calibrated later. Detection engineering includes reviewing whether the original baseline still represents the environment.

Stage 10 — make the engineering decision

DETECTION ENGINEERING DECISION The proposed threshold of 10 events in five minutes was initially arbitrary. Thirty days of historical telemetry were reviewed. The target workstation population typically generated 1–4 events per window, with a 95th percentile of 6 and an observed maximum of 10. The known malicious case generated 27 events. DECISION: Use the historical baseline to test a candidate threshold above 10 events. NEXT: • backtest against history • validate known attacks • measure expected alert volume • review lower-volume attack risk • document population scope • monitor baseline drift THE NUMBER NOW HAS AN EVIDENCE-BASED REASON.

The baseline did not choose the threshold for us

It gave the team evidence for making the decision. Security impact, false positives, false negatives and analyst workload still need to be considered.

The question changed

Instead of asking “what number feels right?”, the team could now ask “what does this number mean compared with the behaviour we actually observe?”

Lesson 83 key takeaways

Do not choose detection thresholds arbitrarily.
Define exactly what event, entity, interval and population you are measuring.
Use enough historical data to capture ordinary variation.
Look beyond averages to distributions, percentiles and outliers.
Rare behaviour is not automatically malicious.
Investigate legitimate outliers before setting the threshold.
Different populations may require different baselines.
Compare known malicious behaviour with the normal distribution.
Every threshold creates a sensitivity and workload trade-off.
Document why the threshold exists and what it may miss.
A baseline represents an observation period, not permanent truth.
Revisit thresholds when the environment changes.
Backtesting is the next step before trusting the detection in production.

Module 9 — Detection Engineering: Turning Findings into Protection

Lesson 83 used historical behaviour to justify a candidate threshold. Lesson 84 will test whether the detection still looks good when that logic is backtested against historical telemetry.

Next: Lesson 84 — The Detection Looked Good Until We Backtested It

Continue your SOC Analyst training

Module 9 focuses on turning validated investigation findings into transparent, tested and operationally useful detections.

SOC Analyst Academy KQL Academy Microsoft Sentinel Academy Defender for Endpoint Academy Microsoft Entra Security Academy Security Copilot Academy Agent Foskett Academy

🔎 SOC Analyst Academy — Module 9: Detection Engineering: Turning Findings into Protection

Turn investigation and hunting findings into explainable, tested and maintainable production detections.

← Previous lesson

Lesson 82 — The Detection Generated Too Many False Positives Tune noise without deleting the security hypothesis.

✅ Current lesson

Lesson 83 — Build a Baseline Before Setting the Threshold Use historical behaviour to justify thresholds.

Next lesson →

Lesson 84 — The Detection Looked Good Until We Backtested It Test logic against historical telemetry.

How do you choose a detection threshold?

Lesson 83 of the Agent Foskett SOC Analyst Academy teaches analysts how to use historical telemetry, distributions, percentiles and outlier analysis to justify detection thresholds rather than relying on arbitrary values.

Building detection baselines with Microsoft Defender XDR

Learn how to define the measured population, analyse normal event frequency, compare malicious behaviour with historical activity and document the security trade-offs behind a production detection threshold.