Lesson 31 — Tuning Analytics Rules and Reducing False Positives
The rule works.
Unfortunately, it works so well that the SOC now receives 600 alerts a day.
Detection engineering is not finished when the alert fires. The next job is tuning the rule so analysts see the activity that matters without filtering away the evidence that could reveal a real attack.
What you will learn
Learn how to reduce false positives without destroying the detection.
Learning objectives
- Explain why false positives occur even in valid detections.
- Investigate noisy entities before adding exclusions.
- Tune query logic and thresholds using evidence.
- Use watchlists to manage repeatable exceptions.
- Choose between automation exceptions and query changes.
The investigation
A failed sign-in rule is firing hundreds of times every day.
The easy fix is to exclude the noisy accounts. The correct first step is to find out which accounts are noisy and why.
A false positive is still evidence
A false positive means the rule detected behaviour matching its logic, but the activity was ultimately determined to be benign in that context.
Common causes include normal service-principal activity, approved security scanning from known IP addresses, and internal addresses that legitimately match otherwise suspicious patterns.
Before excluding anything, prove why it is safe to exclude.
The tuning workflow
Start with the noisy entities
Users, service principals, IP addresses, hosts and applications often explain why a detection is generating repeated benign alerts.
Summarise the data before touching the rule.
Ask why, not just how many
A service account generating 5,000 failures may be a broken application — or an attacker abusing a privileged identity.
Volume alone does not tell you whether an exclusion is safe.
Find the source of the noise with KQL
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
SigninLogs
| where TimeGenerated > ago(7d)
| where ResultType != "0"
| summarize FailedSignIns = count(),
FirstSeen = min(TimeGenerated),
LastSeen = max(TimeGenerated)
by UserPrincipalName, IPAddress
| order by FailedSignIns desc
This turns “the rule is noisy” into something you can investigate: which identity, which IP address, how often, and over what period?
Validate the benign explanation
Confirm the account owner, application, source address, expected behaviour, change record or business process that explains the activity.
An exclusion based on assumption is not tuning. It is loss of visibility.
Make the exception narrow
Perhaps the behaviour is expected only from one IP range, one application or one maintenance window.
A narrow exception preserves more detection coverage than excluding the user everywhere.
Method 1 — tune the analytics query
Modifying the scheduled analytics rule query is the most flexible approach when the exception belongs inside the detection logic. Query changes can use boolean logic, subnet exceptions and watchlists.
- 1
- 2
- 3
SigninLogs
| where ResultType != "0"
| where IPAddress !in ("203.0.113.10", "203.0.113.11")
Hard-coded exclusions have a cost
A short, stable exception may be understandable in the query, but long lists become difficult to manage and audit.
If analysts regularly add and remove exceptions, move that operational data outside the detection logic.
Document why the exception exists
Use readable constants and comments where appropriate.
Six months later, an IP address should not be a mystery value nobody is willing to remove.
Method 2 — manage exceptions with a watchlist
Watchlists are useful when you want to centralise exception management and reuse the same list across multiple rules.
The _GetWatchlist() function lets the query retrieve the current exception list without hard-coding every value into the rule.
- 1
- 2
- 3
- 4
- 5
- 6
let allowlist =
(_GetWatchlist('ipallowlist') | project SearchKey);
SigninLogs
| where ResultType != "0"
| where IPAddress !in (allowlist)
Use the SearchKey
When a watchlist is created, its SearchKey identifies the column expected to be used frequently for joins and lookups.
Using SearchKey is the recommended approach for query performance.
Watchlists improve operations
An approved exception list can be maintained without repeatedly rewriting detection logic.
The same watchlist can support multiple analytics rules, giving the SOC a central place to manage known benign entities.
Method 3 — automation rule exceptions
Automation rules can handle known false positives without changing the analytics rule query itself.
This approach can apply across several analytics rules, preserve an audit trail, close matching incidents with a reason and comment, and support time-limited exceptions such as maintenance activity.
Automation preserves the detection
The suspicious pattern is still detected and recorded. The automation layer handles the known benign context afterwards.
This can be useful when the exception is temporary or when retaining an audit trail matters.
Query tuning stops the match earlier
A query exclusion prevents known activity from becoming a rule result in the first place.
That reduces noise more completely, but you must be confident the excluded behaviour no longer needs to be surfaced by that detection.
Automation or query change?
| Situation | Useful approach |
|---|---|
| Temporary maintenance window | Automation rule with a time-limited exception. |
| Stable known-benign entity | Query exclusion after validation. |
| Exception shared across many detections | Watchlist for central management. |
| Complex subnet or boolean logic | Modify the analytics query. |
| Analyst-managed incident workflow | Automation rule after detection. |
Threshold tuning
Sometimes the activity is not benign — the rule is simply too sensitive.
Changing a threshold from five events to fifty may reduce noise, but only if historical evidence shows the higher threshold still detects the behaviour you care about.
Do not tune by frustration
“The SOC is sick of this alert” proves tuning is needed, but it does not prove what the new threshold should be.
Use historical data to understand normal and suspicious ranges before changing the number.
Baseline before changing the threshold
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
SigninLogs
| where TimeGenerated > ago(14d)
| where ResultType != "0"
| summarize FailedSignIns = count()
by bin(TimeGenerated, 1h), UserPrincipalName
| summarize Min=min(FailedSignIns),
Avg=avg(FailedSignIns),
Max=max(FailedSignIns) by UserPrincipalName
Now the threshold discussion is based on observed behaviour rather than guesswork.
Test against known true positives
Every exclusion should be tested against examples of activity you still want the rule to detect.
A quieter rule that misses the attack is not a better rule.
Test against known false positives
Re-run the tuned query across historical benign examples and confirm the noise is actually reduced.
Tuning should produce measurable improvement, not just a cleaner-looking query.
Measure the rule after tuning
| Measure | Question |
|---|---|
| Alert volume | Did the change reduce repetitive noise? |
| True-positive retention | Do known malicious examples still match? |
| Entity diversity | Did the rule accidentally become too narrow? |
| Analyst outcome | Are analysts spending less time closing predictable benign incidents? |
| Detection latency | Did extra aggregation materially delay the signal? |
Common mistake — excluding the whole account
A service account is noisy from one known application, so the engineer excludes that account everywhere.
An attacker later uses the same account from a different source — and the rule is blind to it.
Common mistake — permanent temporary exceptions
A maintenance exception is added and never removed.
Time-limited automation can be safer than a permanent query exclusion for activity that is only temporarily expected.
Common mistake — giant allowlists
Large hard-coded lists make rules difficult to review and maintain.
Use a watchlist when exceptions become operational data rather than detection logic.
Common mistake — filtering first
If you exclude the activity before understanding it, you may remove the exact evidence that would have explained the compromise.
Investigate first. Tune second.
Agent Foskett investigation exercise
Five hundred alerts come from a known vulnerability scanner, eighty from a service account with a broken password, and twenty involve ordinary users from unexpected IP addresses.
- Validate that the scanner addresses are genuinely approved.
- Decide whether the scanner exception belongs in a watchlist or query logic.
- Do not automatically exclude the broken service account — fix the cause and decide whether its activity should remain detectable.
- Preserve the twenty unexpected user events for investigation.
- Re-test the tuned rule against historical true-positive examples.
Best practices
- Investigate the cause of noise before excluding it.
- Make exceptions as narrow as the evidence allows.
- Use watchlists for centrally managed exception data.
- Use automation for appropriate temporary or workflow-driven exceptions.
- Baseline before changing thresholds.
- Test true positives and false positives after meaningful tuning changes.
Agent Foskett takeaway
The goal is not fewer alerts. The goal is fewer useless alerts while preserving the evidence that matters.
Follow the noise back to its source. Understand it. Then tune it.
Related Agent Foskett learning
Continue learning
Tuning Microsoft Sentinel Analytics Rules and Reducing False Positives
Microsoft Sentinel analytics rules can be tuned using evidence-based KQL changes, thresholds, watchlists and automation rule exceptions to reduce false positives while preserving meaningful security detections.
Microsoft Sentinel Lesson 31
This Agent Foskett Microsoft Sentinel Academy lesson explains false-positive investigation, analytics query tuning, watchlist exceptions, automation rules, threshold baselining and practical methods for reducing SOC alert noise.
