Lesson 23 — Understanding Rule Frequency and Lookup Periods
A scheduled analytics rule can contain excellent KQL and still miss activity — or detect the same activity more than once — if its timing is poorly designed.
Two settings control the basic query window: Run query every and Lookup data from the last.
In this lesson, Agent Foskett follows the clock and examines how frequency, lookback, overlap and ingestion delay affect what a Microsoft Sentinel detection actually sees.
What you will learn
Understand exactly which events a scheduled rule examines each time it runs.
Learning objectives
- Explain the difference between query frequency and lookup period.
- Recognise gaps and overlapping query windows.
- Understand why ingestion delay can cause missed detections.
- Use
ingestion_time()when investigating timing problems. - Choose scheduling settings that fit the detection.
The investigation
A scheduled rule runs every five minutes and looks back five minutes.
The KQL is correct. A suspicious event definitely occurred. Yet no alert appeared.
Agent Foskett's first question is not, “What's wrong with the query?” It is, “When did the event actually arrive?”
The two clocks
| Setting | Controls | Example |
|---|---|---|
| Run query every | How often Sentinel executes the scheduled query. | Every 15 minutes |
| Lookup data from the last | How far back in time each execution examines. | Last 30 minutes |
For scheduled analytics rules, Microsoft currently allows both settings from 5 minutes to 14 days. The query interval must be shorter than or equal to the lookup period.
Scenario 1 — Matching windows
With no ingestion delay, this creates clean adjacent windows. Each period is examined once.
Scenario 2 — Overlapping windows
When the lookup period is longer than the query frequency, consecutive executions examine some of the same time range.
Overlap can be intentional, but the detection must account for the possibility that the same activity is returned more than once.
Why use overlap?
Security data does not always arrive immediately. Network delays, source processing and ingestion pipelines can cause an event to reach the workspace after its event time.
A longer lookup period can help catch late-arriving evidence.
The trade-off
Increasing the lookup period can improve coverage, but overlapping windows can also produce duplicate results.
The answer is not simply “make the window bigger.” The query needs to understand which events belong to the current execution.
TimeGenerated is not ingestion time
TimeGenerated normally represents when the event occurred or was generated for the record.
ingestion_time() tells you when the record was ingested into the relevant table. Comparing the two can expose latency between the event and its availability to Sentinel.
- 1
- 2
- 3
- 4
CommonSecurityLog | extend IngestionTime = ingestion_time() | extend IngestionDelay = IngestionTime - TimeGenerated | project TimeGenerated, IngestionTime, IngestionDelay
The missed-event problem
This is why ingestion timing matters. The event existed, but it was not available when the first query needed it.
Sentinel's built-in delay
Microsoft Sentinel runs scheduled analytics rules on a five-minute delay from their scheduled time to help account for ingestion latency while maintaining coverage.
That helps, but data sources can still have their own latency characteristics that detection engineers need to understand.
Measure before tuning
Do not guess how late your data arrives.
Measure the delay for the actual table and data source, especially when the detection joins multiple tables with different ingestion characteristics.
Handling known ingestion delay
One approach documented by Microsoft is to extend the event-time lookback to include the known ingestion delay, then restrict results using ingestion_time() so the overlap does not repeatedly return the same records.
- 1
- 2
- 3
- 4
- 5
let ingestion_delay = 2min; let rule_look_back = 5min; CommonSecurityLog | where TimeGenerated >= ago(ingestion_delay + rule_look_back) | where ingestion_time() > ago(rule_look_back)
The extended TimeGenerated window helps include late records. The ingestion-time condition limits the results to records that became available during the current rule window.
Frequency affects detection latency
A rule that runs every five minutes evaluates new activity more frequently than a rule that runs hourly.
That does not mean every detection should use the shortest interval. Frequency should match the threat, data availability, query design and operational need.
Lookback affects context
Some detections need a longer observation period to become meaningful.
Ten failed sign-ins in five minutes and ten failed sign-ins across twelve hours describe very different behaviour. The lookup period is part of the detection logic.
Longer isn't automatically better
A large lookback can increase query work, repeat previously examined data and change the meaning of thresholds or aggregations.
Use the smallest window that reliably captures the behaviour and any known ingestion delay.
Shorter isn't automatically faster
A very frequent rule cannot detect evidence that has not arrived yet.
Fast scheduling without understanding ingestion can create a detection that looks responsive on paper but has blind spots in practice.
Agent Foskett timing checklist
| Question | Why it matters |
|---|---|
| How quickly must this behaviour be detected? | Helps choose the query frequency. |
| How much history does the logic need? | Helps choose the lookup period. |
| How late does this data normally arrive? | Identifies potential ingestion gaps. |
| Do query windows overlap? | Highlights duplicate-result risk. |
| Does the query aggregate events? | Overlap can change counts and thresholds. |
| Are multiple tables involved? | Each source can have different ingestion latency. |
Investigation exercise
A rule runs every 10 minutes and looks back 10 minutes. The event occurred at 14:08, but the record was not ingested until 14:12. The 14:10 execution could not see it. By the 14:20 execution, its event time is outside the 14:10–14:20 window.
Before changing the detection logic, investigate the difference between TimeGenerated and ingestion_time(). The failure may be timing, not KQL.
What about NRT rules?
Near-real-time analytics rules are designed for more responsive detection and run at one-minute intervals.
They are a different rule type with their own behaviour and constraints. We will investigate NRT rules separately later in Module 3.
Start running
Scheduled rules can run automatically after creation, and Microsoft also provides a preview option to specify a future first execution time.
This can help align the first run with expected data availability or operational timing.
Common mistake
An analyst sees missed events and doubles the lookup period without considering duplication.
The detection now catches the late event — but may also return events that were already evaluated in the previous run.
Another common mistake
The query uses a count threshold, but nobody tests what overlapping windows do to that count.
Scheduling is part of the detection. Treat it with the same care as the KQL.
Agent Foskett takeaway
Query frequency determines when Sentinel looks. The lookup period determines how far back it looks. Ingestion delay determines whether the evidence is actually there when Sentinel looks.
Related Agent Foskett learning
Continue learning
Microsoft Sentinel Rule Frequency and Lookup Periods
Microsoft Sentinel scheduled analytics rules use query frequency and lookup periods to control how often detections execute and how much historical data each execution examines. Understanding overlap and ingestion delay helps prevent missed or duplicated detections.
Microsoft Sentinel Lesson 23
This Agent Foskett Microsoft Sentinel Academy lesson explains query scheduling, lookback windows, ingestion latency, overlapping detection windows and practical timing considerations for scheduled analytics rules.
