Agent Foskett Academy • Lesson 127 • Advanced KQL

Advanced KQL: Advanced Regular Expressions.

The indicator was not obvious.

It was hidden inside thousands of characters: a filename, a command line, a URL, an encoded payload and a trail of suspicious text.

Most analysts saw noise.

Agent Foskett saw a pattern. Sometimes investigations are not about finding more data. They are about recognising the pattern hidden inside it.

Start Lesson 127 Back to Academy

Agent Foskett Academy lesson advanced regular expressions in KQL for Microsoft Defender XDR

Lesson overview

Learn how to use advanced regular expressions in KQL to identify suspicious patterns, extract indicators and improve threat hunting precision across Microsoft Defender XDR telemetry.

Use matches regex for complex pattern matching

Extract indicators from command lines and URLs

Understand anchors, groups, quantifiers and classes

Build precise Microsoft Defender XDR hunts

Why advanced regular expressions matter

Regular expressions allow analysts to recognise precise text patterns inside large, noisy fields such as ProcessCommandLine, RemoteUrl, AdditionalFields, email subjects and alert evidence.

Hidden indicatorsAttackers often hide evidence inside long command lines, strange filenames, encoded payloads, URLs and script content.

Precision huntingRegex allows analysts to match exactly the pattern they are looking for instead of relying on broad text searches.

Reduce false positivesWell-designed patterns can narrow noisy results and help investigators focus on events that actually matter.

The regular expression mindset

Do not start with the most complex expression possible. Start with the pattern you are trying to prove, then make the expression as simple and precise as the investigation allows.

Match before you extractUse matches regex when the question is whether a field contains a suspicious pattern.

Extract when you need evidenceUse extract() when you need to pull the matching value into its own investigation column.

Prefer simple operators firstIf contains, has_any or startswith solves the problem clearly, do not force regex into the query.

Escape special charactersDots, brackets, slashes and other regex symbols may need escaping when they should be treated as literal text.

Think about performanceRegex is powerful, but expensive patterns over huge datasets can slow investigations. Filter early before using regex.

Document intentA readable pattern with a clear comment is better than a clever expression nobody wants to maintain.

Step 1 — Use matches regex for complex text patterns

Start with matches regex when you need to find a pattern rather than a simple word.

let TimeFrame = 7d;
DeviceProcessEvents
| where Timestamp > ago(TimeFrame)
| where FileName in~ ("powershell.exe", "pwsh.exe")
| where ProcessCommandLine matches regex @"(?i)powershell.*encoded(command)?"
| project Timestamp,
          DeviceName,
          InitiatingProcessAccountName,
          FileName,
          ProcessCommandLine
| order by Timestamp desc

Step 2 — Use anchors to control where a match occurs

The ^ and $ anchors help analysts match the beginning or end of a string, which is useful for filenames, extensions and exact patterns.

let TimeFrame = 14d;
DeviceProcessEvents
| where Timestamp > ago(TimeFrame)
| where FileName matches regex @"(?i)^(powershell|pwsh|cmd)\.exe$"
| project Timestamp,
          DeviceName,
          FileName,
          ProcessCommandLine,
          InitiatingProcessFileName
| order by Timestamp desc

Step 3 — Use character classes for controlled matching

Character classes allow analysts to define allowed characters, such as letters, numbers, hyphens and dots.

let TimeFrame = 7d;
DeviceNetworkEvents
| where Timestamp > ago(TimeFrame)
| where isnotempty(RemoteUrl)
| where RemoteUrl matches regex @"(?i)[a-z0-9-]{8,}\.(top|xyz|click|zip)$"
| project Timestamp,
          DeviceName,
          InitiatingProcessFileName,
          RemoteUrl,
          RemoteIP
| order by Timestamp desc

Step 4 — Use quantifiers to find suspicious repetition

Quantifiers help match repeated characters, long random strings and suspiciously structured values.

let TimeFrame = 14d;
DeviceFileEvents
| where Timestamp > ago(TimeFrame)
| where FileName matches regex @"(?i)^[a-z0-9]{10,}\.(exe|dll|scr|js|vbs)$"
| project Timestamp,
          DeviceName,
          FolderPath,
          FileName,
          SHA256,
          InitiatingProcessFileName
| order by Timestamp desc

Step 5 — Extract indicators with capture groups

Use capture groups with extract() when the matching value needs to become a new column.

let TimeFrame = 7d;
DeviceProcessEvents
| where Timestamp > ago(TimeFrame)
| where ProcessCommandLine has "http"
| extend ExtractedUrl = extract(@"(https?://[^\s'\"]+)", 1, ProcessCommandLine)
| where isnotempty(ExtractedUrl)
| project Timestamp,
          DeviceName,
          FileName,
          ProcessCommandLine,
          ExtractedUrl
| order by Timestamp desc

Step 6 — Hunt encoded PowerShell more precisely

Regex can identify encoded command switches even when attackers vary the spelling or spacing.

let TimeFrame = 30d;
DeviceProcessEvents
| where Timestamp > ago(TimeFrame)
| where FileName in~ ("powershell.exe", "pwsh.exe")
| where ProcessCommandLine matches regex @"(?i)(-|/)(enc|encodedcommand)\s+[A-Za-z0-9+/=]{20,}"
| extend EncodedBlob = extract(@"(?i)(?:-|/)(?:enc|encodedcommand)\s+([A-Za-z0-9+/=]{20,})", 1, ProcessCommandLine)
| project Timestamp,
          DeviceName,
          InitiatingProcessAccountName,
          ProcessCommandLine,
          EncodedBlob
| order by Timestamp desc

Step 7 — Detect suspicious double extensions

Double extensions are common in phishing and malware delivery because the filename is designed to look harmless at a glance.

let TimeFrame = 30d;
DeviceFileEvents
| where Timestamp > ago(TimeFrame)
| where FileName matches regex @"(?i)\.(pdf|docx|xlsx|jpg|png)\.(exe|scr|js|vbs|ps1|hta)$"
| project Timestamp,
          DeviceName,
          FolderPath,
          FileName,
          SHA256,
          InitiatingProcessFileName
| order by Timestamp desc

Step 8 — Extract email domains from sender fields

Regex can help normalise email addresses and extract domains for summarisation and scoping.

let TimeFrame = 30d;
EmailEvents
| where Timestamp > ago(TimeFrame)
| where SenderFromAddress matches regex @"(?i)^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}$"
| extend SenderDomain = tostring(extract(@"(?i)@([A-Z0-9.-]+\.[A-Z]{2,})$", 1, SenderFromAddress))
| summarize Messages = count(), Recipients = dcount(RecipientEmailAddress) by SenderDomain
| order by Messages desc

Step 9 — Combine parse_json() and regex

Parse structured fields first, then apply regex to the extracted property instead of searching the entire raw JSON blob.

let TimeFrame = 14d;
DeviceEvents
| where Timestamp > ago(TimeFrame)
| where isnotempty(AdditionalFields)
| extend ParsedFields = parse_json(AdditionalFields)
| extend Command = tostring(ParsedFields.CommandLine)
| where Command matches regex @"(?i)(downloadstring|invoke-webrequest|iwr|iex)"
| project Timestamp,
          DeviceName,
          ActionType,
          Command,
          AdditionalFields
| order by Timestamp desc

Step 10 — Build a reusable regex hunting pattern

Once a pattern is trusted, wrap it in a reusable function so the same logic can support multiple investigations.

let SuspiciousCommandLinePattern = @"(?i)(encodedcommand|downloadstring|invoke-webrequest|\biwr\b|\biex\b|frombase64string)";
let SuspiciousCommandLines = (Lookback:timespan) {
    DeviceProcessEvents
    | where Timestamp > ago(Lookback)
    | where ProcessCommandLine matches regex SuspiciousCommandLinePattern
    | project Timestamp,
              DeviceName,
              FileName,
              InitiatingProcessAccountName,
              ProcessCommandLine
};
SuspiciousCommandLines(7d)
| order by Timestamp desc

Common regex mistakes

Regex is powerful, but the wrong expression can create noisy results, missed evidence or slow hunting queries.

Making patterns too broadA pattern such as .*powershell.* can match too much and make it difficult to separate normal activity from attacker behaviour.

Forgetting case sensitivityUse (?i) when the same pattern may appear in different uppercase or lowercase forms.

Not escaping special charactersA dot means any character in regex. Use \. when you mean a literal dot, such as in a filename extension.

Extracting the wrong groupCapture group numbers matter. If extract() returns the wrong value, check which parentheses are being captured.

Running regex too earlyFilter by time, table, device, user or simple keywords first, then apply regex to the smaller dataset.

Ignoring readabilityA regex that nobody can understand will be hard to trust during a real incident.

Regex investigation checklist

Use this checklist before adding regex to a Defender XDR hunting query.

Is regex actually needed?Check whether contains, has, has_any, startswith or endswith can solve the problem more simply.

Have I filtered first?Apply timeframe, table and high-value filters before expensive pattern matching.

Have I escaped special characters?Review dots, brackets, slashes, backslashes, question marks and plus signs before trusting the result.

Am I matching or extracting?Use matches regex to test a pattern. Use extract() when you need to create a new evidence column.

Should the pattern ignore case?Use (?i) when attacker-controlled text may use unpredictable casing.

Can this become reusable?If the pattern is useful across multiple hunts, turn it into a named function or reusable let statement.

Related Agent Foskett Academy lessons

These lessons support advanced regex hunting and precision evidence extraction.

Lesson 122 — Advanced KQL: Working with Dynamic DataDynamic fields often need to be converted before applying regex to useful text values.

Lesson 123 — Advanced KQL: Mastering mv-expandUse mv-expand when regex needs to inspect individual values inside arrays.

Lesson 124 — Advanced KQL: Optimising Query PerformanceRegex can be expensive, so performance-aware filtering is essential.

Lesson 125 — Advanced KQL: Building Reusable KQL FunctionsWrap trusted regex patterns into reusable functions for consistent hunting.

Lesson 126 — Advanced KQL: Advanced parse_json()Parse JSON first, then run regex against the specific property that matters.

Lesson 128 — Advanced KQL: Advanced Time Series AnalysisThe next lesson moves from text patterns into behaviour patterns over time.

Coming next

The Advanced KQL series continues by analysing how security events change over time.

Lesson 128 — Advanced KQL: Advanced Time Series AnalysisNext, Agent Foskett explores time windows, bins, trends, spikes, baselines and behavioural patterns across Microsoft Defender XDR telemetry.

Why this mattersRegex reveals patterns inside text. Time series analysis reveals patterns across activity, frequency, behaviour and investigation timelines.

Final thought

Attackers rarely announce themselves. They leave patterns.

Agent Foskett mindsetDo not search for noise. Learn the patterns that reveal suspicious commands, filenames, URLs, domains and encoded evidence.

Advanced KQL SeriesLesson 127 teaches analysts how to use regular expressions for precise Microsoft Defender XDR pattern hunting.

Develop IT. Protect IT.GEMXIT PTY LTD | GEMXIT UK LTD

Advanced KQL Advanced Regular Expressions in Microsoft Defender XDR

Agent Foskett Academy Lesson 127 teaches defenders how to use advanced regular expressions in KQL for Microsoft Defender XDR investigations, including matches regex, extract(), anchors, character classes, quantifiers and capture groups.

Learn regex hunting with KQL and Microsoft Defender XDR

Advanced regular expressions help security analysts identify encoded PowerShell, suspicious filenames, command-line patterns, URLs, email domains and hidden indicators across Defender XDR telemetry.

Microsoft Defender XDR regex tutorial for threat hunting

This lesson explains how to use regex in KQL to match complex strings, extract investigation evidence, reduce false positives and build reusable pattern matching logic for security operations.