Agent Foskett Academy • Microsoft Sentinel • Module 6 • Lesson 60

Lesson 60 — Monitoring and Troubleshooting SOC Automation

Learn to observe, diagnose and safely recover Microsoft Sentinel automation rules and Azure Logic Apps playbooks. Find failed runs, detect silent gaps, and prove your incident workflows are operating as intended.

Module 6 — SOC Automation and Response • Monitoring and troubleshooting
Agent Foskett Microsoft Sentinel automation monitoring and troubleshooting
What you will learn

Operate reliable, auditable SOC automation.

✓ Monitor playbook runs and rule outcomes
✓ Investigate errors with diagnostic logs
✓ Use KQL for reliability monitoring
✓ Test safe recovery and rollback

Agent Foskett case file — The playbook that went quiet

Security analysts believe every high-priority Sentinel incident receives automated enrichment. One morning, a suspicious sign-in incident arrives with no enrichment comment. The automation rule still exists, but the Logic App connection has failed. Agent Foskett follows the execution trail rather than assuming the workflow ran.

Learning objectives

  • Separate Sentinel automation rule matching from Logic Apps playbook execution.
  • Inspect trigger history, run status, connector failures and permissions.
  • Configure diagnostics and query supported Azure Monitor telemetry with KQL.
  • Distinguish no trigger, failed run, partial success and missing incident updates.
  • Design alerting, safe retries, rollback and post-change validation.

Architecture — follow the automation trail

SENTINEL INCIDENT / MATCHING EVENT ↓ AUTOMATION RULE: CONDITIONS + ORDER + ACTION ↓ PLAYBOOK TRIGGER: INVOCATION / PERMISSIONS ↓ LOGIC APP RUN: ACTIONS + CONNECTOR CALLS ↓ INCIDENT UPDATE: COMMENT / TAG / RESULT ↓ DIAGNOSTICS + ALERTS + ANALYST VERIFICATION

Prerequisites and safety boundaries

  • Complete Lessons 52–59 or have equivalent Sentinel automation experience.
  • Access a test Sentinel incident, an automation rule and a Logic Apps playbook.
  • Permission to inspect Logic Apps run history and relevant Azure Monitor diagnostic settings.
  • A Log Analytics workspace with diagnostic logs configured if using the KQL examples.
  • Use a non-destructive lab workflow that writes a test incident comment only.

Important: diagnostic tables and columns depend on Logic Apps hosting model, resource settings and collection configuration. Verify the actual schema in your workspace before using the example queries.

Step 1 — Identify the expected outcome

Define the expected trigger, incident type, run completion window and observable result. For example: “A lab incident matching Rule A should invoke Playbook B and receive one versioned enrichment comment.” Without an expected outcome, a green workflow status can be misleading.

Step 2 — Check the Sentinel automation rule

  1. Open Microsoft Sentinel automation rules in the appropriate portal experience.
  2. Verify that the rule is enabled and its conditions match the test incident.
  3. Review rule order, action configuration and expiration settings.
  4. Confirm the intended playbook is selected and permitted to run.
  5. Record the rule identifier and time of the test event.

Step 3 — Confirm playbook invocation

Inspect the Logic App trigger and run history. A missing run suggests a trigger, rule, permissions or routing problem. A recorded failed run means the playbook was invoked but encountered an error. These are different faults and should not be reported as one generic “automation failure.”

Step 4 — Inspect action-level failures

  • Open the failed run and identify the first failing action.
  • Check the HTTP or connector status, error category and dependency.
  • Review retry attempts and whether a downstream action was skipped.
  • Check managed identity and connector authorization separately.
  • Use secure handling for run inputs and outputs that may contain incident details.

Step 5 — Classify common failure modes

SymptomLikely investigation area
No Logic App runRule conditions, order, trigger, permissions
Run fails immediatelyConnector auth, malformed input, disabled resource
Run times outAPI latency, query volume, retry policy
Run succeeds, no incident commentSkipped branch, wrong incident, action output
Duplicate commentsRetriggers, overlapping rules, idempotency
Intermittent errorsThrottling, transient dependency failures

Step 6 — Enable supported diagnostic collection

Use Azure Monitor diagnostic settings for the relevant Logic Apps resource and route available workflow runtime logs to an approved Log Analytics workspace. The exact diagnostic categories and tables vary between Logic Apps Consumption and Standard. Confirm the selected categories are generating records before relying on a dashboard or alert.

Step 7 — Discover the real telemetry schema

Before copying a monitoring query, open Logs and inspect the tables populated by your Logic Apps diagnostic configuration. The following KQL examples target the legacy AzureDiagnostics table only if your resource sends workflow runtime diagnostics there. In some configurations, resource-specific tables or Application Insights may be used instead.

Step 8 — KQL: discover Logic Apps diagnostic activity

Azure Monitor Logs — AzureDiagnostics discovery
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
AzureDiagnostics
| where TimeGenerated >= ago(24h)
| where ResourceProvider =~ "MICROSOFT.LOGIC"
| summarize Records = count(),
            FirstSeen = min(TimeGenerated),
            LastSeen = max(TimeGenerated)
    by Resource, Category
| order by Records desc

Run this in the diagnostic workspace, not Defender XDR advanced hunting. No rows may mean that diagnostics are not configured, are routed elsewhere or have not yet produced records.

Step 9 — Read diagnostics without assuming success

A record showing workflow execution is not the same as a verified incident update. Match a run to its trigger and inspect the expected downstream action. Use the Logic Apps portal run details to confirm failures, skipped steps and retries.

Step 10 — KQL: find recent workflow errors

Azure Monitor Logs — workflow error search
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
AzureDiagnostics
| where TimeGenerated >= ago(24h)
| where ResourceProvider =~ "MICROSOFT.LOGIC"
| extend RawStatus = tostring(column_ifexists("status_s", "")),
         RawLevel = tostring(column_ifexists("level_s", ""))
| where RawStatus in~ ("Failed", "Faulted", "TimedOut")
    or RawLevel =~ "Error"
| project TimeGenerated, Resource, Category,
          RawStatus, RawLevel, OperationName,
          CorrelationId
| order by TimeGenerated desc
| take 200

This is an exploratory query for the AzureDiagnostics schema. Field names and status values vary; inspect your records before treating it as a complete failure detector. The query deliberately avoids exposing raw incident payloads.

Step 11 — Build a run-level investigation timeline

Correlate the Sentinel incident identifier, automation rule, Logic App run identifier, connector actions and resulting incident comment. Use timestamps in UTC and preserve the difference between a triggering event and the moment the run completed.

Step 12 — KQL: trend diagnostic volume

Azure Monitor Logs — diagnostic trend
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
AzureDiagnostics
| where TimeGenerated >= ago(7d)
| where ResourceProvider =~ "MICROSOFT.LOGIC"
| summarize DiagnosticRecords = count()
    by Resource, bin(TimeGenerated, 1h)
| render timechart

This chart shows diagnostic record volume, not workflow success rate. An increase can reflect more actions or logging changes; a drop may indicate no workload or a telemetry gap.

Step 13 — Detect missing executions

Monitoring only failed runs misses workflows that never start. Establish an expected test frequency or compare known eligible incidents with observed playbook runs. If a scheduled lab canary is used, alert when its expected run or incident comment is absent within the agreed window.

Step 14 — Choose operational signals

  • Expected playbook invocation observed for eligible incidents.
  • Run completion state and action failures.
  • Time from incident event to enrichment comment.
  • Connector authorization and throttling errors.
  • Unexpected repeated runs or duplicate comments.
  • Diagnostic pipeline freshness and ingestion health.

Step 15 — Alert on meaningful failures

Use Azure Monitor alerts or an approved monitoring service based on the telemetry actually available. Set a sensible evaluation period, action group and owner. Separate transient connector errors from sustained failures, and avoid routing alert notifications back into a playbook loop.

Step 16 — Troubleshoot permissions

Check the Sentinel automation rule’s ability to invoke the playbook, the Logic App identity, and each connector’s data or incident permissions independently. A successful trigger does not prove the Log Analytics query or incident comment action has authorization.

Step 17 — Diagnose throttling and timeouts

Inspect connector status codes, dependency latency and retry history. Reduce excessive loops and query result sizes. Use bounded exponential backoff only for retryable failures; do not blindly retry destructive or non-idempotent actions.

Step 18 — Guard against duplicate updates

Repeated incidents, changed incident records and overlapping rules can cause multiple runs. Use a workflow version marker and a tested idempotency strategy. Verify that the monitoring playbook itself does not create a recursive incident-update loop.

Step 19 — Build a safe recovery runbook

SOC AUTOMATION RECOVERY RUNBOOK 1. IDENTIFY: affected rule, playbook and incident 2. CONTAIN: pause faulty automation if authorized 3. PRESERVE: run ID, error, timestamp and inputs 4. DIAGNOSE: rule → trigger → action → outcome 5. REPAIR: permission, schema, dependency or limit 6. VALIDATE: rerun controlled test incident 7. RESTORE: enable with change approval 8. MONITOR: confirm expected incident result

Never replay a workflow against production incidents without confirming whether earlier actions already succeeded.

Step 20 — Document partial success

A workflow can enrich an incident and later fail when writing a second comment. Record the actions that completed, those skipped and any uncertain outcomes. Avoid reporting a binary “success” when the intended operational result was only partly delivered.

Step 21 — Test monitoring scenarios

  1. Healthy incident-triggered run and expected comment.
  2. Disabled automation rule producing no run.
  3. Deliberately invalid lab connector configuration.
  4. Query timeout or bounded retry condition.
  5. Playbook with a skipped comment action.
  6. Repeated incident update and duplicate-run check.
  7. Diagnostic collection temporarily unavailable.

Restore the test configuration after each scenario and confirm the monitoring signals reflect the actual outcome.

Agent Foskett case file — The missing enrichment

The rule was enabled, but the Logic App run history revealed repeated authentication failures after a connector change. The existing monitoring dashboard only counted created incidents, so no alert fired. Agent Foskett added a run-failure signal, an expected-outcome check and a recovery procedure. The next lab incident received its enrichment comment as expected.

Finding: an enabled rule is not evidence of a completed investigation workflow.

Hands-on challenge

  1. Identify one lab automation rule and its Logic App.
  2. Document the expected trigger and incident result.
  3. Inspect a healthy run and record its run ID.
  4. Configure supported diagnostic collection if authorized.
  5. Run the three KQL examples against the correct table.
  6. Simulate a safe failure and trace the first failing action.
  7. Design one failed-run alert and one missing-execution check.
  8. Write and test a rollback-and-recovery checklist.

Operational quality checklist

  • Automation rule, trigger and playbook action outcomes separately verified.
  • Diagnostic table and columns checked for the actual hosting model.
  • All three KQL editors full width with line numbers.
  • Run IDs, UTC timestamps and incident identifiers recorded.
  • Missing runs, failed runs and partial success distinguished.
  • Alerts have owners and avoid recursive workflows.
  • Retries are bounded; non-idempotent actions protected.
  • Safe rollback and post-change verification documented.

Agent Foskett takeaway

Automation you cannot observe is automation you cannot trust. Monitor the rule, the run and the incident outcome — then make sure the recovery path is as carefully designed as the happy path.

The logs already knew. This time, Agent Foskett checked whether the logs arrived.

Lesson 60 complete — SOC automation reliability!
You can now monitor, troubleshoot and safely recover Sentinel automation and Logic Apps playbooks.
Sentinel Academy Home

Related Agent Foskett learning

Review the investigation playbooks that this monitoring workflow helps keep reliable.