Lesson 21 — Microsoft Sentinel Data Collection Rules

Learning objectives

After completing this lesson, you should understand how Data Collection Rules support Microsoft Sentinel and Azure Monitor data ingestion.

• Explain what a Data Collection Rule is.
• Understand Azure Monitor Agent and DCR associations.
• Recognise data sources, streams, data flows and destinations.
• Understand ingestion-time filtering and transformations.
• Apply testing, cost and governance practices.

The problem this solves

Collecting every available event can create unnecessary noise, storage and cost.

DCRs provide a central way to define what supported data should be collected and where it should go.

Azure Monitor Agent

Azure Monitor Agent, commonly called AMA, collects supported guest operating system data according to associated Data Collection Rules.

The agent and DCR model replaces older approaches that relied on the legacy Log Analytics agent.

DCR associations

A Data Collection Rule association links a DCR to a resource such as an Azure virtual machine, virtual machine scale set or Azure Arc-enabled server.

The association determines which rule the monitored resource should use.

Data sources

Data sources identify the information being collected.

Examples can include Windows events, Syslog, performance counters and supported custom log scenarios.

Data streams

A stream represents the shape or category of data moving through the collection pipeline.

The DCR uses streams to connect incoming data with transformations and destinations.

Data flows

Data flows describe how one or more streams are sent to a destination.

They can also reference a transformation that filters or modifies incoming records before storage.

Destinations

Destinations define where collected data is sent.

For Microsoft Sentinel workflows, this commonly includes the Log Analytics workspace connected to Sentinel.

Data Collection Endpoints

A Data Collection Endpoint, or DCE, provides endpoints used by specific collection scenarios for configuration access or data ingestion.

Not every DCR workflow requires a dedicated DCE, so the requirement should be confirmed for the selected collection method.

Central management

DCRs are Azure resources that can be deployed and managed centrally.

This supports consistent configuration across groups of servers and makes collection settings easier to audit.

Ingestion-time transformations

Transformations use supported KQL logic to filter or modify incoming records before they are stored in the Log Analytics workspace.

This can reduce noise, remove unnecessary records or shape fields for the destination table.

Filtering data

Filtering at ingestion can prevent low-value records from being stored.

Because filtered data is no longer available for later investigation, every exclusion should have a documented reason and testing evidence.

Transforming fields

A transformation can add, remove or modify supported fields before ingestion.

The output must match the expected schema for the destination table.

Workspace transformations

Workspace transformation DCRs provide ingestion-time transformations for supported workflows that do not use a source-specific DCR.

They should be reviewed carefully because they can affect data arriving from multiple sources.

Cost considerations

DCRs can help reduce unnecessary ingestion by selecting or filtering data before storage.

Cost reduction should never remove telemetry required for detection, compliance, investigations or incident response.

Security coverage

Before excluding a data source or event category, identify the analytics rules, hunting queries and investigations that depend on it.

Lower volume is not useful if critical attack evidence disappears.

Testing

Test a DCR against controlled resources before broad deployment.

Confirm event coverage, table schema, transformations, ingestion delay and downstream detections.

Troubleshooting

• Confirm the agent is healthy.
• Check the DCR association.
• Review source configuration.
• Validate the transformation output.
• Confirm the destination workspace and table.

Change control

DCR changes can affect detection visibility and ingestion cost across many resources.

Use version control, approvals, testing and a rollback plan for production changes.

Ownership

Assign an owner for each production DCR.

The owner should understand the data source, dependent Sentinel content and business reason for the collection settings.

Common mistake

A common mistake is filtering data only because it appears noisy.

Some high-volume events provide essential context during identity, endpoint and lateral movement investigations.

Another common mistake

Do not assume that creating a DCR automatically applies it to every resource.

For agent-based collection, confirm the required associations and resource scope.

Best practices

• Start with a defined security use case.
• Collect only what has operational value.
• Test transformations with representative data.
• Document associations, owners and dependencies.
• Monitor ingestion volume, health and detection impact.

Agent Foskett takeaway

Data Collection Rules give security teams deliberate control over supported Azure Monitor collection workflows.

Good DCR design balances visibility, performance and cost without removing the evidence analysts will need later.