Agent Foskett • Entra ID • Conditional Access Investigation

The Conditional Access Policy Was In Report-Only Mode

The policy existed.

MFA was configured. Device compliance was configured. Risk controls were configured. The named locations looked right.

On the surface, the tenant looked protected.

But when the suspicious sign-in occurred, the user was still allowed in.

The policy had evaluated the session.

It had even shown what would have happened.

But it never blocked anything.

Because the Conditional Access policy was still in Report-Only mode.

This Agent Foskett briefing investigates a real-world Microsoft Entra ID security gap: controls that were designed, documented and visible — but not actually enforcing protection.

Discuss your environment Back to Investigation Hub

Agent Foskett Microsoft Entra Conditional Access Report-Only mode investigation

Briefing summary

A Conditional Access policy in Report-Only mode can look like a control, but it does not enforce access decisions. GEMXIT investigates policies that evaluate sign-ins, show expected outcomes and still allow risky sessions through.

Review Report-Only policies

Investigate sign-ins that should have been blocked

Validate real enforcement

What happened

The tenant had Conditional Access. The policies had meaningful names. MFA and compliance checks appeared to be part of the design. Everyone assumed the protection was live. But the logs told a different story.

The policy was configured The Conditional Access policy existed, targeted users and showed expected decisions during sign-in evaluation.

It was not enforcing The policy was left in Report-Only mode, so the session was evaluated but still allowed to continue.

The dashboard looked safe Because the policy existed, the environment looked mature. But existence is not the same as enforcement.

The sign-in was still allowed The policy may have predicted a block, MFA prompt or compliance requirement, but Report-Only mode did not apply it.

The control was assumed active Testing phases, migrations and temporary decisions often become permanent when nobody owns the follow-up.

SigninLogs told the truth ConditionalAccessStatus and ConditionalAccessPolicies can reveal whether controls were actually enforced.

Why Report-Only mode matters

Report-Only mode is useful for testing Conditional Access changes safely. The risk appears when testing becomes forgotten production configuration.

It does not block access A Report-Only policy can show that a session would have been blocked while still allowing the user in.

It creates false confidence Security teams may believe MFA, device compliance or risk controls are active when they are only being simulated.

It hides in plain sight The policy is visible in the portal, named correctly and included in reports — but still not protecting the tenant.

First hunt: find sign-ins evaluated by Conditional Access

Start by reviewing sign-ins where Conditional Access evaluated the session. The goal is to separate policies that actually enforced controls from policies that only reported what would have happened.

SigninLogs
| where TimeGenerated > ago(7d)
| project TimeGenerated,
          UserPrincipalName,
          AppDisplayName,
          IPAddress,
          Location,
          ResultType,
          ConditionalAccessStatus,
          ConditionalAccessPolicies,
          DeviceDetail
| order by TimeGenerated desc

What to review Look at ConditionalAccessStatus and the policy details attached to the sign-in, especially where access was allowed.

Why it matters Allowed sign-ins may still show policies that would have blocked or challenged the user if enforcement was enabled.

Best next pivot Parse ConditionalAccessPolicies to identify Report-Only outcomes and compare them with the actual sign-in result.

Second hunt: look for Report-Only outcomes

Report-Only policy results can show whether a policy would have succeeded, failed or interrupted a sign-in. This is where the gap becomes visible.

SigninLogs
| where TimeGenerated > ago(7d)
| extend CAPolicies = todynamic(ConditionalAccessPolicies)
| mv-expand CAPolicies
| extend PolicyName = tostring(CAPolicies.displayName),
         PolicyResult = tostring(CAPolicies.result)
| where PolicyResult has "reportOnly"
| project TimeGenerated,
          UserPrincipalName,
          AppDisplayName,
          PolicyName,
          PolicyResult,
          ResultType,
          IPAddress,
          Location
| order by TimeGenerated desc

What to review Look for reportOnlyFailure, reportOnlySuccess and reportOnlyInterrupted results attached to allowed sign-ins.

Why it matters A Report-Only failure can indicate a sign-in that would have been blocked if the policy was actually enforced.

Best next pivot Review the policy conditions, users, groups, exclusions and grant controls before moving from testing to enforcement.

Third hunt: review policy changes and exclusions

When a policy remains in Report-Only mode, the next question is why. Was it intentionally left there, was it never reviewed, or were exclusions added to avoid disruption?

AuditLogs
| where TimeGenerated > ago(30d)
| where OperationName has_any (
    "Add conditional access policy",
    "Update conditional access policy",
    "Delete conditional access policy"
)
| project TimeGenerated,
          OperationName,
          InitiatedBy,
          TargetResources,
          Result
| order by TimeGenerated desc

What to review Check who changed the policy, when it changed, whether exclusions were added and whether the state was ever moved to enabled.

Why it matters Security drift often begins with a temporary testing state that nobody later converts into enforcement.

Best next pivot Compare policy changes with risky sign-ins, excluded users, privileged accounts and legacy authentication activity.

Where this becomes dangerous

The risk is not that Report-Only mode exists. The risk is believing it is protection. During a real attack, simulated controls do not stop live sessions.

MFA may not be enforced The policy may indicate MFA would be required, but Report-Only mode will not force the challenge.

Device compliance may not matter A non-compliant or unmanaged device may still access resources if the compliance policy is only reporting.

Risky sessions can continue A sign-in that should be blocked because of risk, location or device state may remain successful.

Executives and admins are exposed High-value users may be covered by a policy on paper, but still not protected during real authentication.

Audits become misleading A policy list can look mature while enforcement gaps remain hidden inside policy state and exclusions.

False confidence grows The most dangerous security control is the one everyone thinks is active when it is only observing.

What should organisations do?

Conditional Access should be treated as a living control set. Policies need owners, testing windows, enforcement plans and regular review.

Set a review date Every Report-Only policy should have a planned review date, owner and decision point for enforcement.

Validate with real sign-ins Use SigninLogs to confirm what the policy did, what it would have done, and what actually happened.

Review exclusions Excluded users, groups, apps and locations can create the same risk as a policy that is not enforced.

Move gradually, not blindly Report-Only mode is valuable for testing, but it should lead to staged enforcement rather than permanent simulation.

Alert on policy drift Policy state changes, exclusions and disabled protections should generate review workflows.

Align with Zero Trust Zero Trust depends on enforcement, not intention. Policies must actively evaluate and control access.

How GEMXIT approaches Conditional Access reviews

At GEMXIT, we do not simply ask whether Conditional Access exists. We validate whether it is enforcing the right controls for the right users, devices, applications and risk conditions.

We review policy state Which policies are enabled, disabled or Report-Only? Which ones protect privileged users and high-risk apps?

We validate real access SigninLogs are used to confirm whether MFA, compliance, risk and location controls are actually applying.

We reduce practical risk The goal is not to create policies for show. The goal is to ensure the business is actually protected when risky access occurs.

Final thought

Conditional Access is not protection because it exists.

It is protection when it enforces.

At GEMXIT We help organisations review Microsoft 365, Entra ID, Defender XDR, Sentinel and Conditional Access controls to identify Report-Only policies, exclusions, enforcement gaps and identity risks before they become incidents.

Agent Foskett mindset The important question is not only: “Do we have Conditional Access?”

It is: “Is Conditional Access actually enforcing?”

Continue the investigation with The MFA Was On Just Not Everywhere, Just This One Exception, The Dashboard Was Green, The Login Came Through A Trusted Device, The Session Token Never Expired, The User Was Added To A Privileged Group At 3:12AM, Identity and Access Security, Entra ID Security, Zero Trust and the GEMXIT Security Review.

Develop IT. Protect IT. GEMXIT PTY LTD | GEMXIT UK LTD