So, what is Report-only mode?
Report-Only mode is a new option within a Conditional Access policy. Besides the option to turn the conditional access policy on or off, the option to Report-only has been added.
By enabling the Report-only mode the conditional access is evaluated on the client instead of enforced. Now you might ask yourself, wasn’t this functionality not already available by using the What if analysis tooling. The difference is that the What if tooling only moddeled the applicability of the Conditional Access policy based on the given scenario.
How does Report-only mode work and look?
After the conditional access policy is created, with Report-only mode selected, we have several options to determine the outcome of the evaluation.
Option 1: Azure Active Directory Sign-in logging
Go to the Azure AD administration portal | Monitoring | Sign-ins and select one of the listed sign-ins. Once selected, within the sign-in logging, a new tab has been added titled: “Report-only (Preview)”. Here you can see, in this example that the “I24 – Accept User Terms” conditional access policy reports the result:”Report-only: User action required”
Option 2: Use the power of Log Analytics and the new Azure AD workbooks
If you forward your sign-in logging to a Log Analytics workspace, you can use workbooks to report on the Report-only conditional access policies.
Once the SignInLogs are sent to the Log Analytics workspace you can open the Azure AD workbooks by selecting Monitoring | Workbooks.
Under Workbooks, several Workbooks are available:
- Sign-ins using Legacy Authentication
- App Consent Audit
- Conditional Access Insights (Preview)
- Sign-ins by Conditional Access
- Sign-ins by Grant Controls
- Sign-ins Failure Analysis
For this example the “Conditonal Access Insights (Preview)” workbook is of special interest. By selecting the “I24 – Accept User Terms” under Conditional Access policy you get an overview of the applicability of the Conditional Access policy. As you can see in this example the Conditional Access policy is evaluated 5 times, it will not be applied 4 times and for one scenario, a User action is required.
Caveat so be aware: Policies in report-only mode that require compliant devices may prompt users on Mac, iOS, and Android to select a device certificate during policy evaluation
While at time of writing the Report-only mode is still in preview, it is a must have option if you want to implement your own custom conditional access policies. Before enabling your conditional access policy, first use the Report-only option to test it’s outcome for your users. This can really be a lifesaver, and gives more insights compared to the whatif tooling.
I will try to update the Conditional Access whitepaper and Conditional Access implementation flowchart as soon as possible to integrate this functionality. From now on, using Report-only will be used for sure.
Microsoft docs: What is Conditional Access report-only mode?