Office 365 User Account Security Assessment Deep Dive

Overview

The Office 365 User Account Security Assessment was built to combine information available across the Office 365 and Azure environments along with third-party APIs to provide a unified view of security information for customers. The template produces an Excel report that contains not only a quick dashboard, but the underlying raw data that can be used to create your own insights, whether in Excel, Power BI, or any other tool.
Information included:

• Counts
  • Total Accounts
  • Blocked Accounts
  • Inactive Accounts (configurable)
  • Stale Passwords
  • Accounts with Failed Logins
• Charts
  • MFA Adoption
  • Password Expiration Exemption
• Tables
  • Successful & Failed Login by Country
• Underlying Data
  • User Security Information
  • User Data Quality Information
  • Failed Login Location
  • HaveIBeenPwned Breach Data

While you can use a Global Administrative account to run the assessment, we recommend creating a dedicated account that has the minimum permissions required for each template. This will ensure that if an account is compromised it has the least permissions assigned. It will also ensure that audit logs can easily be searched and actions attributed to the exact template that executed them.

Minimum Permissions

To run the assessment with the minimum permissions, you’ll need an account that has been granted the following roles:

• Office 365
  • Global Reader
• Exchange
  • Custom admin role with only the User Options permission assigned

To quickly create an account with the appropriate permissions, you can use the following sample script.

Sample Script

This script will create a dedicated account in Office 365 and grant it Global Reader permissions. Run this script in an elevated PowerShell session so that the script will be able to install the MSOnline and ExchangeOnline PowerShell modules, if they aren’t already installed. The script will prompt for administrative credentials twice, first for Office 365 and second for Exchange Online.

if (-not (get-module MSOnline)){ Install-Module MSOnline }
if (-not (Get-Module ExchangeOnline)){ Install-Module ExchangeOnline }
Import-Module MSOnline
Import-Module ExchangeOnline
Connect-MsolService
Connect-ExoService
$domain = (Get-MsolDomain | Out-GridView -OutputMode Single).Name
$password = (1..16 | % { ((33..126 |%{[char]$_}) -join "")[(Get-Random -Max 93)]}) -join ""
New-MsolUser -DisplayName "Voleer - Office 365 User Account Security Assessment" -UserPrincipalName "voleer-o365UASA@$domain" -ForceChangePassword $false -PasswordNeverExpires $true -StrongPasswordRequired $false -Password $password
Add-MsolRoleMember -RoleName "Global Reader" -RoleMemberObjectId (get-msoluser -UserPrincipalName "voleer-o365uasa@$domain").objectid
Write-Host "UserName: voleer-o365UASA@$domain`nPassword: $password" -ForegroundColor Red

Leave the PowerShell window open.  You’ll need to temporarily assign a license that includes Exchange Online to replicate the account to the Exchange environment. Once assigned, the account will be visible to Exchange and can be granted privileges. When the license is assigned, run this final command:

New-RoleGroup -Description "Created for Voleer - Office 365 User Security Account Assessment" -DisplayName "Voleer - Inbox Rules Reader" -Members "voleer-o365uasa@$domain" -Name "Voleer - Inbox Rules Reader" -Roles "User Options"

This will create a custom role called Voleer – Inbox Rules Reader and give it the User Options permission set. After it completes successfully, you can remove the Office 365 license.

Enable Unified Audit Logging

One of the powerful features this assessment offers is data about successful and failed login attempts that has been geo-located so that you can view login information by country. To enable this functionality, Unified Audit Logging must be enabled in your Office 365 tenant. If this logging is not enabled, the report will complete successfully but will not contain any login data.

To enable Unified Audit Logging:

1. Sign into the Security & Compliance Center (http://aka.ms/Security-and-Compliance) with your Office 365 admin account.
2. Select Search & Investigation, and then select Audit log search.
3. Select Start recording user and admin activity. If you don't see this link, auditing has already been turned on for your organization. A message alerts you that the audit log is being prepared.

Note that any login activity before Unified Audit Logging is enabled may not be available.

Running the Template

The Office 365 User Account Security Assessment is the first Voleer Template to use our Easy-Onboarding and Recurrence user experience. The first time the template is run, it saves required input for future runs.

1. Enter the account credentials
2. Enter an email address to send the report to
3. Enter a password to protect the Excel file
4. Click Start Assessment.

On subsequent runs in the same Voleer workspace, you’ll see the Select Template Mode screen. There are three options:

Report Walkthrough

The Office 365 User Account Security Assessment consists of a dashboard overview and four additional sheets with the raw information used to produce the report.

Dashboard

The dashboard surfaces a number of important metrics. For each metric, we’ll discuss its significance and possible business cases that it could be used in.

Total Number of Accounts

This metric is useful in combination with the others to provide context to the information provided. In addition, knowing this information can help you estimate the duration of many other templates in Voleer that depend on the size of the Office 365 environment.

Accounts Involved in Breaches

Voleer checks user accounts against the HaveIBeenPwned API to determine if any accounts were involved in public website breach data dumps. If this metric is above 0, the details of the breach will be contained on the All Breaches worksheet. Keep in mind, not all data breaches are alike.

Data breaches that involve compromised plain-text password data have obvious ramifications when business accounts are used. Studies indicate that many users re-use the same password over many accounts, so if a corporate email is used for a third-party service, the chances that the credentials can be successfully used to access a customer’s environment are high. Others may reveal phone numbers or email contacts that can be used in social engineering attacks, allowing hackers to engage in spear-fishing attacks that appear to come from the compromised user.

By combining multiple pieces of compromised data, an ingenious attacker may be able to compromise MFA by impersonating IT staff, calling a user directly with spoofed caller-id information, and convincing the user to provide their temporary MFA code.

To investigate the details of what information was compromised in a breach, visit www.haveibeenpwned.com.

Blocked Accounts

User accounts in Azure can be blocked for several reasons, the simplest of which is for multiple failed login attempts. Azure AD can also block a user account due to a sign-in risk policy or a user risk policy. These policies vary from customer to customer and a thorough understanding of why a particular account is blocked is beyond the scope of this guide.

Proactive monitoring of blocked user accounts can help identify compromised accounts and can help IT staff mitigate blocked user accounts before a support case is opened by the end user.

Stale Accounts

To minimize the attack surface of an organization, it is a best practice to disable any user accounts that are not actively being used and deprovision accounts that are no longer needed. The report helps you identify accounts that have not logged-in in a set period of time. This information may help identify process gaps in user-offboarding solutions. If an account has not logged in for longer than the password expiration, the account should be reviewed for deprovisioning.

Accounts with Stale Passwords

Unless a company is on the bleeding edge of the new “passwordless” paradigm, knowing what accounts have not changed their passwords recently is a must-have. Password expiration policies help reduce the risk posed by accounts with aged passwords, but there are some accounts that are exempted from password policies. Commonly these accounts are used for services or other automation accounts.

Accounts with Failed Login Attempts

Everyone types their password wrong occasionally, so this metric is seldom useful in itself. However, if a majority of the user accounts are represented here, it may indicate that password policies are too strict or that a deeper dive into the logs is warranted to determine if malicious activity is to blame.

MFA vs Non-MFA Enabled Account Chart

This chart gives a quick heads-up view of the MFA status of an organization. This chart can be especially helpful when performing a staged rollout of MFA. By running this assessment on a regular basis throughout a rollout, you can provide an easy view into the adoption of MFA across an organization. After rollout is complete, this chart can help reveal any exempted accounts or accounts that have not enrolled during the rollout period.

Password Expiration Policy Chart

This chart shows the percentage of accounts that are subject to, or exempted from, a password expiration policy. While there are valid use cases for accounts being exempted from password expiration policies, it is important to monitor these accounts and develop a strategy for occasional password refreshes.

Administrative Roles Chart

This chart quickly shows how many users have administrative roles assigned. Having too many accounts with standing privileged access can represent a security risk to organizations.

External Forwarding Inbox Rules

This metric displays the number of users that have Exchange inbox rules set up to forward email to external accounts. Automatically forwarding emails outside the bounds of an organization can be used by attackers to “wire-tap” accounts, can expose private information, and can open an organization to legal liabilities. Any accounts with these rules should be audited for compliance with company policy and any applicable legal requirements.

Full Access Permissions

These two metrics give visibility into accounts that grant or are granted full access permissions to an exchange mailbox.

The first box contains the number of accounts that grant these permissions. While all such accounts should be investigated, a high number here may indicate a malicious user has created an account that can be used to access other users’ mailboxes.

The second box is a mirror of the first: it shows how many accounts have been granted full access to at least one mailbox other than their own.

Failed and Successful Login Tables

The final two elements of the dashboard view are tables that show the successful and failed login counts by country. This information can be used to identify anomalies and aid in tuning Azure AD Conditional Access Policies.

Data Worksheets

While dashboards are great for a quick overview, being able to dig into the data behind the metrics is important. The User Account Security Assessment includes a number of additional worksheets that contain the processed information used to create the report.

User Security Information

The User Security Information worksheet contains much of the data that the dashboard is based upon. The worksheet contains the following data for all user accounts:

• User Principal Name
• Display Name
• Admin Roles (comma separated)
• Is MFA Enabled?
• Days Since Last Password Change
• Does the Password Expire?
• Is the Account Blocked?
• Days Since Last Login
• Successful Login Locations in the Last 30 Days
• Failed Login Locations in the Last 30 Days
• Verified Account Breaches from HaveIBeenPwned.com
• Days Since Last Account Breach
• External Forwarding Inbox Rules
• Has Full Access Permissions to Mailboxes
• Grants Full Access Permissions to Mailboxes

User Data Quality Information

As an added benefit, the User Data Quality Information worksheet contains the following data for all user accounts:

• User Principal Name
• Display Name
• Licenses
• Created Date
• Title
• Mobile Phone
• Mobile Phone Format
• Phone Number
• Phone Number Format
• Postal Code
• Postal Code Format

This information is included for two reasons. First, it aids in maintaining proper AD hygiene. Second, it provides contact information for users so that technicians can quickly contact the owner of accounts that are of interest, whether due to security or customer support issues. It also provides basic license information that can be used to determine if an affected account is used for Office 365, a replicated copy of an on-premise account, or an inactive account.

Failed Login Locations

To help in investigating failed logins, this worksheet provides a count of failed logins from individual IP addresses, along with their geo-located country of origin. This can help identify malicious activity as well as help identify networks that may require attention in Azure AD Conditional Access Policies or in MFA configurations.

All Breaches

The All Breaches worksheet contains information retrieved from the HaveIBeenPwned.com API. Each entry represents an account involved in a single data breach. The information included is:

• Title
• Domain
• BreachDate
• AddedDate
• UserPrincipalName
• DaysSinceBreach

A few things should be noted. The BreachDate and the AddedDate are sometimes separated by long periods of time. Many data breach information dumps are not made available publicly or on the darkweb for days, months, and sometimes years after the breach. Depending on the account, password expiration policy, and nature of the data breach, no action may be necessary. However, each breach is different and it is important to gain more information into the specifics.

To find out more about a particular breach, visit https://haveibeenpwned.com/PwnedWebsites and locate the site referenced in the Title column. The breach information will give more details about the breach and most importantly, the Compromised Data involved in the breach. As mentioned earlier, just because passwords were not involved in a data breach does not mean that no action is needed. Security questions, PII, and other information can be used in social engineering or targeted fishing attacks long after any passwords would have been changed.