How-To: O365 Event Ingest into LogScale via Microsoft Graph (using pre-defined CrowdStream O365 Activity/Services)

Multiple types and sets of information can be pulled from Microsoft depending on the options configured and the permissions granted to the app / assigned user used to access Microsoft data

  • Using one of our integration options like packages to ingest and parse Microsoft data, see Microsoft for more information.

  • With CrowdStrike XDR, the following options are defined as part of setting up the CrowdStrike plug-in app (Microsoft Graph API: Defender O365 & Azure AD plugin app is available in the CrowdStrike store.)

    • Microsoft Graph API Permissions to be set include:

      • For Microsoft Azure AD

        • AuditLog.Read.All

      • For Microsoft Defender for O365

        • SecurityAlert.Read.All

        • SecurityEvents.Read.All

        • SecurityIncident.Read.All

  • CrowdStream / Cribl Stream supports receiving multiple types of Microsoft data, depending on which data type and option selected:

    • Office 365 Activity

      • The Office 365 Management Activity API facilitates analyzing actions and events on Azure Active Directory, Exchange, and SharePoint, along with global auditing and Data Loss Prevention data.

      • https://docs.cribl.io/stream/sources-office-365-activity/

        • In Azure Active Directory, the application representing your Cribl Stream instance must be granted the following permissions to pull data. Each permission's Type must be Application - Delegated is not sufficient:

          • ActivityFeed.Read — Required for all Content Types except DLP.All.

          • ActivityFeed.ReadDlp — Required for the DLP.All Content Type.

    • Office 365 Services

      • The Microsoft Graph service communications API facilitates analyzing the status and history of service incidents on multiple Microsoft cloud services, along with associated incident and Message Center communications. For details, see Microsoft's Overview of the Graph API.

      • https://docs.cribl.io/stream/sources-office-365-services/

        • In Azure Active Directory, the application representing your Cribl Stream instance must be granted the following permissions to pull data. (The permission Type for both must be Application; Delegated is not sufficient:)

          • ServiceHealth.Read.All

          • ServiceMessage.Read.All

    • Office 365 Message Trace not covered in this walkthrough

      • This mail-flow metadata can be used to detect and report on malicious activity including bulk emails, spoofed-domain emails, and data exfiltration.

      • https://docs.cribl.io/stream/sources-office365-msg-trace/

        • At a minimum, your Office 365 service account should include a role with Message Tracking and View Only Recipients permissions, assigned to the Office 365 user that will integrate with Cribl Stream. Assign these permissions in the Exchange admin center (https://admin.exchange.microsoft.com).

Macro Steps to Collect O365 Details into LogScale (for example via CrowdStream)

  1. Register Microsoft application, generate secret, and add permissions

    1. Note: The Client Secret received from Microsoft Azure AD requires periodic rotation according to the expiration duration that you select.

  2. Add permissions for Microsoft Azure AD

  3. Add permissions for Microsoft Defender for O365

  4. For LogScale use case, kick-off Microsoft Content Subscription (CrowdStream/Cribl pre-req to be done in Microsoft via PowerShell or curl command)

  5. Verify successful data connection via CrowdStream / CriblStream

  6. Verify successful data ingest in LogScale

Step-by-Step Config (example for use with CrowdStream)

Create and Register a new app

  • Select App registrations

  • Click on New registration

  • Provide a name and leave the rest as default

  • Click the Register button at the bottom of the page

  • Navigate to overview and copy the Application (client) ID:

  • Select Endpoints and copy the OAuth 2.0 token:

Provide the appropriate API permissions

  • Once it is created, navigate to API permissions

  • Click on Microsoft Graph

  • You should see an additional panel to allow for selection of permissions

    Note that there are 2 types of permissions:

    • Delegated

    • Application

    • For the purpose of pulling audit logs, ensure you select Application Permissions

    • For Microsoft Azure AD permissions, search for audit and you should see it appear.

      • Click the Application permissions checkbox (Delegated is not sufficient per Cribl)

      • Select the checkbox for AuditLog.Read.All

    • For Microsoft Defender for O365, repeat the steps above for additional security events such as:

      • SecurityEvents.Read.All

      • SecurityAlert.Read.All

      • SecurityIncident.Read.All

    • For permissions related to the data collection outlined in CrowdStream docs (across MS Activity, Services, Trace), the following additional permissions can be added:

      • Office 365 Activity

        • The Office 365 Management Activity API facilitates analyzing actions and events on Azure Active Directory, Exchange, and SharePoint, along with global auditing and Data Loss Prevention data.

          • https://docs.cribl.io/stream/sources-office-365-activity/

            • In Azure Active Directory, the application representing your Cribl Stream instance must be granted the following permissions to pull data. Each permission's Type must be Application - Delegated is not sufficient:

              • ActivityFeed.Read - Required for all Content Types except DLP.All.

              • ActivityFeed.ReadDlp - Required for the DLP.All Content Type.

      • Office 365 Services

        • The Microsoft Graph service communications API facilitates analyzing the status and history of service incidents on multiple Microsoft cloud services, along with associated incident and Message Center communications. For details, see Microsoft's Overview of the Graph API.

        • https://docs.cribl.io/stream/sources-office-365-services/

          • In Azure Active Directory, the application representing your Cribl Stream instance must be granted the following permissions to pull data. (The permission Type for both must be Application - Delegated is not sufficient:)

            • ServiceHealth.Read.All

            • ServiceMessage.Read.All

      • Office 365 Message Trace not covered in this walkthrough

        • This mail-flow metadata can be used to detect and report on malicious activity including bulk emails, spoofed-domain emails, and data exfiltration.

        • https://docs.cribl.io/stream/sources-office365-msg-trace/

          • At a minimum, your Office 365 service account should include a role with Message Tracking and View Only Recipients permissions, assigned to the Office 365 user that will integrate with Cribl Stream. Assign these permissions in the Exchange admin center (https://admin.exchange.microsoft.com).

    • Location for Activity / Services permissions:

      • Under Microsoft Graph (for O365 Services info) - Service options need to be permission type Application - Delegated is not sufficient

        • ServiceHealth.Read.All

        • ServiceMessage.Read.All

        • User.Read

      • Under Office 365 Management APIs (for O365 Activity) - all need to be permission type Application - Delegated is not sufficient

        • ActivityFeed.Read

        • ActivityFeed.ReadDlp

        • ServiceHealth.Read

    • Click Update permissions button at bottom of the page

    • You should now have the same setup as below with a status that the permission is Not granted

    • Select Grant admin consent for Default Directory (your customer/prospect name may show up vs. Default Directory to grant consent

Summary Example of Broad Set of Applied Permissions for the App

Depending on the specific data to be collected, permissions for the new app will vary

In this example, permissions across both LogScale XDR pre-reqs as well as Cribl pre-reqs have been combined

Create API key to allow LogScale to access the logs

  • Navigate to Certificates & Secrets and create a new client secret

  • Click New client secret

  • Provide a description and expiry date for the API key

  • Click the Add button at the bottom of the page

  • Copy this value (immediately) as you will only see it once

Define / Configure Sources for O365 in CrowdStream
Accessing O365 Source Options

  • Log in to your CrowdStream / Cribl instance

    For example, this can be a CrowdStream instance at a customer / POV, or it can be a Cribl eval instance you signed up for (both behave the same)

  • From the main CrowdStream / CriblStream page, click on Manage Stream

  • Click on the Worker Group which will be used

    Typically in a POV / initial setup, the Worker Group will be called default

  • Click Data → Sources

  • In the upper-right of the screen, click in the Filter Sources field and search for 365

    • In a net-new setup, three results should display:

    • For this walk-through, the permissions set earlier in this doc (here) cover Office 365 Services and Office 365 Activity

      • The same app created in this doc (which has both sets of permissions needed) can be used with two different sources

Creating O365 Activity Source
Configuring the O365 Activity Source

  • After filtering on 365 for sources, click on the Office 365 Activity tile

  • In the upper-right of the page, click on "Add Source"

  • In the General Settings section, fill in the following values for the new source

    • Input ID

      Can be anything you want - just the name for the source in CrowdStream / CriblStream

      For example: o365_activity_internal_lab

    • Tenant ID

      The Office 365 Azure Microsoft tenant ID for your environment

    • App ID

      The Office 365 Azure Application ID for the newly created Microsoft app

    • Subscription plan

      By default, using Office 365 Enterprise

    • Authentication method

      Left the default of Manual

    • Client secret

      Include the client secret value (not the Client secret ID) that was created in the earlier setup process (link here)

    • In the Content Types section at the bottom of the page, click the Enabled slider depending on the types of data to be collected

    • Click on the Connected Destinations link at the bottom of the left nav panel

    • Click the QuickConnect option (from the default of Send to Routes)

      • For this walkthrough, we're passing data directly from the O365 pull to the LogScale destination

      • When prompted about the change, click Yes

      • Click Save

        Status widget should display

      • After saving the new change, in the upper-right part of page, click Commit & Deploy to deploy the new configuration

      • Add a comment for the update

      • Click Commit and Deploy in the lower-right part of the popup

        • Status widgets should display

Enabling Microsoft Subscription to Generate Data the Activity Source Can Access

The App Subscription kick-off is a requirement to start collection of data (O365 Activity) by CrowdStream / CriblStream

  • From Cribl documentation:

    • https://docs.cribl.io/stream/sources-office-365-activity/#start-subscriptions

      • Content subscriptions (a different concept from the O365 subscription plans) are required in order for Cribl Stream / CrowdStream to be able to begin retrieving O365 data

      • There is a separate subscription required for each Content Type. If you are using an existing Azure-registered application ID that already has subscriptions started, then you can ignore this section. But if you are:

        • Using a newly registered application ID, and therefore never had any subscriptions started, or

        • Reusing an application ID that had subscriptions started, but are currently stopped

      • Then you will need to use this procedure to manually start the necessary subscriptions. Follow either of the two methods below, using (respectively) PowerShell or curl.

    • For example, the following content subscriptions are called out in the Cribl docs relative to collecting Microsoft Office 365 Activity logs

      • Audit.AzureActiveDirectory

      • Audit.Exchange

      • Audit.SharePoint

      • Audit.General

      • DLP.All

  • Browse to a Windows host that can call PowerShell commands

  • From the Cribl page, https://docs.cribl.io/stream/sources-office-365-activity/#start-subscriptions, create and run a PowerShell script file using the provided content that includes the contents / keys for the PowerShell script

    • On the Windows host:

      • Browse to the following url:

      • Copy the sample PowerShell script listed in the Using PowerShell section on the page

      • Open notepad.exe and paste in the sample PowerShell script contents

      • At the top of the file, replace the following placeholder values for actual values from your environment / newly-created app:

        • $AppID

        • $ClientSecret

        • $TenantID

      • Save the PowerShell script

        • For example, as c:\temp\o365_subscriptions.ps1 (can be any name you want, with ps1 as the file extension)

      • Run the PowerShell script

        From cmd window, type: powershell

        • Prompt will update to reflect the change into PowerShell

        • Call the script using fully-qualified path (if not already in the dir)

          • In my case:

            • c:\temp\o365_subscriptions.ps1

              • Status changes are displayed in the CMD window as commands are run

Creating O365 Services Source
Setting up the Source

After filtering on 365 for sources, click on the Office 365 Services tile

  • In the upper-right of the page, click on Add Source

  • Fill in the following values for the new source

    • Input ID

      Can be anything you want - just the name for the source in CrowdStream / CriblStream

      For example: o365_services_internal_lab

    • Tenant ID

      The Office 365 Azure Microsoft tenant ID for your environment

    • App ID

      The Office 365 Azure Application ID for the newly created Microsoft app

    • Subscription plan

      By default, using Office 365 Enterprise

    • Authentication method

      Left the default of Manual

    • Client secret

      Include the client secret value (not the Client secret ID) that was created in the earlier setup process (link here)

  • In the Content Types section at the bottom of the page:

    • Click the Enabled slider depending on the types of data to be collected

    • Set the Interval time (default = 15 minutes)

  • Click on the Connected Destinations link at the bottom of the left nav panel

  • Click the QuickConnect option (from the default of Send to Routes)

    For this walkthrough, we're passing data directly from the O365 pull to the LogScale destination

    When prompted about the change, click Yes

  • Click Save

    • Status widget should display

    • After saving the new change, in the upper-right part of page, click Commit & Deploy to deploy the new configuration

    • Add a comment for the update

    • Click Commit and Deploy in the lower-right part of the popup

      • Status widgets should display

O365 Message Trace Source

Because O365 Message Trace requires additional setup permissions (link here), the Message Trace Source is not configured in this example

From the Cribl O365 Message Trace page:

At a minimum, your Office 365 service account should include a role with Message Tracking and View Only Recipients permissions, assigned to the Office 365 user that will integrate with Cribl Stream. Assign these permissions in the Exchange admin center (https://admin.exchange.microsoft.com).

Check the Status of Each Source

From the main CrowdStream / CriblStream page, click on Data → Sources

  • A list of available Push / Pull sources is displayed

  • To refine down list, click on the Pull filter on left and click on the Configured only slider on the right

  • Once connections configured, if any connection shows a status issue (red for errors, orange for warnings or indicates you had red, even if green now, green for status good), click on the tile that shows the error condition

    • For example, in this case, the "Office 365 Services" connection is reporting an error

    • From the row(s) that display, click on the row that shows the error

    • At the top of the page, click on the Job Inspector link

    • In the job listings that display, click on any of the rows showing an error

    • In the top of the page, click on Task Errors

    • Expand and review any errors listed to determine the root cause of the issue

      For example, in this case, 403 HTTP errors are listed (The HTTP 403 Forbidden response status code indicates that the server understands the request but refuses to authorize it.)

Define / Configure LogScale Destination in CrowdStream

Step-by-step example of setting up a CrowdStream Destination (with a pointer to LogScale repository) available How-To: How to Configure CrowdStream LogScale Destination.

Connect the CrowdStream O365 Source(s) to a LogScale Destination
Add the New O365 Activity Source to the Source Side

  • Log in to your CrowdStream / Cribl instance

    • For example, this can be a CrowdStream instance at a customer / POV, or it can be a Cribl eval instance you signed up for (both behave the same)

  • From the main CrowdStream / CriblStream page, click on Manage Stream

  • Click on the Worker Group which will be used

    • Typically in a POV / initial setup, the Worker Group will be called default

  • Click on the Overview link (upper-left) to see the UI reference to QuickConnect

  • Click on the Quick Connect widget to display the QuickConnect page

  • Click on the Add Source box on left

    • A list of available Push / Pull sources is displayed

    • To refine down list, click on the Pull filter on left and click on the Configured only slider on the right

  • Click on the Office 365 Services tile, and select Select Existing

  • Click on the existing configured source (anywhere in the row)

    • When prompted about switching to QuickConnect from Routes, click Yes

  • Click Add Source again and repeat the process with the Office 365 Activity tile

Connect the Entries from the Sources side to the LogScale Destination

  • Click and hold on the + symbol on the right side of each source, and drag a line over to the CrowdStrike Falcon LogScale entry on the Destination side

  • When prompted for the type of connection configuration, leave Passthru selected, and click Save

  • Repeat process for the other O365 source

  • When all changes complete, in the upper-right corner, click on Commit & Deploy to deploy the changes

    • Add a comment about the work done

    • Click Commit and Deploy in the lower-right of the widget

      • As changes are being updated to the CrowdStream / CriblStream cluster, the status of the cluster will show as yellow in the upper right of the page

      • Once the changes have been committed and applied and all services update, the status of the workers will change to green