Set up a New Azure Ingest Feed

The following steps guide you through creating a new ingest feed for Azure Event hub data.

  1. Go to Repositories and views page and select a the repository in which you want to ingest the data.

  2. Click Settings, under Ingest on the side menu click Ingest feeds.

    Screenshot of the LogScale Ingest feeds settings page showing the interface where users begin creating a new ingest feed. The page displays any existing configured feeds in a tabular format with columns for feed name, preprocessing method, parser configuration, last activity timestamp, and status. At the top of the interface is the '+ New ingest feed' button which users click to initiate the Azure feed configuration process. This interface serves as the starting point for setting up data ingestion from Azure Event Hubs into LogScale repositories.

    Figure 99. Ingest Feeds


  3. On the Ingest feeds page, click + New ingest feed, select New Azure ingest feed and specify:

    • Name

      Enter a name for the feed (required).

    • Description

      Enter a meaningful description for the feed.

    • Preprocessing

      Define how the data is processed prior to ingestion, it can be split by Azure records or by new line. The way data should be preprocessed depends on the log source.

    • Parser

      Select the parser to apply to ingest data (required). The parser can be any parser that is on repository, predefined, from a package, or custom. See Parse Data for more information.

    Screenshot of the LogScale 'New Azure ingest feed' configuration dialog showing the initial setup interface for Azure Event Hub data ingestion. The image displays the first step in the Azure feed creation workflow where administrators enter the fundamental configuration parameters. The form includes several key input fields: a required Name field for providing a unique identifier for the feed, an optional Description field for documentation purposes, a Preprocessing dropdown for selecting how incoming data should be processed (with options for splitting by Azure records or new lines), and a required Parser selection field for choosing which parser will extract and structure the data. The interface follows LogScale's standardized configuration pattern with a clean, organized layout that guides administrators through the initial setup before proceeding to Azure-specific credential configuration in subsequent steps. This panel establishes the basic parameters that determine how Event Hub data will be processed when it reaches LogScale, setting the foundation for the complete Azure data ingestion pipeline.

    Figure 100. Ingest Feeds - Add


  4. Click Next.

Set up Ingest Feed - Credentials

In the Add credentials dialog box define:

  • Fully qualified namespace The namespace of the Event hub, the name can found in the event hub as Host Name, see Microsoft documentation for more information. The structure is usually: 

    <namespace>.servicebus.windows.net

  • Event hub name The event hub name can be found on the event hub page in the Azure Portal or CLI.

  • Consumer group default: $Default) the consumer group can be found on the event hub page in the Azure portal or CLI. We recommend adding a dedicated consumer group.

  • Blob storage endpoint Can be found through the CLI structured as follows: https://<storage account name>.blob.core.windows.net/

  • Storage Blob container name Can be found in the azure portal or CLI.

  • Default checkpoint Oldest position, Custom position, Newest position

    • Oldest position: begin receiving events from the first event.

    • Custom position: add filter to specify the position in the partition to begin receiving events.

    • Newest position: begin receiving events from the event that is queued.

Screenshot of the LogScale 'Add credentials' dialog for Azure Event Hub configuration showing the connection parameters form. This image displays the essential credential and connection settings interface where administrators enter Azure-specific details required to establish the data pipeline. The form includes several critical input fields: the Fully qualified namespace field for entering the Event Hub's host name (typically in the format namespace.servicebus.windows.net), the Event hub name field for specifying the exact hub to connect to, the Consumer group dropdown (defaulting to '$Default' but allowing for dedicated groups), the Blob storage endpoint field for the storage account URL, the Storage Blob container name field for checkpointing data, and a Default checkpoint selection area with radio buttons for determining the starting position for data consumption (Oldest, Custom, or Newest). The interface provides a structured approach to gathering all necessary connection parameters while presenting helpful guidance on the expected format of each value. This configuration panel establishes the secure connection between LogScale and Azure's event streaming infrastructure, ensuring proper authentication and data access permissions for the ingestion process.

Figure 101. Azure Credentials


Set up Ingest feed - Authentication

Provide authentication for the app as specified in App Configurations.

  1. In the Add authentication dialog box specify:

    • Client ID : The ID which is generated when you create the APP.

    • Client secret : the secret which is generated when you create the APP.

    • Secret ID : an ID you can assign to the secret. Click Reset Client Secret to delete the secret and insert a new value.

    • Tenant ID : The directory ID on the App.

  2. Click Test Azure feeds to verify the feed is correctly configured.

  3. Click Save to save the ingest feed.

Azure Authentication

Figure 102. Azure Credentials