Set up a New Azure Ingest Feed

The following steps guide you through creating a new ingest feed for Azure Event hub data.

  1. Go to Repositories and views page and select a the repository in which you want to ingest the data.

  2. Click Settings, under Ingest on the side menu click Ingest feeds.

    Screenshot of the LogScale Ingest feeds settings page showing the interface where users begin creating a new ingest feed. The page displays any existing configured feeds in a tabular format with columns for feed name, preprocessing method, parser configuration, last activity timestamp, and status. At the top of the interface is the '+ New ingest feed' button which users click to initiate the Azure feed configuration process. This interface serves as the starting point for setting up data ingestion from Azure Event Hubs into LogScale repositories.

    Figure 120. Ingest Feeds


  3. On the Ingest feeds page, click + New ingest feed, select New Azure ingest feed and specify:

    • Name

      Enter a name for the feed (required).

    • Description

      Enter a meaningful description for the feed.

    • Preprocessing

      Define how the data is processed prior to ingestion, it can be split by Azure records or by new line. The way data should be preprocessed depends on the log source.

    • Parser

      Select the parser to apply to ingest data (required). The parser can be any parser that is on repository, predefined, from a package, or custom. See Parse Data for more information.

    Screenshot of the LogScale 'New Azure ingest feed' configuration dialog showing the initial setup interface for Azure Event Hub data ingestion. The image displays the first step in the Azure feed creation workflow where administrators enter the fundamental configuration parameters. The form includes several key input fields: a required Name field for providing a unique identifier for the feed, an optional Description field for documentation purposes, a Preprocessing dropdown for selecting how incoming data should be processed (with options for splitting by Azure records or new lines), and a required Parser selection field for choosing which parser will extract and structure the data. The interface follows LogScale's standardized configuration pattern with a clean, organized layout that guides administrators through the initial setup before proceeding to Azure-specific credential configuration in subsequent steps. This panel establishes the basic parameters that determine how Event Hub data will be processed when it reaches LogScale, setting the foundation for the complete Azure data ingestion pipeline.

    Figure 121. Ingest Feeds - Add


  4. Click Next.

Set up Ingest Feed - Credentials

In the Add credentials dialog box define:

  • Fully qualified namespace The namespace of the Event hub, the name can found in the event hub as Host Name, see Microsoft documentation for more information. The structure is usually: 

    <namespace>.servicebus.windows.net

  • Event hub name The event hub name can be found on the event hub page in the Azure Portal or CLI.

  • Consumer group default: $Default) the consumer group can be found on the event hub page in the Azure portal or CLI. We recommend adding a dedicated consumer group.

  • Blob storage endpoint Can be found through the CLI structured as follows: https://<storage account name>.blob.core.windows.net/

  • Storage Blob container name Can be found in the azure portal or CLI.

  • Default checkpoint Oldest position, Custom position, Newest position

    • Oldest position: begin receiving events from the first event.

    • Custom position: add filter to specify the position in the partition to begin receiving events.

    • Newest position: begin receiving events from the event that is queued.

Screenshot of the LogScale 'Add credentials' dialog for Azure Event Hub configuration showing the connection parameters form. This image displays the essential credential and connection settings interface where administrators enter Azure-specific details required to establish the data pipeline. The form includes several critical input fields: the Fully qualified namespace field for entering the Event Hub's host name (typically in the format namespace.servicebus.windows.net), the Event hub name field for specifying the exact hub to connect to, the Consumer group dropdown (defaulting to '$Default' but allowing for dedicated groups), the Blob storage endpoint field for the storage account URL, the Storage Blob container name field for checkpointing data, and a Default checkpoint selection area with radio buttons for determining the starting position for data consumption (Oldest, Custom, or Newest). The interface provides a structured approach to gathering all necessary connection parameters while presenting helpful guidance on the expected format of each value. This configuration panel establishes the secure connection between LogScale and Azure's event streaming infrastructure, ensuring proper authentication and data access permissions for the ingestion process.

Figure 122. Azure Credentials


Test Azure Feed

  1. Click Test Azure feeds to verify the feed is correctly configured.

  2. Click Save to save the ingest feed.

Screenshot of the LogScale Azure feed test verification dialog showing the results of a connection test between LogScale and Azure Event Hub. This image displays the feedback interface that appears after clicking the 'Test Azure feeds' button, presenting the outcome of LogScale's attempt to validate the provided connection parameters and credentials. The interface shows a connection status indicator that confirms whether LogScale could successfully authenticate with Azure using the provided credentials, connect to the specified Event Hub namespace, locate the named Event Hub, access the specified storage account, and verify permissions on the blob container. This verification step is critical in the configuration workflow as it catches potential connection issues before they disrupt production data ingestion, helping administrators identify and fix credential problems, permission issues, or typos in connection parameters. The test results provide immediate feedback about each component of the Azure connection chain, allowing administrators to confidently save working configurations or troubleshoot specific connection problems before proceeding with the final setup.

Figure 123. Azure feed test verification