Calculate Total Log Volume Per Service

Analyze log volume across services using the groupBy() function with sum()

Query

logscale
event_type="log_event"
| groupBy([service], function=sum(field="bytes", as=TotalBytes))
| sort(field="TotalBytes", order="desc")

Introduction

The groupBy() function can be used to perform aggregate calculations on grouped data, allowing analysis of metrics like log volume across different services or systems.

In this example, the groupBy() is used with sum() to calculate total log volume per service, then sorted to identify which services generate the most data.

Example incoming data might look like this:

@timestampevent_typeservicebytes
1686837825000log_eventauth-service4200
1686837825000log_eventapi-gateway15800
1686837825000log_eventauth-service3100
1686837826000log_eventbilling-service9200
1686837826000log_eventapi-gateway22400
1686837826000log_eventauth-service5600
1686837827000log_eventpayment-service18300
1686837827000log_eventbilling-service7800
1686837827000log_eventapi-gateway19200
1686837828000log_eventpayment-service21000
1686837828000log_eventbilling-service6400
1686837828000log_eventauth-service4800

Step-by-Step

  1. Starting with the source repository events.

  2. logscale
    event_type="log_event"

    Filters events to include only those where event_type equals log_event.

  3. logscale
    | groupBy([service], function=sum(field="bytes", as=TotalBytes))

    Groups the data by the service field and calculates the sum of bytes, storing the result in a field named TotalBytes.

  4. logscale
    | sort(field="TotalBytes", order="desc")

    Sorts the results based on the TotalBytes field in descending order (order=desc), showing services with the highest log volume first.

  5. Event Result set.

Summary and Results

The query is used to analyze log volume across services, ranked from highest to lowest.

This query is useful, for example, to identify services generating disproportionate log volume, plan ingestion capacity, or investigate unexpected spikes in data output.

Sample output from the incoming example data:

serviceTotalBytes
api-gateway57400
payment-service39300
billing-service23400
auth-service17700

Note that the volume values are in bytes and that each row represents the aggregated log volume for a unique service.