Use Case: Log Management

Modern IT systems create a lot of operational data and it can be difficult to manage. Often, this data is stored in logs.

The term log management relates to producing, shipping, normalizing, and querying this data. It's a complex activity, with many moving parts. A specialized log management tool like LogScale can save you a lot of time and money.

Unfortunately, there are no standards across applications for logging data. We recommend focusing on your key systems first, then expanding and developing your log management setup as needed.

Log Sources

Log data is the lowest common denominator for getting both real-time and historical insights into running systems.

Most systems and applications append log data to log files on disk. However, some systems also support sending logs directly to a log management systems like LogScale.

Sending Logs

In most situations, it's necessary to add a 'log shipping' layer to your system. A 'log shipper' is an application that can take logs from a file on disk and send them directly to a log management system.

An important aspect of shipping is how faults are handled. When evaluating a log shipper, examine what situations will cause the loss of log lines, or duplication. Also check which kinds of failures it can tolerate.

Parsing Incoming Data

Because most log formats are unique, at some point you'll need to parse your logs. Parsing lets you take logs that are essentially lines of text, and extract their structure in order to incorporate a more in-depth analysis.

For example, you can extract useful items of data such as a timestamp or key-value pairs.

LogScale can parse logs for you. If you are familiar with Fluentd or LogStash, they can also parse logs.

Note

LogScale integrates with both Fluentd and LogStash log shipping frameworks.