Modern IT systems create a lot of operational data and it can be difficult to manage. Often, this data is stored in logs.
The term log management relates to producing, shipping, normalizing, and querying this data. It's a complex activity, with many moving parts. A specialized log management tool like Humio can save you a lot of time and money.
Unfortunately, there are no standards across applications for logging data. We recommend focusing on your key systems first, then expanding and developing your log management setup as needed.
Log data is the lowest common denominator for getting both real-time and historical insights into running systems.
Most systems and applications append log data to log files on disk. However, some systems also support sending logs directly to a log management systems like Humio.
In most situations, it's necessary to add a 'log shipping' layer to your system. A 'log shipper' is an application that can take logs from a file on disk and send them directly to a log management system.
An important aspect of shipping is how faults are handled. When evaluating a log shipper, examine what situations will cause the loss of log lines, or duplication. Also check which kinds of failures it can tolerate.
Parsing Incoming Data
Because most log formats are unique, at some point you'll need to parse your logs. Parsing lets you take logs that are essentially lines of text, and extract their structure in order to incorporate a more in-depth analysis.
For example, you can extract useful items of data such as a timestamp or key-value pairs.
Humio integrates with both Fluentd and LogStash log shipping frameworks.
Querying is where you will feel the real benefit of deploying a log management tool like Humio. The system creates value by letting you ask a variety of questions about the data in ways it might not be possible to with a raw data log.
The list below shows some of the use cases where a log management system like Humio could be invaluable:
Investigating incidents and anomalies
Following operations in real-time - is the deployment causing errors?
Discovering internal and external communication patterns
Seeing how events in one part of the system affect other parts
"Feeling the Hum of your System" - all systems have a rhythm. What is yours?
Seeing real-time status - are your systems up?
Finding the answers to ad-hoc questions
Making the system visible - put dashboards on big screens so everybody can "feel the hum"