Action Type: Webhooks
Security Requirements and Controls
Change triggers and actions
permission
HTTP Webhook Action is LogScale's most flexible type of action. The Webhook action can perform an HTTP(S) request to any URL and can therefore be used to integrate third-party services that LogScale doesn't have natively integrated.
Figure 222. Configuring Webhooks Action
Parameter | Description |
---|---|
Name | The name given to the Webhook action. |
Endpoint URL | The URL of the service the webhook is contacting. |
Method |
HTTP method of the call, usually POST .
|
HTTP Headers |
A list of HTTP Headers. We recommend adding a
Content-Type header with
the corresponding content type, such as
application/json for JSON
message bodies. Add more headers by clicking the
+ on the right
|
Message Body Template | Optional. The body of the HTTP call. Can be of any form: JSON, Text, or even XML. The Message Body accepts our Message Templates and Variables as well. |
You can use a service like Webhook.site to test your action while you are getting the message body and headers correct.
Example Using Webhooks to Trigger a Shell Script
Security Requirements and Controls
Change triggers and actions
permission
Alerts are actionable and one way to service them is by triggering a script on a particular host. For example, while exploring logs and metrics using LogScale, occasionally a message arrives informing that a system's root partition is filling up. You can decide to make this into an alert and notify everyone through Slack so that someone can ssh into that system and run:
sudo apt-get -y autoremove
As the cleanup required could be automated, you could write a script that fixes this issue without human intervention. In order to do so, LogScale needs to execute a shell script or other executable that resides on some other host: webhooks can help in this.
This section will show how to setup and run a program that will listen
for these Alerts by providing
"hooks" that are RESTful HTTP GET
or
POST
calls.
These simple web services listen for requests and execute scripts on the target system. In this way LogScale can be used to automate and resolve all sorts of issues without anyone needing a human involved in the process. More log messages can be written from these scripts and feed those back into LogScale as well, so to ensure everything is working as intended.
Here is an outline of the steps required to enable this process. The following prerequirements are needed:
LogScale is setup, operational, and ingesting your log data.
A separate host system, let's call it the "target", where to run a script when alerted.
A script, or other executable to trigger on that host system
A network that allows traffic to routed from LogScale to the target system.
Setup on Target System
Security Requirements and Controls
Change triggers and actions
permission
Webhooks are user-defined HTTP callbacks. In our case it's the LogScale server making an HTTP request using a URL you supply. This is another use of a RESTful pattern in practice. LogScale will issue an HTTP POST to the supplied URL and include in that request's body the information you provide. The target system then runs a process that is essentially a simple web server listening for requests, that's the URL.
In this example, the target system has:
a DNS name
target.example.com
that resolves to a routable IP address, anda script in
/var/scripts/cleanup.sh
that you'd like to trigger.
The agent that will run on the host and listen for the message is called webhook. This is a simple program designed to advertise RESTful HTTP endpoints known as "webhooks" (hence the name) that when triggered via HTTP/GET or POST will execute scripts. While the authors of this open source program intended it to be used with the Hookdoo service it is a general purpose tool and in our case we're reusing it and not their service.
Installing Webhook
Find the release appropriate for your system, and install it. This is a program written to integrate with the Hookdoo service, but in our case we're reusing their tool and not their service. If you're on Linux you'll eventually want to setup a startup script that will execute this program and restart it when necessary.
Write a simple configuration script in JSON once you have the
webhook
binary in your path. The configuration file lists the endpoints to make available on this system and associates them with the script to execute when that endpoint is triggered. You can have as many endpoint/script pairs as you'd like. The target can been a script or any other executable file you designate.Create an empty file named
hooks.json
. This file will contain an array of hooks thewebhook
will serve. Check the hook definition page to see the detailed description of what properties a hook can contain, and how to use them.Define a simple hook named
cleanup-webhook
that will run our cleanup.sh script located in/var/scripts/cleanup.sh
. Make sure that your bash script has#!/bin/sh
shebang on top and is executable:chmod +x cleanup.sh
Create a file that will define the webhooks available on this system, we'll call it
hooks.json
but you can call it anything you'd like. Here's what it should look like:json[ { "id": "cleanup-webhook", "execute-command": "/var/scripts/cleanup.sh", "command-working-directory": "/tmp" } ]
id
- is simply a name you've given the hook, it should be unique in this file, but otherwise it can be whatever you'd like it to be.execute-command
- is the full path to the script you'd like to run when this webhook triggers.command-working-directory
- is the path that the script will find as it's the current working directory when executing.
Run the
webhook
binary we downloaded earlier using:shell$ /path/to/webhook -hooks hooks.json -verbose
It will start up on default port 9000 and will provide you with one HTTP endpoint.
httphttp://yourserver:9000/hooks/redeploy-webhook
Copy this endpoint as it will be needed during LogScale's webhook action configuration. The
yourserver
part can either be an IP address of the host or the DNS name that resolves to the IP of the host like any other HTTP URI. For more information on this program, see Webhook documentation.Add to the command line which launches
webhook
the -secure flag and reference a certificate using-cert /path/to/cert.pem
and-key /path/to/key.pem
flags.This must be done because once webhooks are setup, anyone with a route to the host could trigger the hook and run the script potentially causing harm or gaining access to data that is sensitive. If you want
webhook
to serve secure content have a verifiable identity you'll want to use HTTPS.Make sure you have a system in place to generate, rotate and protect these self-signed certificates and that the system running LogScale system is setup to acknowledge the veracity of them. If you have a certificate signed by a certificate authority, then the cert file name should be the concatenation of the server's certificate followed by the CA's certificate.
Setup on LogScale
In LogScale events can cause alerts and trigger actions. The first thing to do is to create an alert, which in turn will trigger an action, in this case a Webhook action.
Create an alert as described in Creating Alerts
Create a new Webhook action as described in Creating Actions.
Give your new Webhook a name, in this example we'll call it
Cleanup YourServer when disk space is low
.Fill in the Endpoint URL field with the URL you saved when Installting Webhhok.
logscalehttp://yourserver:9000/hooks/redeploy-webhook
The message body template is initially filled in with a JSON template that includes all the information pertaining to the Alert and event(s) that triggered it.
javascript{ "repository": "{repo_name}", "timestamp": {triggered_timestamp}, "alert": { "name": "{name}", "description": "{description}", "query": { "queryString": "{query_string} ", "end": "{query_time_end}", "start": "{query_time_start}" }, "notifierID": "{action_id}", "id": "{id}" }, "warnings": "{warnings}", "events": {events}, "numberOfEvents": {event_count} }
Because the default template body is formatted as JSON we've added a default
Content-Type
header ofapplication-json
. You can use this default body template or edit it to suit your needs. It need not be JSON, but using JSON enables you to use features ofwebhook
that allow you to select pieces of the JSON document to pass to your script as command line arguments or environment variables.Test it by simply starting up the
webhook
executable in verbose mode and watch it execute your script.ini Syntax./webhook -verbose -port 9999 [webhook] 2018/12/12 12:09:48 version 2.6.9 starting [webhook] 2018/12/12 12:09:48 setting up os signal watcher [webhook] 2018/12/12 12:09:48 attempting to load hooks from hooks.json [webhook] 2018/12/12 12:09:48 os signal watcher ready [webhook] 2018/12/12 12:09:48 found 1 hook(s) in file [webhook] 2018/12/12 12:09:48 loaded: cleanup-webhook [webhook] 2018/12/12 12:09:48 serving hooks on http://0.0.0.0:9999/hooks/{id} [webhook] 2018/12/12 12:11:23 [7415e3] incoming HTTP request from [::1]:44872 [webhook] 2018/12/12 12:11:23 [7415e3] cleanup-webhook got matched [webhook] 2018/12/12 12:11:23 [7415e3] error parsing JSON payload EOF [webhook] 2018/12/12 12:11:23 [7415e3] cleanup-webhook hook triggered successfully [webhook] 2018/12/12 12:11:23 200 | 132.063's | localhost:9999 | POST /hooks/cleanup-webhook [webhook] 2018/12/12 12:11:23 [7415e3] executing /var/scripts/cleanup-webhook.sh (/var/scripts/cleanup-webhook.sh) with arguments ["/var/scripts/cleanup-webhook.sh"] and environment [HOOK_payload={ ... JSON ...}} HOOK_headers={"Accept":"*/*","Content-Type":"application/json"} HOOK_query={}] using /tmp as cwd [webhook] 2018/12/12 12:11:23 [7415e3] command output: [webhook] 2018/12/12 12:11:23 [7415e3] finished handling cleanup-webhook
Save your action and start the
webhook
server and the next time the alert fires your script on the remote system will be triggered.