Alerts are actionable and one way to service them is by triggering a script on a particular host. For example, while exploring your logs and metrics using LogScale you might notice that occasionally a message arrives informing you that a system's root partition is filling up. You decide to make this into an alert and notify everyone through Slack so that someone can ssh into that system and run

sudo apt-get -y autoremove

But then you realize that the cleanup required could be automated, you could write a script that fixes this issue without human intervention.

Great idea, but how can LogScale execute a shell script or other executable that resides on some other host? Here is where webhooks can bridge the gap. This guide will show you how to setup and run a program that will listen for these Alerts by providing "hooks" that are RESTful HTTP GET or POST calls.

These simple web services listen for requests and execute scripts on the target system. In this way you can use LogScale to automate and resolve all sorts of issues without anyone needing a human involved in the process. And don't forget to write more log messages from these scripts and feed those back into LogScale as well, so you can be sure everything is working as intended.

Here is an outline of the steps required to enable this process. We assume for the sake of simplicity that you have the following:

Webhooks are user-defined HTTP callbacks. In our case it's the LogScale server making an HTTP request using a URL you supply. This is another use of a RESTful pattern in practice. LogScale will issue an HTTP POST to the supplied URL and include in that request's body the information you provide. The target system then runs a process that is essentially a simple web server listening for requests, that's the URL.

In this example, the target system has:

The agent that will run on the host and listen for the message is called webhook. This is a simple program designed to advertise RESTful HTTP endpoints known as "webhooks" (hence the name) that when triggered via HTTP/GET or POST will execute scripts. While the authors of this open source program intended it to be used with the Hookdoo service it is a general purpose tool and in our case we're reusing it and not their service.

Find the release appropriate for your system, and install it. This is a program written to integrate with the Hookdoo service, but in our case we're reusing their tool and not their service. If you're on Linux you'll eventually want to setup a startup script that will execute this program and restart it when necessary.

Once you have the webhook binary in your path you'll need to write a simple configuration script in JSON. The configuration file lists the endpoints to make available on this system and associates them with the script to execute when that endpoint is triggered. You can have as many endpoint/script pairs as you'd like. The target can been a script or any other executable file you designate.

Begin by creating an empty file named hooks.json. This file will contain an array of hooks the webhook will serve. Check the hook definition page to see the detailed description of what properties a hook can contain, and how to use them.

Let's define a simple hook named cleanup-webhook that will run our cleanup.sh script located in /var/scripts/cleanup.sh. Make sure that your bash script has #!/bin/sh shebang on top and is executable:

chmod +x cleanup.sh

Create a file that will define the webhooks available on this system, we'll call it hooks.json but you can call it anything you'd like. Here's what it should look like:

[
  {
    "id": "cleanup-webhook",
    "execute-command": "/var/scripts/cleanup.sh",
    "command-working-directory": "/tmp"
  }
]

You can now run the webhook binary we downloaded earlier using.

$ /path/to/webhook -hooks hooks.json -verbose

It will start up on default port 9000 and will provide you with one HTTP endpoint.

http://yourserver:9000/hooks/redeploy-webhook

This endpoint is what you'll use in LogScale's webhook Alert configuration, so copy it and move on to the next step. The yourserver part can either be an IP address of the host or the DNS name that resolves to the IP of the host like any other HTTP URI. For other questions about this program please take a look at their documentation.

One final note about webhooks in general, they are a good tool but once setup anyone with a route to the host could trigger the hook and run the script potentially causing harm or gaining access to data that is sensitive. If you want webhook to serve secure content have a verifiable identity you'll want to use HTTPS. Simply add to the command line which launches webhook the -secure flag and reference a certificate using -cert /path/to/cert.pem and -key /path/to/key.pem flags.

Make sure you have a system in place to generate, rotate and protect these self-signed certificates and that the system running LogScale system is setup to acknowledge the veracity of them. If you have a certificate signed by a certificate authority, then the cert file name should be the concatenation of the server's certificate followed by the CA's certificate.

In LogScale events can cause Alerts and trigger Actions. The first thing to do is to create an alert, see Creating Alerts. Alerts in turn trigger actions, in this case we want a Webhook action.

Give your new webhook a name, in this example we'll call it "Cleanup YourServer when disk space is low". Next, fill in the Endpoint URL field with the URL you saved from the previous step.

http://yourserver:9000/hooks/redeploy-webhook

The message body template is initially filled in with a JSON template that includes all the information pertaining to the Alert and event(s) that triggered it.

{
    "repository": "{repo_name}",
    "timestamp": {triggered_timestamp},
    "alert": {
       "name": "{name}",
       "description": "{description}",
       "query": {
          "queryString": "{query_string} ",
          "end": "{query_time_end}",
          "start": "{query_time_start}"
      },
      "notifierID": "{action_id}",
      "id": "{id}"
    },
    "warnings": "{warnings}",
    "events": {events},
    "numberOfEvents": {event_count}
}

Because the default template body is formatted as JSON we've added a default Content-Type header of application-json. You can use this default body template or edit it to suit your needs. It need not be JSON, but using JSON enables you to use features of webhook that allow you to select pieces of the JSON document to pass to your script as command line arguments or environment variables.

To test simply startup the webhook executable in verbose mode and watch it execute your script.

./webhook -verbose -port 9999
[webhook] 2018/12/12 12:09:48 version 2.6.9 starting
[webhook] 2018/12/12 12:09:48 setting up os signal watcher
[webhook] 2018/12/12 12:09:48 attempting to load hooks from hooks.json
[webhook] 2018/12/12 12:09:48 os signal watcher ready
[webhook] 2018/12/12 12:09:48 found 1 hook(s) in file
[webhook] 2018/12/12 12:09:48    loaded: cleanup-webhook
[webhook] 2018/12/12 12:09:48 serving hooks on http://0.0.0.0:9999/hooks/{id}
[webhook] 2018/12/12 12:11:23 [7415e3] incoming HTTP request from [::1]:44872
[webhook] 2018/12/12 12:11:23 [7415e3] cleanup-webhook got matched
[webhook] 2018/12/12 12:11:23 [7415e3] error parsing JSON payload EOF
[webhook] 2018/12/12 12:11:23 [7415e3] cleanup-webhook hook triggered successfully
[webhook] 2018/12/12 12:11:23 200 | 132.063's | localhost:9999 | POST /hooks/cleanup-webhook
[webhook] 2018/12/12 12:11:23 [7415e3] executing /var/scripts/cleanup-webhook.sh (/var/scripts/cleanup-webhook.sh) with arguments ["/var/scripts/cleanup-webhook.sh"] and environment [HOOK_payload={ ... JSON ...}} HOOK_headers={"Accept":"*/*","Content-Type":"application/json"} HOOK_query={}] using /tmp as cwd
[webhook] 2018/12/12 12:11:23 [7415e3] command output:
[webhook] 2018/12/12 12:11:23 [7415e3] finished handling cleanup-webhook

Now save your action and start the webhook server and the next time the alert fires your script on the remote system will be triggered.