Action Type: Webhooks

Security Requirements and Controls

HTTP Webhook Action is LogScale's most flexible type of action. The Webhook action can perform an HTTP(S) request to any URL and can therefore be used to integrate third-party services that LogScale doesn't have natively integrated.

Configuring Webhooks Action

Figure 198. Configuring Webhooks Action


Parameter Description
Name The name given to the Webhook action.
Endpoint URL The URL of the service the webhook is contacting.
Method HTTP method of the call, usually POST .
  
HTTP Headers A list of HTTP Headers. We recommend adding a Content-Type header with the corresponding content type, such as application/json for JSON message bodies. Add more headers by clicking the + on the right
Message Body Template Optional. The body of the HTTP call. Can be of any form: JSON, Text, or even XML. The Message Body accepts our Message Templates and Variables as well.

You can use a service like Webhook.site to test your action while you are getting the message body and headers correct.

Example Using Webhooks to Trigger a Shell Script

Alerts are actionable and one way to service them is by triggering a script on a particular host. For example, while exploring logs and metrics using LogScale, occasionally a message arrives informing that a system's root partition is filling up. You can decide to make this into an alert and notify everyone through Slack so that someone can ssh into that system and run:

shell
sudo apt-get -y autoremove

As the cleanup required could be automated, you could write a script that fixes this issue without human intervention. In order to do so, LogScale needs to execute a shell script or other executable that resides on some other host: webhooks can help in this.

This section will show how to setup and run a program that will listen for these Alerts by providing "hooks" that are RESTful HTTP GET or POST calls.

These simple web services listen for requests and execute scripts on the target system. In this way LogScale can be used to automate and resolve all sorts of issues without anyone needing a human involved in the process. More log messages can be written from these scripts and feed those back into LogScale as well, so to ensure everything is working as intended.

Here is an outline of the steps required to enable this process. The following prerequirements are needed:

  • LogScale is setup, operational, and ingesting your log data.

  • A separate host system, let's call it the "target", where to run a script when alerted.

  • A script, or other executable to trigger on that host system

  • A network that allows traffic to routed from LogScale to the target system.

Setup on Target System

Webhooks are user-defined HTTP callbacks. In our case it's the LogScale server making an HTTP request using a URL you supply. This is another use of a RESTful pattern in practice. LogScale will issue an HTTP POST to the supplied URL and include in that request's body the information you provide. The target system then runs a process that is essentially a simple web server listening for requests, that's the URL.

In this example, the target system has:

  • a DNS name target.example.com that resolves to a routable IP address, and

  • a script in /var/scripts/cleanup.sh that you'd like to trigger.

The agent that will run on the host and listen for the message is called webhook. This is a simple program designed to advertise RESTful HTTP endpoints known as "webhooks" (hence the name) that when triggered via HTTP/GET or POST will execute scripts. While the authors of this open source program intended it to be used with the Hookdoo service it is a general purpose tool and in our case we're reusing it and not their service.

Installing Webhook

  1. Find the release appropriate for your system, and install it. This is a program written to integrate with the Hookdoo service, but in our case we're reusing their tool and not their service. If you're on Linux you'll eventually want to setup a startup script that will execute this program and restart it when necessary.

  2. Write a simple configuration script in JSON once you have the webhook binary in your path. The configuration file lists the endpoints to make available on this system and associates them with the script to execute when that endpoint is triggered. You can have as many endpoint/script pairs as you'd like. The target can been a script or any other executable file you designate.

  3. Create an empty file named hooks.json. This file will contain an array of hooks the webhook will serve. Check the hook definition page to see the detailed description of what properties a hook can contain, and how to use them.

  4. Define a simple hook named cleanup-webhook that will run our cleanup.sh script located in /var/scripts/cleanup.sh. Make sure that your bash script has #!/bin/sh shebang on top and is executable:

    chmod +x cleanup.sh
  5. Create a file that will define the webhooks available on this system, we'll call it hooks.json but you can call it anything you'd like. Here's what it should look like:

    json
    [
      {
        "id": "cleanup-webhook",
        "execute-command": "/var/scripts/cleanup.sh",
        "command-working-directory": "/tmp"
      }
    ]
    • id - is simply a name you've given the hook, it should be unique in this file, but otherwise it can be whatever you'd like it to be.

    • execute-command - is the full path to the script you'd like to run when this webhook triggers.

    • command-working-directory - is the path that the script will find as it's the current working directory when executing.

  6. Run the webhook binary we downloaded earlier using:

    shell
    $ /path/to/webhook -hooks hooks.json -verbose

    It will start up on default port 9000 and will provide you with one HTTP endpoint.

    http
    http://yourserver:9000/hooks/redeploy-webhook
  7. Copy this endpoint as it will be needed during LogScale's webhook action configuration. The yourserver part can either be an IP address of the host or the DNS name that resolves to the IP of the host like any other HTTP URI. For more information on this program, see Webhook documentation.

  8. Add to the command line which launches webhook the -secure flag and reference a certificate using -cert /path/to/cert.pem and -key /path/to/key.pem flags.

    This must be done because once webhooks are setup, anyone with a route to the host could trigger the hook and run the script potentially causing harm or gaining access to data that is sensitive. If you want webhook to serve secure content have a verifiable identity you'll want to use HTTPS.

    Make sure you have a system in place to generate, rotate and protect these self-signed certificates and that the system running LogScale system is setup to acknowledge the veracity of them. If you have a certificate signed by a certificate authority, then the cert file name should be the concatenation of the server's certificate followed by the CA's certificate.

Setup on LogScale

In LogScale events can cause alerts and trigger actions. The first thing to do is to create an alert, which in turn will trigger an action, in this case a Webhook action.

  1. Create an alert as described in Creating Alerts

  2. Create a new Webhook action as described in Creating Actions.

  3. Give your new Webhook a name, in this example we'll call it Cleanup YourServer when disk space is low.

  4. Fill in the Endpoint URL field with the URL you saved when Installting Webhhok.

    logscale
    http://yourserver:9000/hooks/redeploy-webhook
  5. The message body template is initially filled in with a JSON template that includes all the information pertaining to the Alert and event(s) that triggered it.

    javascript
    {
        "repository": "{repo_name}",
        "timestamp": {triggered_timestamp},
        "alert": {
           "name": "{name}",
           "description": "{description}",
           "query": {
              "queryString": "{query_string} ",
              "end": "{query_time_end}",
              "start": "{query_time_start}"
          },
          "notifierID": "{action_id}",
          "id": "{id}"
        },
        "warnings": "{warnings}",
        "events": {events},
        "numberOfEvents": {event_count}
    }
  6. Because the default template body is formatted as JSON we've added a default Content-Type header of application-json. You can use this default body template or edit it to suit your needs. It need not be JSON, but using JSON enables you to use features of webhook that allow you to select pieces of the JSON document to pass to your script as command line arguments or environment variables.

  7. Test it by simply starting up the webhook executable in verbose mode and watch it execute your script.

    ini
    ./webhook -verbose -port 9999
    [webhook] 2018/12/12 12:09:48 version 2.6.9 starting
    [webhook] 2018/12/12 12:09:48 setting up os signal watcher
    [webhook] 2018/12/12 12:09:48 attempting to load hooks from hooks.json
    [webhook] 2018/12/12 12:09:48 os signal watcher ready
    [webhook] 2018/12/12 12:09:48 found 1 hook(s) in file
    [webhook] 2018/12/12 12:09:48    loaded: cleanup-webhook
    [webhook] 2018/12/12 12:09:48 serving hooks on http://0.0.0.0:9999/hooks/{id}
    [webhook] 2018/12/12 12:11:23 [7415e3] incoming HTTP request from [::1]:44872
    [webhook] 2018/12/12 12:11:23 [7415e3] cleanup-webhook got matched
    [webhook] 2018/12/12 12:11:23 [7415e3] error parsing JSON payload EOF
    [webhook] 2018/12/12 12:11:23 [7415e3] cleanup-webhook hook triggered successfully
    [webhook] 2018/12/12 12:11:23 200 | 132.063's | localhost:9999 | POST /hooks/cleanup-webhook
    [webhook] 2018/12/12 12:11:23 [7415e3] executing /var/scripts/cleanup-webhook.sh (/var/scripts/cleanup-webhook.sh) with arguments ["/var/scripts/cleanup-webhook.sh"] and environment [HOOK_payload={ ... JSON ...}} HOOK_headers={"Accept":"*/*","Content-Type":"application/json"} HOOK_query={}] using /tmp as cwd
    [webhook] 2018/12/12 12:11:23 [7415e3] command output:
    [webhook] 2018/12/12 12:11:23 [7415e3] finished handling cleanup-webhook
  8. Save your action and start the webhook server and the next time the alert fires your script on the remote system will be triggered.