Redact Events API

Security Requirements and Controls

Change data deletion permissions permission
ChangeDataDeletionPermissions API permission

Falcon LogScale has support for redacting individual events from the compressed segment files.

The redactEvents() API is intended to support removal of a small number of events from LogScale, allowing you to eliminate specific events that must be removed. For example:

Removing personally identifiable information (e.g., due to a GDPR request)
Removing accidentally logged passwords
Removing confidential data

The redaction API makes no promises that redacting events will recover disk space, and is not intended to support data management or bulk deletion. If you want to bulk delete data, you may want to set Data Retention, or use the GraphQL API to delete the relevant dataspaces or datasources.

The redaction mechanism works by:

Initially excluding the events you mark for redaction from future queries by filtering all query results
Once LogScale determines that it is safe to do so, it will rewrite the affected segments, excluding the events that were marked for redaction.

As rewriting segments is an expensive operation, we strongly discourage using this API in cases for which appropriate retention settings, or explicit deletions of dataspaces, would suffice.

Note

Field Aliasing is not enabled when using deletion filter queries. Ensure that you are not using aliased fields in the filter query executed through this API. For more information, see Searches with Query Prefixes.

Submitting a Redaction Request

The redactEvents() GraphQL mutation can be used to submit deletions.

This is an example redacting all events with a password field in the specified time interval in milliseconds.

Raw

graphql

mutation {
  redactEvents(
    input: {
      repositoryName: "humio"
      start: "2021-09-17T03:00:00.000Z"
      end: "2021-09-17T03:15:00.000Z"
      query: "password=*"
    }
  )
}

Mac OS or Linux (curl)

shell

curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{"query" : "mutation {
  redactEvents(
    input: {
      repositoryName: \"humio\"
      start: \"2021-09-17T03:00:00.000Z\"
      end: \"2021-09-17T03:15:00.000Z\"
      query: \"password=*\"
    }
  )
}"
}
EOF

Mac OS or Linux (curl) One-line

shell

curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{"query" : "mutation {
  redactEvents(
    input: {
      repositoryName: \"humio\"
      start: \"2021-09-17T03:00:00.000Z\"
      end: \"2021-09-17T03:15:00.000Z\"
      query: \"password=*\"
    }
  )
}"
}
EOF

Windows Cmd and curl

shell

curl -v -X POST $YOUR_LOGSCALE_URL/graphql ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d @'{"query" : "mutation { ^
  redactEvents( ^
    input: { ^
      repositoryName: \"humio\" ^
      start: \"2021-09-17T03:00:00.000Z\" ^
      end: \"2021-09-17T03:15:00.000Z\" ^
      query: \"password=*\" ^
    } ^
  ) ^
}" ^
} '

Windows Powershell and curl

powershell

curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '{"query" : "mutation {
  redactEvents(
    input: {
      repositoryName: \"humio\"
      start: \"2021-09-17T03:00:00.000Z\"
      end: \"2021-09-17T03:15:00.000Z\"
      query: \"password=*\"
    }
  )
}"
}'
"$YOUR_LOGSCALE_URL/graphql"

Perl

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $INGEST_TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/graphql';

my $json = '{"query" : "mutation {
  redactEvents(
    input: {
      repositoryName: \"humio\"
      start: \"2021-09-17T03:00:00.000Z\"
      end: \"2021-09-17T03:15:00.000Z\"
      query: \"password=*\"
    }
  )
}"
}';
my $req = HTTP::Request->new("POST", $uri );

$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");

$req->content( $json );

my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";

Python

python

#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/graphql'
mydata = r'''{"query" : "mutation {
  redactEvents(
    input: {
      repositoryName: \"humio\"
      start: \"2021-09-17T03:00:00.000Z\"
      end: \"2021-09-17T03:15:00.000Z\"
      query: \"password=*\"
    }
  )
}"
}'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

Node.js

javascript

const https = require('https');

const data = JSON.stringify(
    {"query" : "mutation {
  redactEvents(
    input: {
      repositoryName: \"humio\"
      start: \"2021-09-17T03:00:00.000Z\"
      end: \"2021-09-17T03:15:00.000Z\"
      query: \"password=*\"
    }
  )
}"
}
);


const options = {
  hostname: '$YOUR_LOGSCALE_URL/graphql',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

The mutation will return the ID of the submitted redaction task:

json

{
   "data" : {
      "redactEvents" : "e4G6TWjXVxbNyF5hDLrvBJAV"
   }
}

Viewing Existing Requests

The redactEvents() GraphQL query will return the list of redaction tasks that have not yet completed segment rewrites.

Raw

graphql

query {
  redactEvents(repositoryName:"humio")
  {id, created, start, end, query}
}

Mac OS or Linux (curl)

shell

curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{"query" : "query {
  redactEvents(repositoryName:\"humio\")
  {id, created, start, end, query}
}"
}
EOF

Mac OS or Linux (curl) One-line

shell

curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{"query" : "query {
  redactEvents(repositoryName:\"humio\")
  {id, created, start, end, query}
}"
}
EOF

Windows Cmd and curl

shell

curl -v -X POST $YOUR_LOGSCALE_URL/graphql ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d @'{"query" : "query { ^
  redactEvents(repositoryName:\"humio\") ^
  {id, created, start, end, query} ^
}" ^
} '

Windows Powershell and curl

powershell

curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '{"query" : "query {
  redactEvents(repositoryName:\"humio\")
  {id, created, start, end, query}
}"
}'
"$YOUR_LOGSCALE_URL/graphql"

Perl

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $INGEST_TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/graphql';

my $json = '{"query" : "query {
  redactEvents(repositoryName:\"humio\")
  {id, created, start, end, query}
}"
}';
my $req = HTTP::Request->new("POST", $uri );

$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");

$req->content( $json );

my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";

Python

python

#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/graphql'
mydata = r'''{"query" : "query {
  redactEvents(repositoryName:\"humio\")
  {id, created, start, end, query}
}"
}'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

Node.js

javascript

const https = require('https');

const data = JSON.stringify(
    {"query" : "query {
  redactEvents(repositoryName:\"humio\")
  {id, created, start, end, query}
}"
}
);


const options = {
  hostname: '$YOUR_LOGSCALE_URL/graphql',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

Returns:

json

{
   "data" : {
      "redactEvents" : "e4G6TWjXVxbNyF5hDLrvBJAV"
   }
}

It is possible to cancel submitted redactions via the cancelRedactEvents() mutation.

Cancellation is best-effort, and if events have already been redacted from segments, they will not be restored.

Field Aliasing is not enabled when using a deletion filter query, meaning that aliased fields won't be available for queries executed through this API. Ensure that you are not using aliased fields in the filter query used with the API. For more information, see Searches with Query Prefixes.

APIs

API Authentication

Cluster Management API

Health Check API

Ingest API

Lookup API

Redact Events API

Search API

Packages API

Software Libraries

Redact Events API

Security Requirements and Controls

Note

Submitting a Redaction Request

Viewing Existing Requests

Version Support

Enter search term