This page has recently been moved to this URL; please update any bookmarks or links
Falcon LogScale has support for redacting individual events from the compressed
segment files.
The redactEvents() API is intended to support
removal of a small number of events from LogScale, allowing you to
eliminate specific events that must be removed. For example:
Removing personally identifiable information (e.g., due to a GDPR
request)
Removing accidentally logged passwords
Removing confidential data
The redaction API makes no promises that redacting events will recover disk
space, and is not intended to support data management or bulk deletion. If
you want to bulk delete data, you may want to set
Data Retention , or use the
GraphQL API to delete the relevant
dataspaces or datasources.
The redaction mechanism works by:
Initially excluding the events you mark for redaction from future
queries by filtering all query results
Once LogScale determines that it is safe to do so, it will
rewrite the affected segments, excluding the events that were marked for
redaction.
As rewriting segments is an expensive operation, we strongly discourage
using this API in cases for which appropriate retention settings, or
explicit deletions of dataspaces, would suffice.
Note
Field Aliasing is not enabled when using
deletion filter queries. Ensure that you are not using aliased fields in
the filter query executed through this API. For more information, see
Searches with Query Prefixes .
Submitting a Redaction Request
The redactEvents() GraphQL mutation can be used
to submit deletions.
This is an example redacting all events with a password field in the
specified time interval in milliseconds.
Show:
Raw Mac OS or Linux (curl) Mac OS or Linux (curl) One-line Windows Cmd and curl Windows Powershell and curl Perl Python Node.js
Raw mutation {
redactEvents(
input : {
repositoryName : "humio"
start : "2021-09-17T03:00:00.000Z"
end : "2021-09-17T03:15:00.000Z"
query : "password=*"
}
)
}
Mac OS or Linux (curl) curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d @- << EOF
{"query" : "mutation {
redactEvents(
input: {
repositoryName: \"humio\"
start: \"2021-09-17T03:00:00.000Z\"
end: \"2021-09-17T03:15:00.000Z\"
query: \"password=*\"
}
)
}"
}
EOF
Mac OS or Linux (curl) One-line curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d @- << EOF
{"query" : "mutation {
redactEvents(
input: {
repositoryName: \"humio\"
start: \"2021-09-17T03:00:00.000Z\"
end: \"2021-09-17T03:15:00.000Z\"
query: \"password=*\"
}
)
}"
}
EOF
Windows Cmd and curl curl -v -X POST $YOUR_LOGSCALE_URL/graphql ^
-H "Authorization: Bearer $TOKEN" ^
-H "Content-Type: application/json" ^
-d @'{"query" : "mutation { ^
redactEvents( ^
input: { ^
repositoryName: \"humio\" ^
start: \"2021-09-17T03:00:00.000Z\" ^
end: \"2021-09-17T03:15:00.000Z\" ^
query: \"password=*\" ^
} ^
) ^
}" ^
} '
Windows Powershell and curl curl.exe -X POST
-H "Authorization: Bearer $TOKEN "
-H "Content-Type: application/json"
-d '{"query" : "mutation {
redactEvents(
input: {
repositoryName: \"humio\"
start: \"2021-09-17T03:00:00.000Z\"
end: \"2021-09-17T03:15:00.000Z\"
query: \"password=*\"
}
)
}"
}'
"$YOUR_LOGSCALE_URL /graphql"
Perl
use HTTP::Request;
use LWP;
my $INGEST_TOKEN = "TOKEN" ;
my $uri = '$YOUR_LOGSCALE_URL/graphql' ;
my $json = '{"query" : "mutation {
redactEvents(
input: {
repositoryName: \"humio\"
start: \"2021-09-17T03:00:00.000Z\"
end: \"2021-09-17T03:15:00.000Z\"
query: \"password=*\"
}
)
}"
}' ;
my $req = HTTP::Request->new("POST" , $uri );
$req->header("Authorization" => "Bearer $TOKEN" );
$req->header("Content-Type" => "application/json" );
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content" },"\n" ;
Python
import requests
url = '$YOUR_LOGSCALE_URL/graphql'
mydata = r'''{"query" : "mutation {
redactEvents(
input: {
repositoryName: \"humio\"
start: \"2021-09-17T03:00:00.000Z\"
end: \"2021-09-17T03:15:00.000Z\"
query: \"password=*\"
}
)
}"
}'''
resp = requests.post(url,
data = mydata,
headers = {
"Authorization" : "Bearer $TOKEN" ,
"Content-Type" : "application/json"
}
)
print (resp.text)
Node.js const https = require ('https' );
const data = JSON .stringify (
{"query" : "mutation {
redactEvents(
input: {
repositoryName: \"humio\"
start: \"2021-09-17T03:00:00.000Z\"
end: \"2021-09-17T03:15:00.000Z\"
query: \"password=*\"
}
)
}"
}
);
const options = {
hostname : '$YOUR_LOGSCALE_URL/graphql' ,
path : '/graphql' ,
port : 443 ,
method : 'POST' ,
headers : {
'Content-Type' : 'application/json' ,
'Content-Length' : data.length ,
Authorization : 'BEARER ' + process.env .TOKEN ,
'User-Agent' : 'Node' ,
},
};
const req = https.request (options, (res ) => {
let data = '' ;
console .log (`statusCode: ${res.statusCode} ` );
res.on ('data' , (d ) => {
data += d;
});
res.on ('end' , () => {
console .log (JSON .parse (data).data );
});
});
req.on ('error' , (error ) => {
console .error (error);
});
req.write (data);
req.end ();
The mutation will return the ID of the submitted redaction task:
{
"data" : {
"redactEvents" : "e4G6TWjXVxbNyF5hDLrvBJAV"
}
}
Viewing Existing Requests
The redactEvents() GraphQL query will return the list
of redaction tasks that have not yet completed segment rewrites.
Show:
Raw Mac OS or Linux (curl) Mac OS or Linux (curl) One-line Windows Cmd and curl Windows Powershell and curl Perl Python Node.js
Raw query {
redactEvents( repositoryName : "humio" )
{ id, created, start, end, query }
}
Mac OS or Linux (curl) curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d @- << EOF
{"query" : "query {
redactEvents(repositoryName:\"humio\")
{id, created, start, end, query}
}"
}
EOF
Mac OS or Linux (curl) One-line curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d @- << EOF
{"query" : "query {
redactEvents(repositoryName:\"humio\")
{id, created, start, end, query}
}"
}
EOF
Windows Cmd and curl curl -v -X POST $YOUR_LOGSCALE_URL/graphql ^
-H "Authorization: Bearer $TOKEN" ^
-H "Content-Type: application/json" ^
-d @'{"query" : "query { ^
redactEvents(repositoryName:\"humio\") ^
{id, created, start, end, query} ^
}" ^
} '
Windows Powershell and curl curl.exe -X POST
-H "Authorization: Bearer $TOKEN "
-H "Content-Type: application/json"
-d '{"query" : "query {
redactEvents(repositoryName:\"humio\")
{id, created, start, end, query}
}"
}'
"$YOUR_LOGSCALE_URL /graphql"
Perl
use HTTP::Request;
use LWP;
my $INGEST_TOKEN = "TOKEN" ;
my $uri = '$YOUR_LOGSCALE_URL/graphql' ;
my $json = '{"query" : "query {
redactEvents(repositoryName:\"humio\")
{id, created, start, end, query}
}"
}' ;
my $req = HTTP::Request->new("POST" , $uri );
$req->header("Authorization" => "Bearer $TOKEN" );
$req->header("Content-Type" => "application/json" );
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content" },"\n" ;
Python
import requests
url = '$YOUR_LOGSCALE_URL/graphql'
mydata = r'''{"query" : "query {
redactEvents(repositoryName:\"humio\")
{id, created, start, end, query}
}"
}'''
resp = requests.post(url,
data = mydata,
headers = {
"Authorization" : "Bearer $TOKEN" ,
"Content-Type" : "application/json"
}
)
print (resp.text)
Node.js const https = require ('https' );
const data = JSON .stringify (
{"query" : "query {
redactEvents(repositoryName:\"humio\")
{id, created, start, end, query}
}"
}
);
const options = {
hostname : '$YOUR_LOGSCALE_URL/graphql' ,
path : '/graphql' ,
port : 443 ,
method : 'POST' ,
headers : {
'Content-Type' : 'application/json' ,
'Content-Length' : data.length ,
Authorization : 'BEARER ' + process.env .TOKEN ,
'User-Agent' : 'Node' ,
},
};
const req = https.request (options, (res ) => {
let data = '' ;
console .log (`statusCode: ${res.statusCode} ` );
res.on ('data' , (d ) => {
data += d;
});
res.on ('end' , () => {
console .log (JSON .parse (data).data );
});
});
req.on ('error' , (error ) => {
console .error (error);
});
req.write (data);
req.end ();
Returns:
{
"data" : {
"redactEvents" : "e4G6TWjXVxbNyF5hDLrvBJAV"
}
}
It is possible to cancel submitted redactions via the
cancelRedactEvents() mutation.
Cancellation is best-effort, and if events have already been redacted from
segments, they will not be restored.
Field Aliasing is not enabled when using
a deletion filter query, meaning that aliased fields won't be available
for queries executed through this API. Ensure that you are not using
aliased fields in the filter query used with the API. For more
information, see
Searches with Query Prefixes .