|
Basic Query Structure
|
Check for missing/mismatched parentheses
() - try using an editor with
bracket matching (including the
Query Editor).
Verify pipe characters (|)
between each query step. Pipes can be omitted for filter
expressions, but other elements of the pipeline are required
to be separated by the pipe symbol.
Remove any spaces from function names and confirm the case-
some functions are mixed case.
Confirm all commas are present between function parameters,
that parameters have a name, and that the correct value or
markup is applied. See Datatypes in CQL.
Ensure proper quotation usage and pay attention to whether the
start and end of the string have the correct quote.
|
|
Field Verification
|
Confirm the field exists in the dataset. Make use of the
Event Fields panel to
determine which fields are available at different stages of
the query.
Check that field name case matches exactly. Field names are
case sensitive, especially when using a parser from a standard
package (see Package Marketplace) which are
standardized to a mixed case fieldname.
Put double quotes around field names that include hyphens or
spaces.
Verify correct format for specialist fields where the
interpretation of the datatype is significant. For example, a
field that should be interpreted as an array requires
fieldname[]. See
Datatypes in CQL.
Check field type matches function requirements;
LogScale will attempt to translate the value to the
right type during execution. See
Datatypes in CQL.
|
|
Time-Related Checks
|
Verify data has timestamps, and that the right field with a
timestamp value is used. Timestamps should be recorded as an
Epoch value.
Use parseTimestamp() if the value is
required to be used as a timestamp value. LogScale
does not automatically translate human-readable dates or time
strings- relying on a time value for sorting, filtering,
and/or selection means the value will need to be parsed first.
Check time range settings are appropriate for the data.
Whether querying through the UI or API, ensure the data is
within the timestamp range.
Verify timezone configurations.
If the data or fields containing a timestamp are not clear or
consistent, check whether using
findTimestamp() is better for looking for
a potential value- using parseTimestamp()
can sometimes be more explicit than necessary.
|
|
Data Quality Checks
|
Verify data exists in the selected time range. Use the UI and
Change the Time Interval to view the
data across different ranges.
Check for null or empty values in the data. These may cause
data to be hidden or skipped due to match errors. Use the
default() function to set a value if one
has not been set.
Look for unexpected characters or whitespaces. If data has
been copied from other sources, it may include additional
spaces or hidden Unicode characters.
Confirm the data matches the expected format. For example,
confirm that timestamps are correctly identified, or
identifies values in an array or field.
|
|
Function Implementation
|
Verify that function names are spelled correctly, use the
correct case, and use the colon correctly.
Ensure that required parameters are included. Check the
function definitions to ensure all required parameters have
been provided, and the default values applied to unspecified
parameters are not affecting the way the query is executed.
Check the pipeline function order to ensure it's correct and
logical. For example, a user might sort on a field that is
only generated by a function after using the
sort() function, then ensure that
different functions have not been used in the wrong location.
Ensure aggregations are used when required. Aggregations
summarize data, and should be used both to simplify and
summarize data. However, remember that aggregation will
combine values of multiple events into a simplified set. This
reduces the event set, removes field selection to only
aggregated values, but also changes how the data can be
processed or filtered later.
Verify aggregations are in the correct position. Aggregating
too early may hide data that is needed, or create very large
aggregated datasets that still require significant processing.
Test all nested functions individually first. Try executing
each pipeline and/or function in isolation to ensure that the
input and output of the function matches what is being
processed. Stepping through each query by each individual
pipeline element helps confirm the execution and expected
values.
|
|
Logic and Filtering
|
Verify the operators AND and
OR are used correctly.
Remember that LogScale binds operators together with
OR closer than
AND, unlike many other
database environments.
Check comparison operators (=, !=,
>, <) are used correctly and
performing the right logical operation.
Test regular expressions. Regular expressions can be complex,
so confirming that the right information is being extracted,
filtered, and/or identified before including them as part of a
larger query is critical.
Avoid over-filtering too early in a query. While early
filtering is encouraged, it can filter events before later
stages like aggregating or reformatting are able to operate on
the data, restricting results.
|
|
Performance Considerations
|
Narrow the time range when possible. Large time ranges require
loading a lot of data. If the time range is older, then it
will need to be loaded from slower storage. If the time range
is large, it may require a significant amount of memory to
process the results.
Minimize wildcard usage; wildcards require significant memory
for processing, and may ultimately include more data than you
expect or actually need to process.
Reduce complex regular expression operations. Regular
expressions must be executed individually on each field of
each event. Across 10 million events, this is a significant
resource (CPU and memory) cost.
Remove unnecessary grouping/sorting. Sorting data before it is
closer to the final dataset usually provides no benefit.
Unless you need grouping or sorting as part of the query
process, reserving that part of the process until the end
avoids keeping the entire event set in memory
Break complex queries into steps and monitor the output. Try
each pipeline and identify how many events or fields are
output at each stage to determine whether each event set is
required. For example, filter, aggregate, filter the aggregate
set, then format the output if you determine too many values
are returned from a given stage.
|
|
Visualization Issues
|
Select the appropriate widget for the given data type. The
user interface will usually pick a suitable widget based on
the input data, but also consider what you are displaying and
how it could be displayed.
Verify required fields exist for the chosen visualization.
Visualizations pick the data and fields automatically, but if
showing multiple datasets, make sure they're configured
correctly.
Check if aggregation is needed for visualization. Most
visualizations (charts, etc.) require aggregation. Timecharts
are aggregations over time, and most other types are flexible
for the X or Y axes on what can be displayed.
Confirm data format matches visualization requirements.
|
|
Access and Repository
|
Verify the correct repository is selected. If querying a view,
make sure the view relates to the right repositories, and that
a prefilter has not been configured on the repository to
filter the events included in the view.
Check user permissions to access the data, particularly when
using a view.
Confirm the repository is receiving data. If ingestion has
recently been configured, ensure that the right repository is
being used, correct ingest token is applied, and appropriate
parser has been selected.
Verify retention settings for the repository to confirm that
data has not expired from the repository as part of the
retention settings. The UI will not allow you to select data
older than the configured repository retention time.
|
|
Best Practice Steps
|
Build queries incrementally. When building a complex query,
add each pipeline step individually to confirm the input and
output events are valid and reporting the correct values.
Test each step separately. If necessary, try each individual
pipeline in isolation to test whether the function or
statement is operating in the way you expect.
Document complex queries with comments. Comments can assist
with readability and with explaining what you are trying to
achieve at each step. See
Comments.
Start with broader filters. If at first there are no events
returned, try widening your filters (or removing them
altogether) to validate the operation against a wider dataset.
It's possible the filters may be removing all content before
it can be processed.
|
