Function Input/Output

Before looking at the specifics of the different Function Types, functions operate as part of the pipeline of processing that occurs when a query is executed, as detailed in Query Pipeline. The operation of different functions falls into three broad categories:

  • Event Modifying Functions

    Functions that modify each event in the event set; for example adding a field, or modifying/updating the value of a field in each event.

    For example, given the input data:

    user
    kingm
    edwardst
    howardn
    woodm
    walkerd
    blackj
    rogersd
    thomasb
    stevensp

    The query:

    logscale
    upper(user,as=upperuser)

    Creates a new field, upperuser:

    userupperuser
    kingmKINGM
    edwardstEDWARDST
    howardnHOWARDN
    woodmWOODM
    walkerdWALKERD
    blackjBLACKJ
    rogersdROGERSD
    thomasbTHOMASB
    stevenspSTEVENSP

    When modifying events in this way all of the event data and fields for each event is retained.

  • Event Creation Functions

    Functions that add new events to the event set, or that remove events from the event set. This has the affect of changing the list of events in each set. For example, using split() to split an array duplicates the original event for each element of the array. So with this input data (where each is row is an event):

    permlist[0]permlist[1]permlist[2]userpidcmd
    engineeringreliabilityperformanceevansm43219/usr/bin/ssh
    salesrenewals<no value>edwardst25678/sbin/iptables
    productinnovation<no value>davisr50123/usr/bin/curl

    Shows three events, and using:

    logscale
    split(permlist)
    permlistuserpidcmd
    performanceevansm43219/usr/bin/ssh
    reliabilityevansm43219/usr/bin/ssh
    engineeringevansm43219/usr/bin/ssh
    renewalsedwardst25678/sbin/iptables
    salesedwardst25678/sbin/iptables
    innovationdavisr50123/usr/bin/curl
    productdavisr50123/usr/bin/curl

    Now with a result set of seven events. The original list of events has been updated with the new set, similar to aggregation, but with similar information.

    Other functions that perform this creation of events that may be similar to (or completely replace) the original set include defineTable(), join functions, including join(), and readFile().

  • Event Set Modifying Functions

    Functions that replace the incoming event set with a new set of events and fields; most often this is due to an aggregation, but it might also be functions that create new events such as copyEvent() or remove events such as dropEvent(), or joins where new events are composed based combination of two or more events.

    With groupBy() for example, the input data:

    userpidcmdperms
    evansm43219/usr/bin/sshengineering,reliability,performance
    edwardst25678/sbin/iptablessales,renewals
    davisr50123/usr/bin/curlproduct,innovation
    cooperp11789/bin/psfinance,billing
    clarkd33456/usr/bin/topoperations,process_improvement
    brownr57891/sbin/ifconfighr,diversity_inclusion
    blackj20123/usr/bin/findmarketing,partner_marketing
    bakerm45678/bin/lsit,automation
    andersonk27891/usr/sbin/useraddengineering,ai,ml
    ...   

    The events and fields are completely replaced with the aggregated dataset:

    user_count
    adamsb3
    andersonk3
    bakerm3
    blackj3
    brownr3
    clarkd3
    cooperp3
    ...Â