When you first create a new Pipeline, it's empty, making it equivalent to Passthru. CrowdStream provides a set of out of the box Pipelines for specific purposes. As you can see below, one of these is a passthru Pipeline. This similarly contains no Functions — making it the Data Routes UI's equivalent of QuickConnect's Passthru button.

For these reasons, it's easy to start with an empty Pipeline, and then add Functions as you need them.

A Pipeline, as you populate it, is a stack of processing Functions and optional comments. Use the accordions on the left to expand or collapse each Function. Use the sliders on the left to enable/disable each Function — preserving structure as you develop and debug.

Use the right Sample Data pane to preview how the Pipeline's current state transforms a set of events. (Controls here enable you to capture live data, save it to sample or Datagen files, and reload those files.)

Although you'll configure most data shaping in processing Pipelines between Sources and Destinations, you can also attach a pre processing Pipeline to any Source instance, and a post processing Pipeline to any Destination instance. Use these options when you need to “condition” the specific data type handled by a particular inbound or outbound integration.

To select a connection type in QuickConnect, click the appropriate button in the pop up dialog shown above. To accept the default Passthru option, just click Save. Your Source and Destination will now be connected.

If you select Pipeline, you'll see an Add Pipeline to Connection modal. Here, you can click to select an existing Pipeline, or click Add Pipeline to define a new one. Then click Save here to establish the connection.

Selecting Pack opens an Add Pack to Connection modal that works exactly the same way.

Whichever option you choose here, your connection between this Source and Destination is now complete.

To get data flowing, jump ahead to commit and deploy your new configuration.

The Data Routes UI enables you to finely configure multiple, interdependent connections between Sources and Destinations. Events clone and cascade across a Routing table, based on the filtering rules and sequence you define.

In exchange for this flexibility, you sacrifice QuickConnect's one stop, snap together UI metaphor. After configuring your Sources and Destinations — which you've already done above — you need to connect them by separately setting up at least one Pipeline and one Route.

From a Group's Manage tab, select Processing > Pipelines. In addition to the minimal passthru Pipeline identified above, other simple building blocks here include main (which initially contains only a simple Eval Function) and devnull (which contains a single Function that just drops events).

Out of the box Pipelines are already connected to Routes. When you click into a Pipeline, you can hover to display the connected Route, as shown here. But as our final connection step, let's next look at how to connect up everything (Source, Pipeline, Destination) in a Route.

In the Data Routes UI, a Route is where you complete a data flow connection. Each Route connects exactly one Pipeline with exactly one Destination.

This combination can consume data from one, multiple, or all the Sources you've configured on the Data Routes side. (QuickConnect Sources are separate, but you can open those Sources' configs to move them to the Data Routes UI.) This is governed by the filtering we'll see below.

To open the Routing table, from a Group's Manage tab, select Routing > Data Routes.

As you can see, CrowdStream ships with one starter Route named default. It's configured as follows:

To get data flowing, jump ahead to commit and deploy your new configuration.

When you want a Route to ingest data from only one Source, or a subset of Sources, you expand its Filter expression beyond the default true catch all.

Filter is itself a drop down. Open it to select among typeahead suggestions for several common inbound data types.

You can select one of these expressions and then modify it, using the same __inputId syntax. In the example expression below, note that the identifier after the : is — as with the Output options we saw above — the arbitrary name of one Syslog Source instance:

__inputId.startsWith('syslog:in_syslog:')

For More Information see Cribl Expression Syntax > Filters