Data Flow Pipelines

The pipeline is a series of processes or stages through which data flow systematically and efficiently. It helps to visualize the flow between nodes, rules, and filters applied for the data flow.

Create Pipeline View

Navigate to explore view, and click on the New Pipeline button to create a new pipeline view

On click of the New Pipeline button, a new dialog opens, select the desired date-time range, select the desired namespace, and click Create to create a new pipeline view.

On click of Create button a new pipeline tab will get added with the pipeline view as shown below

Pipeline View: The pipeline view shows the data flow between the nodes. each node is connected with the edges to show the data flow between the nodes. The node will have the following details.

1. Name of the node 2. Total number of Events 3. Total volume size of the events 4. Total savings for application and target node Minimap: Minimap is a tool for quick preview and exploration of large graphs. Time Range Slider: Helps to select the time range between the selected date and time Zoom In/Out: Tool to zoom in and zoom out the graph Side Bar: Helps to make the following changes 1. Change the theme 2. Change the default title of the pipeline tab 3. Select a different date time range 4. Select a different namespace, can do multi-select 5. Shows the overall EPS and Valume savings.

Add/Choose the existing rule(FILTER, EXTRACT, SIEM, TAG, REWRITE), for namespace and application.

Click the + icon to create or choose the existing rule

On click of + icon, Add Rule dialog will show, with the options 1. Choose Existing Rule 2. Create New Rule

Select the Choose Existing Rule to choose an existing rule for the namespace and application

Select Create New Rule button to create new rules for the namespace and application

The selected/created rule will be displayed under the respective group nodes

Add Rule(Forwarder)

Before adding the forwarder rule, create a forwarder and map the forwarder in the data flow tab.

After adding the forwarder the new node will be added as below

Click on the + icon to choose an existing or create a new forwarder rule for the application and target. After adding the forwarder rule, a new node will be shown between the application and the forwarder.

Rule group node

Each rule type will have separate rule groups(FILTER, EXTRACT, SIEM, TAG, REWRITE, FORWARDER). Each rule node will have the following details. 1. Rule active status within the time range 2. Info icon, on click will show the rule details 3. Name of the rule 4. Savings done by the rule if any