Creating Workflows

Workflow Builder Interface

The workflow builder provides a visual canvas where you can design your data pipelines by dragging and connecting nodes.

Interface Components

• Node Palette - Library of available node types
• Canvas - Workspace for building the workflow
• Properties Panel - Configure selected node settings
• Toolbar - Actions like save, test, run, zoom

Node Types

Source Nodes

Read data from various sources:

• Database Query - Execute SQL queries
• Table Read - Read entire tables
• File Read - Read files from storage
• API Request - Fetch data from REST APIs

Transform Nodes

Process and transform data:

• Filter - Filter rows based on conditions
• Map - Transform column values
• Aggregate - Group and aggregate data
• Join - Combine data from multiple sources
• Sort - Order data by columns
• Deduplicate - Remove duplicate records

Destination Nodes

Write data to targets:

• Database Write - Insert, update, or upsert to tables
• File Write - Save to CSV, JSON, Parquet files
• Cloud Upload - Upload to S3, Azure, GCS
• API Post - Send data to REST endpoints

Logic Nodes

Control workflow execution:

• Condition - Branch based on conditions
• Loop - Iterate over data
• Error Handler - Handle errors gracefully
• Notification - Send alerts and notifications

Building a Workflow

1. Add Source Node

   Drag a source node to the canvas and configure the data source

2. Add Transform Nodes

   Add transformation nodes to process the data as needed

3. Connect Nodes

   Drag from output port to input port to connect nodes

4. Configure Settings

   Select each node and configure its properties

5. Add Destination

   Add a destination node to specify where data should be written

6. Test the Workflow

   Use the Test button to run with sample data

7. Save and Schedule

   Save the workflow and optionally set up a schedule