Data Automation Overview
Discover how Data Automation pipelines ingest, transform, and deliver data across cloud sources, the Forecasting Engine, and Dashboards & Reports.
Data Automation pipelines automate data movement and transformation across your organization. Ingest raw files from cloud storage, databases, or web APIs, clean records using SQL, and load structured tables directly into the Forecasting Engine and Dashboards & Reports.
Pipeline Documentation Topics
- Pipeline Basics: Learn the Source → Transform → Sink architecture pattern.
- Connections: Connect cloud stores, files, and lakehouses.
- REST Connection: Ingest data from web APIs with automated authentication and pagination.
In-Browser playground.
What This Pipeline Does
Business Goal: Analyze e-commerce sales data to identify top markets and customer purchasing patterns.
Workflow Steps:
- Load Data: Reads analytics-ready Parquet datasets (converted from raw file inputs).
- Clean & Prepare: Removes invalid transactions and computes row-level line totals.
- Analyze: Groups metrics by country to calculate sales performance.
- Deliver Insights: Returns top markets and average order metrics.
Business Impact
- Centralized Aggregation: Consolidate disparate transactional files into unified analytical views.
- Automated Processing: Eliminate manual spreadsheet cleaning and reduce routine data prep work.
- Faster Decision-Making: Supply updated data directly to the Forecasting Engine and Dashboards & Reports.