Connections
Understand connection types, locators, authentication configs, and automated record version deduplication.
← Back to Documentation Hub / Data Automation
A Connection acts as your data pipeline's access controller. It defines where your external data resides and how the engine authenticates with that data source.
Key Structural Components
- Kind: Specifies the data storage engine dialect (
BigQuery,ClickHouse,Deltatable,Parquet,CSV,JSON,Rest). - Locator: Defines the precise address of your dataset (such as table names, file paths, or cloud storage URLs).
- Config: Supplies authentication parameters, API keys, or cloud credentials.
connection:
kind: Parquet
locator: sales/q1.parquet
config:
kind: s3
bucket: my-company-data
region: us-east-1
access_key_id: "<<secrets.get('AWS_KEY')>>"
secret_access_key: "<<secrets.get('AWS_SECRET')>>"
Supported Connection Types
| Connection Kind | Description | Use Cases |
|---|---|---|
Parquet |
Optimized columnar files | Fast analytical querying over cloud files. |
Deltatable |
Versioned data lakes | Historical change tracking and append logs. |
BigQuery |
Google cloud warehouse | Large enterprise analytical tables. |
ClickHouse |
Real-time analytical store | High-speed transactional analytics. |
CSV / JSON |
Simple structured files | Export files and lightweight integrations. |
HTTP |
Web API endpoints | Ingesting data from SaaS services. |
File |
File connections | XLSX, CSV and many other formats. |
Source Query Filtering
When reading from large database tables or data lakes (BigQuery, Delta Lake, ClickHouse), add a query block directly to the connection to prune incoming records at the database level:
connection:
kind: Deltatable
locator: customer_orders
query: |
SELECT customer_id, order_date, total_amount
FROM Deltatable
WHERE order_date >= '2026-01-01'
Automated Record Deduplication
When querying version-tracked data tables (like Delta tables), the connection engine automatically deduplicates rows to return the latest record per primary key.
Manual deduplication:
/* Automatically applied window deduplication */
SELECT * FROM (
SELECT *, ROW_NUMBER() OVER (PARTITION BY tenant, customer_id ORDER BY last_updated DESC) AS __rank__
FROM sales_data
) WHERE __rank__ = 1
Automatic deduplication:
connection:
kind: Deltatable
locator: customer_orders
key: [tenant, customer_id]
version: last_updated
qualifier: RowNumber
query: |
SELECT customer_id, order_date, total_amount
FROM Deltatable
WHERE order_date >= '2026-01-01'
Automatic deduplication allows to write semantic queries that answer business questions rather than addressing storage layer nuances.