Documentation

Connections

Understand connection types, locators, authentication configs, and automated record version deduplication.

← Back to Documentation Hub / Data Automation

A Connection acts as your data pipeline's access controller. It defines where your external data resides and how the engine authenticates with that data source.


Key Structural Components

  1. Kind: Specifies the data storage engine dialect (BigQuery, ClickHouse, Deltatable, Parquet, CSV, JSON, Rest).
  2. Locator: Defines the precise address of your dataset (such as table names, file paths, or cloud storage URLs).
  3. Config: Supplies authentication parameters, API keys, or cloud credentials.
connection:
  kind: Parquet
  locator: sales/q1.parquet
  config:
    kind: s3
    bucket: my-company-data
    region: us-east-1
    access_key_id: "<<secrets.get('AWS_KEY')>>"
    secret_access_key: "<<secrets.get('AWS_SECRET')>>"

Supported Connection Types

Connection Kind Description Use Cases
Parquet Optimized columnar files Fast analytical querying over cloud files.
Deltatable Versioned data lakes Historical change tracking and append logs.
BigQuery Google cloud warehouse Large enterprise analytical tables.
ClickHouse Real-time analytical store High-speed transactional analytics.
CSV / JSON Simple structured files Export files and lightweight integrations.
HTTP Web API endpoints Ingesting data from SaaS services.
File File connections XLSX, CSV and many other formats.

Source Query Filtering

When reading from large database tables or data lakes (BigQuery, Delta Lake, ClickHouse), add a query block directly to the connection to prune incoming records at the database level:

connection:
  kind: Deltatable
  locator: customer_orders
query: |
  SELECT customer_id, order_date, total_amount 
  FROM Deltatable 
  WHERE order_date >= '2026-01-01'

Automated Record Deduplication

When querying version-tracked data tables (like Delta tables), the connection engine automatically deduplicates rows to return the latest record per primary key.

Manual deduplication:

/* Automatically applied window deduplication */
SELECT * FROM (
  SELECT *, ROW_NUMBER() OVER (PARTITION BY tenant, customer_id ORDER BY last_updated DESC) AS __rank__
  FROM sales_data
) WHERE __rank__ = 1

Automatic deduplication:

connection:
  kind: Deltatable
  locator: customer_orders
  key: [tenant, customer_id]
  version: last_updated
  qualifier: RowNumber
query: |
  SELECT customer_id, order_date, total_amount 
  FROM Deltatable 
  WHERE order_date >= '2026-01-01'

Automatic deduplication allows to write semantic queries that answer business questions rather than addressing storage layer nuances.


← Back to Documentation Hub / Data Automation