HTTP Connection
← Back to Documentation Hub / Data Automation
Overview
An HTTP Connection lets your data pipelines exchange data directly with web applications and REST APIs. Data Automation uses HTTP connections to:
- Fetch data from external services (Salesforce, Shopify, CRM, or marketing tools).
- Push processed results to third-party endpoints or internal web applications.
- Automate recurring data transfers across business applications.
Complete HTTP Connection Reference Schema
connection:
kind: HTTP # REQUIRED: Must be "HTTP" or "Rest"
# ====== CLIENT CONFIGURATION (API Identity & Connection) ======
client: # REQUIRED: API client settings
base_url: "[https://api.example.com/v1](https://api.example.com/v1)" # REQUIRED: Base API URL
timeout: 30 # OPTIONAL: Timeout in seconds (default 5, 0 = no timeout)
# --- Authentication ---
auth: # CONDITIONAL: Required if API needs authentication
method: basic # REQUIRED: basic | digest | header | oauth2 | bearer_token | api_key
values: # REQUIRED: Key-value pairs for chosen auth method
username: "user"
password: "<<secrets.get('API_PASS')>>"
# --- Advanced Client Settings ---
headers: {} # OPTIONAL: Additional HTTP headers for all requests
proxies: {} # OPTIONAL: Proxy configuration if needed
verify_ssl: true # OPTIONAL: Verify SSL certificates (default true)
# ====== REQUEST CONFIGURATION ======
request: # REQUIRED: Request definition
method: get # REQUIRED: get | post | put | patch | delete
endpoint: "/customers" # REQUIRED: API endpoint path
content_type: "application/json" # REQUIRED: Content type header
# --- Request Parameters ---
query: # OPTIONAL: URL query parameters
status: "active"
limit: 100
headers: {} # OPTIONAL: Request-specific headers
body: "" # OPTIONAL: Request body for POST/PUT/PATCH methods
# --- Error Handling & Retries ---
errorhandler:
error_status_codes: [400, 401, 403, 500]
condition: "error == true"
message: "error.message"
max_retries: 3
initial_backoff: 1.0
# ====== RESPONSE HANDLING ======
response: # OPTIONAL: Response processing rules
content_type: json # OPTIONAL: json | xml | csv | text
locator: "data.results[*]" # OPTIONAL: JMESPath expression to extract array
# --- Pagination & Polling Handlers ---
handler:
kind: paginator # paginator | state_poller | url_poller | basic | cursor_paginator
page_param: "page"
page_size: 100
increment: 1
throttle: 1
# ====== OUTPUT SCHEMA ======
fields: # OPTIONAL: Explicit output schema definition
- name: customer_id
type: BIGINT
- name: email
type: STRING
Core Components: The 3-Part Setup
Every HTTP connection consists of three primary parts:
1. The Client (Identity & Access)
Defines the API base URL, timeout limits, and authentication credentials.
client:
base_url: https://api.salesforce.com/v1
timeout: 30
auth:
method: basic
values:
username: "company_api_user"
password: "<<secrets.get('SECURE_PASSWORD')>>"
2. The Request (Payload & Path)
Defines the HTTP verb, target endpoint, URL query filters, or payload body.
request:
method: get
endpoint: "/customers"
query:
status: "active"
created_after: "2024-01-01"
3. The Response (Parsing & Extraction)
Defines how Data Automation parses incoming HTTP payloads using JMESPath locators.
response:
content_type: json
locator: "results.customers[*]"
Authentication Methods
# Method 1: Basic Auth
auth:
method: basic
values:
username: "user"
password: "<<secrets.get('DB_PASS')>>"
# Method 2: Header-based Auth (API Keys)
auth:
method: header
values:
X-API-Key: "<<secrets.get('API_KEY')>>"
# Method 3: Bearer Token
auth:
method: bearer
values:
token: "<<secrets.get('OAUTH_TOKEN')>>"
Working Examples
Example 1: Paginated GET Request (CRM Source)
- name: fetch_customers
kind: source
connection:
kind: Rest
client:
base_url: https://api.crm.com/v2
auth:
method: bearer
values:
token: "<<secrets.get('CRM_TOKEN')>>"
request:
method: get
endpoint: "/customers"
query:
status: "active"
response:
content_type: json
locator: "data"
handler:
kind: paginator
page_param: "page"
page_size: 100
increment: 1
throttle: 1
Example 2: POST Payload (ERP Sink)
- name: send_orders
kind: sink
connection:
kind: Rest
client:
base_url: https://api.erp.com
auth:
method: basic
values:
username: "<<secrets.get('ERP_USER')>>"
password: "<<secrets.get('ERP_PASS')>>"
request:
method: post
endpoint: "/orders/batch"
content_type: "application/json"
body: >
@json{
"batch_id": "<<batch_id>>",
"orders": <<orders_data>>
}
response:
content_type: json
locator: "result.ids"
Response Parsing with JMESPath
Use JMESPath in the locator field to target specific nested JSON structures:
- Extract target array:
locator: "results.customers" - Filter records:
locator: "data[?active==\true]" - Select nested attributes:
locator: "orders[*].items[?quantity > 10]"
Dynamic Variables & Date Templates
Pass variables directly into request parameters or endpoints:
request:
endpoint: "/sales/<<API.dt().format('MM')>>"
query:
start_date: "<<API.dt().shift(days=-1).format('YYYY-MM-DD')>>"
end_date: "<<API.dt().format('YYYY-MM-DD')>>"
Troubleshooting Reference
| HTTP Error / Issue | Common Cause | Recommended Solution |
|---|---|---|
401 Unauthorized |
Invalid or expired token | Verify auth method and secret variables. |
404 Not Found |
Incorrect path | Double-check base URL and endpoint string. |
| High latency / timeouts | Unpaginated large payload | Add paginator handler and limit page sizes. |
| Empty output | Misconfigured locator | Set show: 10 and inspect raw JSON with locator: "@". |
Debugging Setup
To debug payload schemas during testing, enable temporary logging parameters:
- name: debug_api_source
kind: source
log_level: DEBUG
show: 10
show_schema: true
connection:
kind: Rest
client:
base_url: https://api.example.com
request:
method: get
endpoint: "/data"