Documentation

HTTP Connection

← Back to Documentation Hub / Data Automation

Overview

An HTTP Connection lets your data pipelines exchange data directly with web applications and REST APIs. Data Automation uses HTTP connections to:

  • Fetch data from external services (Salesforce, Shopify, CRM, or marketing tools).
  • Push processed results to third-party endpoints or internal web applications.
  • Automate recurring data transfers across business applications.

Complete HTTP Connection Reference Schema

connection:
  kind: HTTP # REQUIRED: Must be "HTTP" or "Rest"

  # ====== CLIENT CONFIGURATION (API Identity & Connection) ======
  client: # REQUIRED: API client settings
    base_url: "[https://api.example.com/v1](https://api.example.com/v1)" # REQUIRED: Base API URL
    timeout: 30 # OPTIONAL: Timeout in seconds (default 5, 0 = no timeout)

    # --- Authentication ---
    auth: # CONDITIONAL: Required if API needs authentication
      method: basic # REQUIRED: basic | digest | header | oauth2 | bearer_token | api_key
      values: # REQUIRED: Key-value pairs for chosen auth method
        username: "user"
        password: "<<secrets.get('API_PASS')>>"

    # --- Advanced Client Settings ---
    headers: {} # OPTIONAL: Additional HTTP headers for all requests
    proxies: {} # OPTIONAL: Proxy configuration if needed
    verify_ssl: true # OPTIONAL: Verify SSL certificates (default true)

  # ====== REQUEST CONFIGURATION ======
  request: # REQUIRED: Request definition
    method: get # REQUIRED: get | post | put | patch | delete
    endpoint: "/customers" # REQUIRED: API endpoint path
    content_type: "application/json" # REQUIRED: Content type header

    # --- Request Parameters ---
    query: # OPTIONAL: URL query parameters
      status: "active"
      limit: 100
    headers: {} # OPTIONAL: Request-specific headers
    body: "" # OPTIONAL: Request body for POST/PUT/PATCH methods

    # --- Error Handling & Retries ---
    errorhandler:
      error_status_codes: [400, 401, 403, 500]
      condition: "error == true"
      message: "error.message"
    max_retries: 3
    initial_backoff: 1.0

  # ====== RESPONSE HANDLING ======
  response: # OPTIONAL: Response processing rules
    content_type: json # OPTIONAL: json | xml | csv | text
    locator: "data.results[*]" # OPTIONAL: JMESPath expression to extract array

    # --- Pagination & Polling Handlers ---
    handler:
      kind: paginator # paginator | state_poller | url_poller | basic | cursor_paginator
      page_param: "page"
      page_size: 100
      increment: 1
      throttle: 1

  # ====== OUTPUT SCHEMA ======
  fields: # OPTIONAL: Explicit output schema definition
    - name: customer_id
      type: BIGINT
    - name: email
      type: STRING

Core Components: The 3-Part Setup

Every HTTP connection consists of three primary parts:

1. The Client (Identity & Access)

Defines the API base URL, timeout limits, and authentication credentials.

client:
  base_url: https://api.salesforce.com/v1
  timeout: 30
  auth:
    method: basic
    values:
      username: "company_api_user"
      password: "<<secrets.get('SECURE_PASSWORD')>>"

2. The Request (Payload & Path)

Defines the HTTP verb, target endpoint, URL query filters, or payload body.

request:
  method: get
  endpoint: "/customers"
  query:
    status: "active"
    created_after: "2024-01-01"

3. The Response (Parsing & Extraction)

Defines how Data Automation parses incoming HTTP payloads using JMESPath locators.

response:
  content_type: json
  locator: "results.customers[*]"

Authentication Methods

# Method 1: Basic Auth
auth:
  method: basic
  values:
    username: "user"
    password: "<<secrets.get('DB_PASS')>>"

# Method 2: Header-based Auth (API Keys)
auth:
  method: header
  values:
    X-API-Key: "<<secrets.get('API_KEY')>>"

# Method 3: Bearer Token
auth:
  method: bearer
  values:
    token: "<<secrets.get('OAUTH_TOKEN')>>"

Working Examples

Example 1: Paginated GET Request (CRM Source)

- name: fetch_customers
  kind: source
  connection:
    kind: Rest
    client:
      base_url: https://api.crm.com/v2
      auth:
        method: bearer
        values:
          token: "<<secrets.get('CRM_TOKEN')>>"
    request:
      method: get
      endpoint: "/customers"
      query:
        status: "active"
    response:
      content_type: json
      locator: "data"
      handler:
        kind: paginator
        page_param: "page"
        page_size: 100
        increment: 1
        throttle: 1

Example 2: POST Payload (ERP Sink)

- name: send_orders
  kind: sink
  connection:
    kind: Rest
    client:
      base_url: https://api.erp.com
      auth:
        method: basic
        values:
          username: "<<secrets.get('ERP_USER')>>"
          password: "<<secrets.get('ERP_PASS')>>"
    request:
      method: post
      endpoint: "/orders/batch"
      content_type: "application/json"
      body: >
        @json{
          "batch_id": "<<batch_id>>",
          "orders": <<orders_data>>
        }
    response:
      content_type: json
      locator: "result.ids"

Response Parsing with JMESPath

Use JMESPath in the locator field to target specific nested JSON structures:

  • Extract target array: locator: "results.customers"
  • Filter records: locator: "data[?active==\true]"
  • Select nested attributes: locator: "orders[*].items[?quantity > 10]"

Dynamic Variables & Date Templates

Pass variables directly into request parameters or endpoints:

request:
  endpoint: "/sales/<<API.dt().format('MM')>>"
  query:
    start_date: "<<API.dt().shift(days=-1).format('YYYY-MM-DD')>>"
    end_date: "<<API.dt().format('YYYY-MM-DD')>>"

Troubleshooting Reference

HTTP Error / Issue Common Cause Recommended Solution
401 Unauthorized Invalid or expired token Verify auth method and secret variables.
404 Not Found Incorrect path Double-check base URL and endpoint string.
High latency / timeouts Unpaginated large payload Add paginator handler and limit page sizes.
Empty output Misconfigured locator Set show: 10 and inspect raw JSON with locator: "@".

Debugging Setup

To debug payload schemas during testing, enable temporary logging parameters:

- name: debug_api_source
  kind: source
  log_level: DEBUG
  show: 10
  show_schema: true
  connection:
    kind: Rest
    client:
      base_url: https://api.example.com
    request:
      method: get
      endpoint: "/data"