Automated Parquet Snapshot Sync via cURL / PowerShell
Modern BI stacks often force mid-sized teams to stand up and pay for an external cloud data warehouse. With Automat BI, you can materialize Parquet snapshots directly from sources into dashboards.
For BI tools and data workflows that read local or mounted Parquet files directly—such as Tableau Desktop/Server, QlikView / Qlik Sense, local DuckDB, or Python scripts—you can automate snapshot downloads using a lightweight shell script. Because Automat BI uses stateful session tokens tracked by Token ID (tid) on the backend, a single curl command is not enough: the script must perform a two-step token validation/refresh handshake and persist the rotated token for subsequent runs.
Step 1: Export a Parquet Snapshot Sink
Configure your scheduled Automat BI pipeline (e.g., running at 04:00 right after overnight ingestion) to output a flat .parquet snapshot:
- name: SaveOrdersSnapshot
kind: sink
connection:
kind: Parquet
locator: orders_snapshot.parquet
Step 2: Linux / macOS Shell Script (bash + curl)
Save this script as /usr/local/bin/sync-automatbi-snapshots.sh (chmod +x), seed /etc/automatbi/token.txt with your initial API token (chmod 600), and schedule it via cron or a systemd timer after your overnight pipeline completes.
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="https://eu.automatbi.com"
TOKEN_FILE="/etc/automatbi/token.txt"
OUT_DIR="/var/data/automatbi"
FILES=("orders_snapshot.parquet")
if [[ ! -s "$TOKEN_FILE" ]]; then
echo "Error: Token file $TOKEN_FILE is missing or empty." >&2
exit 1
fi
INITIAL_TOKEN=$(tr -d '[:space:]' < "$TOKEN_FILE")
# 1. Validate & Refresh Session Token (checks active tid on backend)
HTTP_RESPONSE=$(curl -sS -w "\n%{http_code}" \
-H "accept: text/plain" \
-H "X-Auth-Token: $INITIAL_TOKEN" \
"$BASE_URL/api/v1/auth/validate")
HTTP_STATUS=$(echo "$HTTP_RESPONSE" | tail -n1)
NEW_TOKEN=$(echo "$HTTP_RESPONSE" | sed '$d' | tr -d '[:space:]')
if [[ "$HTTP_STATUS" == "401" ]]; then
echo "Authentication Failed (401): Session token revoked on backend." >&2
exit 1
elif [[ "$HTTP_STATUS" == "403" ]]; then
echo "Access Denied (403): Insufficient permissions for this resource." >&2
exit 1
elif [[ "$HTTP_STATUS" != "200" ]]; then
echo "Unexpected auth status: HTTP $HTTP_STATUS" >&2
exit 1
fi
# Persist rotated token if returned
if [[ -n "$NEW_TOKEN" ]]; then
printf '%s' "$NEW_TOKEN" > "$TOKEN_FILE"
EFFECTIVE_TOKEN="$NEW_TOKEN"
else
EFFECTIVE_TOKEN="$INITIAL_TOKEN"
fi
# 2. Download Parquet Snapshots Atomically
mkdir -p "$OUT_DIR"
for FILE in "${FILES[@]}"; do
TMP_FILE="$OUT_DIR/.${FILE}.tmp"
curl -fsS --max-time 600 \
-H "accept: application/octet-stream" \
-H "X-Auth-Token: $EFFECTIVE_TOKEN" \
"$BASE_URL/api/v1/data/download/data/$FILE" \
-o "$TMP_FILE"
mv -f "$TMP_FILE" "$OUT_DIR/$FILE"
done
Step 3: Windows Script (PowerShell)
For Windows-based Tableau Server or QlikView EXECUTE reload tasks, use the equivalent PowerShell script:
$ErrorActionPreference = "Stop"
$BaseUrl = "https://eu.automatbi.com"
$TokenFile = "C:\AutomatBI\token.txt"
$OutDir = "C:\AutomatBI\data"
$Files = @("orders_snapshot.parquet")
$InitialToken = (Get-Content -Path $TokenFile -Raw).Trim()
# 1. Validate & Refresh Session Token
try {
$AuthResponse = Invoke-WebRequest -Uri "$BaseUrl/api/v1/auth/validate" `
-Headers @{ "accept" = "text/plain"; "X-Auth-Token" = $InitialToken } `
-TimeoutSec 600
} catch {
$StatusCode = [int]$_.Exception.Response.StatusCode
if ($StatusCode -eq 401) { throw "Authentication Failed (401): Session revoked in API." }
if ($StatusCode -eq 403) { throw "Access Denied (403): Insufficient permissions." }
throw
}
$NewToken = $AuthResponse.Content.Trim()
if ($NewToken.Length -gt 0) {
Set-Content -Path $TokenFile -Value $NewToken -NoNewline
$EffectiveToken = $NewToken
} else {
$EffectiveToken = $InitialToken
}
# 2. Download Parquet Snapshots Atomically
New-Item -ItemType Directory -Force -Path $OutDir | Out-Null
foreach ($File in $Files) {
$TmpPath = Join-Path $OutDir "$File.tmp"
$FinalPath = Join-Path $OutDir $File
Invoke-WebRequest -Uri "$BaseUrl/api/v1/data/download/data/$File" `
-Headers @{ "accept" = "application/octet-stream"; "X-Auth-Token" = $EffectiveToken } `
-TimeoutSec 600 `
-OutFile $TmpPath
Move-Item -Path $TmpPath -Destination $FinalPath -Force
}
Operational Best Practices
- Stateful token persistence: Always write
NEW_TOKENback toTOKEN_FILE(chmod 600on Linux or restricted NTFS ACLs on Windows) so each scheduled run refreshes the active session cleanly without manual re-authentication. - Atomic file replacement (
.tmp+mv): Writing to a temporary file first guarantees that downstream tools like Tableau Server or QlikView never read a partially downloaded Parquet file if a refresh triggers mid-transfer. - Push-to-bucket alternative: If your environment prefers cloud-native storage without managing local pull scripts, configure an Automat BI sink to push
.parquetsnapshots directly to your own S3, Azure Blob, or Google Cloud Storage bucket using your own IAM credentials.