Skip to main content

Airflow Azure Deployment

This guide covers deploying Apache Airflow on Azure Kubernetes Service.

Overview​

The Airflow chart deploys Apache Airflow 3.x with Azure-specific configuration: Workload Identity, External Secrets for TLS and connections, and optional Azure Blob Storage for DAGs.

Prerequisites​

  • Platform Azure deployed (or standalone: ESO, Key Vault, ingress)
  • TLS certificate in Key Vault (wildcard-dev-az-bighammer-crt, wildcard-dev-az-bighammer-key) — or bootstrap via script (see TLS Bootstrap)
  • Namespace airflow (or configure in values)

TLS Bootstrap​

If Key Vault does not yet have the wildcard TLS cert, bootstrap it from the platform namespace by exporting the TLS secret and pushing it to Key Vault (wildcard-dev-az-bighammer-crt, wildcard-dev-az-bighammer-key) using kubectl and az keyvault secret set.

Deployment​

# Navigate to the Airflow chart directory
# Update dependencies
helm dependency update

# Deploy
helm upgrade --install airflow . \
  -f values-dev.yaml \
  -n airflow \
  --create-namespace \
  --wait

Configuration​

Key values in values-dev.yaml:

SectionPurpose
airflow.ingressHost, TLS secret, ingress class
airflow.envAZURE_TENANT_ID, AZURE_FEDERATED_TOKEN_FILE, Celery timeouts
airflow.volumesProjected service account token for Workload Identity
airflow.dagsGit sync or Blob CSI for DAGs
global.clusterSecretStoreClusterSecretStore for ExternalSecrets

External Secrets​

The chart creates ExternalSecrets for:

  • Database password (if using external PostgreSQL)
  • TLS certificate (wildcard)
  • Azure Blob connection (if using Blob mounts)
  • SSO OAuth (if configured)

DAG Sources​

  • Git sync: Configure airflow.dags.gitSync with repo URL and credentials
  • Azure Blob: Use values-dev-blob-mounts.yaml pattern with Blob CSI mounts

Post-Deployment​

# Check pods
kubectl get pods -n airflow

# Check ExternalSecrets
kubectl get externalsecret -n airflow

# Airflow UI (default host in values-dev)
# https://airflow.<env>.az.<domain>

References​