Skip to main content

Airflow Azure Deployment

This guide covers deploying Apache Airflow on Azure Kubernetes Service.

Overview​

The Airflow chart deploys Apache Airflow 3.x with Azure-specific configuration: Workload Identity, External Secrets for TLS and connections, and optional Azure Blob Storage for DAGs.

Prerequisites​

  • Platform Deployment completed (or standalone: ESO, Key Vault, ingress)
  • TLS certificate in Key Vault (<env>-az-bighammer-tls-crt, <env>-az-bighammer-tls-key) — or bootstrap via script (see TLS Bootstrap)
  • Namespace airflow (or configure in values)

TLS Bootstrap​

If Key Vault does not yet have the wildcard TLS cert, bootstrap it from the platform namespace by exporting the TLS secret and pushing it to Key Vault (<env>-az-bighammer-tls-crt, <env>-az-bighammer-tls-key) using kubectl and az keyvault secret set.

Deployment​

# Navigate to the Airflow chart directory
# Update dependencies
helm dependency update

# Deploy
helm upgrade --install airflow . \
  -f values-dev.yaml \
  -n airflow \
  --create-namespace \
  --wait

Configuration​

Key values in values-dev.yaml:

SectionPurpose
airflow.ingressHost, TLS secret, ingress class
airflow.envAZURE_TENANT_ID, AZURE_FEDERATED_TOKEN_FILE, Celery timeouts
airflow.volumesProjected service account token for Workload Identity
airflow.dagsGit sync or Blob CSI for DAGs
global.clusterSecretStoreClusterSecretStore for ExternalSecrets

External Secrets​

The chart creates ExternalSecrets for:

  • Database password (if using external PostgreSQL)
  • TLS certificate (wildcard)
  • Azure Blob connection (if using Blob mounts)
  • SSO OAuth (if configured)

DAG Sources​

  • Git sync: Configure airflow.dags.gitSync with repo URL and credentials
  • Azure Blob: Use values-dev-blob-mounts.yaml pattern with Blob CSI mounts

Post-Deployment​

# Check pods
kubectl get pods -n airflow

# Check ExternalSecrets
kubectl get externalsecret -n airflow

# Airflow UI (default host in values-dev)
# https://airflow.<env>.az.<domain>

Relationship to platform chart​

Deploy the platform umbrella chart first (or at minimum: ESO, Key Vault secrets, ingress, TLS). Airflow reuses:

  • Wildcard TLS secrets from Key Vault (via ClusterSecretStore or copied secret)
  • PostgreSQL (airflow database — create separately or extend create-dbs-azure.sh)
  • The same Workload Identity and Key Vault patterns

Use copy-tls-from-bighammer.sh or a shared ClusterSecretStore to avoid duplicating certificate sync logic.

References​