Azure Troubleshooting

Quick checks for common deployment issues on AKS. Database bootstrap guide: Database Bootstrap Troubleshooting.

External Secrets / Key Vault

kubectl describe secretstore azure-secret-store -n bh-control-plane
kubectl get externalsecret -n bh-control-plane
kubectl logs -n external-secrets-system -l app.kubernetes.io/name=external-secrets

Issue	Fix
Workload Identity auth failure	Verify federated credentials; K8s SA annotation azure.workload.identity/client-id
Secret not found	Key Vault name must match global.remoteSecrets
Key Vault Forbidden (403)	Grant Key Vault Secrets User to ESO managed identity; wait 60–120s for RBAC propagation
Stale secret	ESO refreshInterval; delete K8s secret to force resync
SecretStore not ready	kubectl describe secretstore — check vaultUrl, tenant ID, managed identity client ID

Restart ESO after RBAC changes:

kubectl rollout restart deployment external-secrets -n external-secrets-system

PostgreSQL

Full bootstrap and schema permission guide: Database Bootstrap Troubleshooting.

Symptom	Cause	Fix
password authentication failed for bh_app_user	bh-app-user-password ≠ PostgreSQL bh_app_user password	Align Key Vault secret with DB user
permission denied for schema / Alembic fails	Bootstrap not run, wrong schema owner, or wrong POSTGRES_USER	Run create-dbs-azure.sh before deploy; use bh_dev_user + bh-dev-user-password
password authentication failed for bh_dev_user	POSTGRES_PASSWORD mapped to wrong Key Vault secret	Set global.remoteSecrets.postgres.password → bh-dev-user-password
Keycloak DB auth failure	KC_DB_URL / KC_DB_USERNAME mismatch or wrong bh-kcdbpwd	Database keycloak_db_<env>, user keycloak_db_user, Key Vault bh-kcdbpwd
Connection refused / timeout	Firewall, private endpoint, or VNet rules	Allow AKS egress to PostgreSQL; verify PGSSLMODE=require
SSL verify errors	Server CA not in Key Vault	Upload bh-postgres-ssl-cert; set postgresSSLMode: require

Ingress / load balancer

Symptom	Fix
LB unhealthy	Set azure-load-balancer-health-probe-request-path: /healthz on ingress-nginx Service
No EXTERNAL-IP	Wait for Azure LB provisioning; check Service events
502 from nginx	Check backend pods; Keycloak logs for DB/TLS errors

kubectl describe ingress bh-ingress -n bh-control-plane
kubectl get svc -n ingress-nginx

TLS

Symptom	Fix
cert-manager challenge pending	DNS Zone Contributor on cert-manager identity; Azure DNS zone correct — TLS Options
Ingress TLS missing	K8s secret global.ingress.tlsSecretName exists with tls.crt / tls.key
Private cert not applied	Upload to Key Vault keys in global.remoteSecrets.tls; recreate K8s TLS secret
PushSecret / Key Vault sync	Verify PushSecret status; Key Vault secret names match global.remoteSecrets.tls

Workload Identity

Symptom	Fix
Federated credential creation fails	Caller needs Managed Identity Contributor on target identity
Wrong identity name	az identity list --output table
Pod cannot access Key Vault	Check SA annotations and federated credential subject

Images / registry

Symptom	Fix
ImagePullBackOff	Tag exists in ACR; kubelet identity has AcrPull
Wrong image	Check global.image.repository and tag in values

Storage

Symptom	Fix
PV attachment timeout	Verify storage class (managed-csi, azurefile-csi); check node pool SKU
Blob CSI mount fails	Enable Blob CSI add-on; verify Workload Identity + storage RBAC

Helm

helm dependency build .
helm lint . -f values.yaml -f values-<env>.yaml

Issue	Fix
Nil pointer in templates	Enable subchart in values (global.<svc>.enabled: true)
ExternalSecret not ready	Complete Key Vault bootstrap before Helm install

Support escalation

Include: chart version, values overlay (redacted), kubectl get pods,externalsecret,ingress -n bh-control-plane, and relevant pod logs.

Related

Topic	Link
Database bootstrap	Database Bootstrap Troubleshooting
Key Vault mapping	Key Vault Secrets
GCP equivalent	GCP Troubleshooting