GCP Troubleshooting
Quick checks for common deployment issues. Gateway-specific guidance: bh-gateway and Gateway API Deployment.
External Secrets
kubectl describe secretstore gcp-secret-store -n bh-control-plane
kubectl get externalsecret -n bh-control-plane
kubectl logs -n external-secrets-system -l app.kubernetes.io/name=external-secrets
| Issue | Fix |
|---|---|
| WI auth failure | Verify Terraform wif_service_accounts applied; K8s SA annotation |
| Secret not found | GSM name must match global.remoteSecrets |
| Stale secret | ESO refreshInterval; delete K8s secret to force resync |
Cloud SQL
Full bootstrap and schema permission guide: Database Bootstrap Troubleshooting.
| Symptom | Cause | Fix |
|---|---|---|
password authentication failed for bh_app_user | bh-app-user-password ≠ Cloud SQL bh_app_user password | Align GSM secret with Cloud SQL user |
permission denied for schema / Alembic fails | Bootstrap not run, wrong schema owner, or wrong POSTGRES_USER | Run bighammer_db_setup.sql before deploy; use bh_dev_user + bh-dev-user-password — Database Bootstrap Troubleshooting |
password authentication failed for bh_dev_user | POSTGRES_PASSWORD mapped to wrong GSM secret | Set global.remoteSecrets.postgres.password → bh-dev-user-password |
| Keycloak DB auth failure | KC_DB_URL / KC_DB_USERNAME mismatch or wrong bh-kcdbpwd | Database keycloak_db_<env>, user keycloak_db_user, GSM bh-kcdbpwd |
| SSL verify / IP mismatch | Private IP + cert verify | Use postgresSSLMode: require |
Keycloak Invalid lenByte | DER key corrupted in GSM | Re-upload binary DER for db-client-private-key-der |
Test connectivity:
Use your environment connectivity test script (for example a `psql` or Cloud SQL Auth Proxy smoke test).
Ingress / load balancer
| Symptom | Fix |
|---|---|
| LB 0% healthy | Host VPC firewall TCP 10256 (and 80/443) to node SA |
| No EXTERNAL-IP | Wait for GKE LB provisioning; check Service events |
| 502 from nginx | Check backend pods; Keycloak logs for DB/TLS errors |
kubectl describe ingress bh-ingress -n bh-control-plane
kubectl get svc -n ingress-nginx
Gateway API
| Symptom | Fix |
|---|---|
| Gateway not Programmed | Envoy controller running; check kubectl describe gateway bh-gateway |
| HTTPRoute not attached | Add namespace to referenceGrants.routeNamespaces in bh-gateway values |
| Wrong health check port | Use externalTrafficPolicy: Cluster; port 10256 |
kubectl get gateway,httproute -A
kubectl get pods -n envoy-gateway-system
TLS
| Symptom | Fix |
|---|---|
| cert-manager challenge pending | DNS admin on cert-manager SA; Cloud DNS zone correct — Cert troubleshooting |
| Ingress TLS missing | K8s secret global.ingress.tlsSecretName exists with tls.crt / tls.key |
| Gateway HTTPS error | bh-gateway: gateway.listeners.https.tlsSecretName + tlsSecretNamespace — TLS Options |
| Private cert not applied | Upload to GSM keys in global.remoteSecrets.tls; recreate K8s TLS secret |
| PushSecret / GSM sync | Cert troubleshooting — PushSecret |
Images / registry
| Symptom | Fix |
|---|---|
| ImagePullBackOff | Tag exists in GAR/ECR; node SA has artifactregistry.reader |
| Wrong image | Check global.image.repository and tag in values |
Helm
helm dependency build .
helm lint . -f values.yaml -f values-dev.yaml
| Issue | Fix |
|---|---|
| Nil pointer in templates | Enable subchart in values (global.<svc>.enabled: true) |
| CRD conflicts (gateway) | Install gateway-crds once; use --skip-crds on bighammer |
Support escalation
Include: chart version, values overlay (redacted), kubectl get pods,externalsecret,ingress,httproute -n bh-control-plane, and relevant pod logs.
Related
| Topic | Link |
|---|---|
| Database bootstrap | Database Bootstrap Troubleshooting |
| Secret mapping | Secret Manager Secrets |
| Azure equivalent | Azure Troubleshooting |