SSL/TLS encryption
BigHammer uses SSL/TLS (HTTPS) at the cluster edge to encrypt traffic between clients and the platform. TLS terminates on the edge proxy — either nginx Ingress or Envoy Gateway — not on the GCP external load balancer (L4 passthrough).
This page covers edge HTTPS certificates. Cloud SQL client SSL is separate — see postgresSSLMode in Platform Deployment and Secret Manager Secrets.
Certificate lifecycle (Option A — Let's Encrypt)
- cert-manager proves domain control via DNS-01 (TXT records in Cloud DNS).
- Let's Encrypt issues a certificate; cert-manager stores it in Kubernetes secret
<env>-gcp-bighammer-tls. - PushSecret (External Secrets Operator) copies
tls.crt/tls.keyto Secret Manager. - Edge (Ingress or Gateway) mounts the Kubernetes TLS secret for HTTPS.
- Application pods (for example Keycloak) read PEM material from GSM via
global.remoteSecrets.tlsand ExternalSecrets.
How TLS fits the two edge modes
The same Kubernetes TLS secret can serve both modes. What changes is which chart reads the secret name:
| Edge mode | Chart | Values path | Doc |
|---|---|---|---|
| NGINX Ingress | bighammer | global.ingress.tlsSecretName | NGINX Ingress Deployment |
| Gateway API | bh-gateway | gateway.listeners.https.tlsSecretName + tlsSecretNamespace | bh-gateway — TLS |
global.gateway.* in the bighammer chart controls HTTPRoutes only — not TLS. Configure the HTTPS listener on the bh-gateway release (install).
Where certificates are stored
| Store | Name | Keys / format | Written by | Read by |
|---|---|---|---|---|
| Kubernetes Secret | <env>-gcp-bighammer-tls in bh-control-plane | tls.crt, tls.key (kubernetes.io/tls) | cert-manager (Certificate) or kubectl create secret tls | nginx Ingress, bh-gateway HTTPS listener |
| Secret Manager | <env>-gcp-bighammer-tls-crt | PEM certificate | PushSecret (Option A) or gcloud (Option B) | bighammer ExternalSecrets → Keycloak and other consumers |
| Secret Manager | <env>-gcp-bighammer-tls-key | PEM private key | PushSecret (Option A) or gcloud (Option B) | Same |
Default GSM secret names are configured in bighammer values:
global:
remoteSecrets:
tls:
cert: <env>-gcp-bighammer-tls-crt
key: <env>-gcp-bighammer-tls-key
How the bighammer chart references TLS
| Consumer | Mechanism | Values / template |
|---|---|---|
| nginx Ingress | spec.tls.secretName on bh-ingress | global.ingress.tlsSecretName → charts/bhui/templates/ingress.yaml |
| Gateway HTTPS | certificateRefs on bh-gateway listener | bh-gateway values: gateway.listeners.https.tlsSecretName + tlsSecretNamespace |
| Keycloak | ExternalSecret pulls GSM PEM into pod secret | global.remoteSecrets.tls.cert / .key → charts/bhkeycloak/templates/secret.yaml |
Ingress mode
global:
ingress:
enabled: true
tlsSecretName: <env>-gcp-bighammer-tls
gateway:
enabled: false
All hostnames listed under global.ingress.*HostName use this TLS secret on the single bh-ingress resource.
Gateway mode
# bh-gateway release (envoy-gateway-system)
gateway:
listeners:
https:
tlsSecretName: <env>-gcp-bighammer-tls
tlsSecretNamespace: bh-control-plane
# bighammer release
global:
ingress:
enabled: false
gateway:
enabled: true
Keycloak (GSM path)
Keycloak does not read the Ingress TLS secret directly. It syncs certificate and key from Secret Manager using the same remote keys PushSecret populates:
# ExternalSecret data keys (from chart template)
KC_HTTPS_CERTIFICATE_FILE → GSM: global.remoteSecrets.tls.cert
KC_HTTPS_CERTIFICATE_KEY_FILE → GSM: global.remoteSecrets.tls.key
Therefore Option A requires both the Kubernetes TLS secret (edge) and GSM copies (apps).
Option A — Let's Encrypt (cert-manager + DNS-01)
Best for: Dev / non-prod with a public Cloud DNS zone.
Sample manifests in the Helm chart
The bighammer chart ships reference manifests under cert-manager/ (applied with kubectl, not via helm install):
| File | Kubernetes resource |
|---|---|
cert-manager/cluster-issuer.yaml | Issuer — Let's Encrypt + Cloud DNS |
cert-manager/wildcard-cert.yaml | Certificate — wildcard request |
cert-manager/push-secret.yaml | PushSecret — sync to Secret Manager |
Obtain the files from your chart package:
# After helm pull of bighammer OCI chart
tar -xzf bighammer-<version>.tgz
ls bighammer/cert-manager/
Prerequisites
Complete before applying manifests:
| # | Requirement | Doc |
|---|---|---|
| 1 | cert-manager cert-manager-v1.19.2 in cluster | Prerequisites — add-ons |
| 2 | External Secrets Operator | External Secrets Operator |
| 3 | Workload Identity — ESO + cert-manager DNS access | Workload Identity |
| 4 | Cloud DNS public zone for your domain | Infrastructure |
| 5 | cert-manager controller SA with roles/dns.admin | DNS-01 challenge (GCP SA bound to cert-manager/cert-manager KSA) |
| 6 | SecretStore/gcp-secret-store in bh-control-plane | Created by bighammer chart or platform bootstrap |
| 7 | PushSecret-capable ESO version | Reference: ESO 0.18.x (PushSecret is v1alpha1) |
Install bighammer once (or apply equivalent bootstrap) so SecretStore/gcp-secret-store exists before push-secret.yaml.
Provision — apply order
kubectl apply -f cluster-issuer.yaml
kubectl apply -f wildcard-cert.yaml
# Wait until Certificate is Ready and secret <env>-gcp-bighammer-tls exists
kubectl apply -f push-secret.yaml
Manifest 1 — cluster-issuer.yaml
Defines a namespace-scoped Issuer that uses Let's Encrypt production ACME with DNS-01 via Cloud DNS. cert-manager creates temporary TXT records to prove you control the zone.
Customize before apply: email, spec.acme.solvers[].dns01.cloudDNS.project (GCP project hosting the DNS zone).
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-prod
namespace: bh-control-plane
spec:
acme:
# Let's Encrypt production directory (use staging for tests)
server: https://acme-v02.api.letsencrypt.org/directory
email: admin@example.com # ← your operations contact
privateKeySecretRef:
name: letsencrypt-prod-account-key # ACME account key (auto-created)
solvers:
- dns01:
cloudDNS:
project: <PROJECT_ID> # ← GCP project with Cloud DNS zone
# Workload Identity: omit serviceAccountSecretRef.
# Bind cert-manager/cert-manager KSA → GCP SA with roles/dns.admin
| Field | Purpose |
|---|---|
kind: Issuer | Namespace-scoped; lives in bh-control-plane with the Certificate |
server | ACME directory URL (staging URL for testing) |
email | Let's Encrypt account / expiry notifications |
privateKeySecretRef | Stores ACME account private key in the cluster |
dns01.cloudDNS.project | Project where Cloud DNS zone exists |
| WI note | On GKE, cert-manager uses Workload Identity — no JSON key secret |
Manifest 2 — wildcard-cert.yaml
Requests a wildcard certificate from the Issuer above. cert-manager writes the issued cert and key into the Kubernetes TLS secret named secretName.
Customize: dnsNames to match your zone (must be covered by the Cloud DNS zone).
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: <env>-gcp-bighammer-wildcard
namespace: bh-control-plane
spec:
secretName: <env>-gcp-bighammer-tls # ← K8s TLS secret Ingress/Gateway use
issuerRef:
name: letsencrypt-prod
kind: Issuer
dnsNames:
- "*.<env>.<your-domain>" # ← wildcard for app hostnames
- "<env>.<your-domain>" # ← apex (optional, if needed)
| Field | Purpose |
|---|---|
metadata.name | Certificate resource name (not the secret name) |
spec.secretName | Output — must match global.ingress.tlsSecretName and bh-gateway tlsSecretName |
issuerRef | Links to Issuer/letsencrypt-prod |
dnsNames | SANs on the certificate; align with global.ingress.*HostName / gateway hostnames |
After apply, wait for readiness:
kubectl get certificate <env>-gcp-bighammer-wildcard -n bh-control-plane
kubectl get secret <env>-gcp-bighammer-tls -n bh-control-plane
Expected: Certificate status Ready=True; secret type kubernetes.io/tls with tls.crt and tls.key.
Manifest 3 — push-secret.yaml
Uses External Secrets Operator PushSecret to copy the Kubernetes TLS secret into GCP Secret Manager so bighammer ExternalSecrets (Keycloak, etc.) can read PEM files.
Requires: SecretStore/gcp-secret-store in bh-control-plane (from bighammer chart).
# PushSecret is v1alpha1 on ESO 0.18.x
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
name: push-wildcard-cert
namespace: bh-control-plane
spec:
refreshInterval: 12h
updatePolicy: Replace
secretStoreRefs:
- name: gcp-secret-store
kind: SecretStore
selector:
secret:
name: <env>-gcp-bighammer-tls # ← source K8s secret from Certificate
data:
- match:
secretKey: tls.crt
remoteRef:
remoteKey: <env>-gcp-bighammer-tls-crt # ← GSM name (global.remoteSecrets.tls.cert)
- match:
secretKey: tls.key
remoteRef:
remoteKey: <env>-gcp-bighammer-tls-key # ← GSM name (global.remoteSecrets.tls.key)
| Field | Purpose |
|---|---|
secretStoreRefs | Must match global.secretStore.name (gcp-secret-store) |
selector.secret.name | Source — cert-manager TLS secret |
data[].match.secretKey | Standard TLS secret keys |
remoteRef.remoteKey | Target GSM secret names — must match global.remoteSecrets.tls in values |
refreshInterval | How often ESO checks for changes (renewal updates GSM when K8s secret rotates) |
updatePolicy: Replace | Overwrite GSM secret versions when cert renews |
Verify:
kubectl get pushsecret push-wildcard-cert -n bh-control-plane
gcloud secrets versions access latest --secret=<env>-gcp-bighammer-tls-crt --project=<PROJECT_ID>
Wire up bighammer values
After all three manifests succeed:
global:
remoteSecrets:
tls:
cert: <env>-gcp-bighammer-tls-crt
key: <env>-gcp-bighammer-tls-key
ingress:
enabled: true # or gateway via bh-gateway
tlsSecretName: <env>-gcp-bighammer-tls
Deploy or upgrade bighammer and confirm edge + Keycloak:
helm upgrade --install bighammer <chart> -f values.yaml -f values-<env>.yaml -n bh-control-plane
Renewal
Automatic — cert-manager renews ~30 days before expiry and updates <env>-gcp-bighammer-tls. PushSecret propagates new PEM to GSM on its refreshInterval. No Helm value changes needed if secret names stay the same.
Option B — Private SSL/TLS certificate
Best for: Production, corporate CA, or environments without public DNS-01.
Steps
- Obtain wildcard or SAN certificate from your CA (PEM cert + key).
- Store in Secret Manager (names must match
global.remoteSecrets.tls):
gcloud secrets create <env>-gcp-bighammer-tls-crt \
--project=<PROJECT_ID> \
--replication-policy=automatic \
--data-file=./wildcard.crt
gcloud secrets create <env>-gcp-bighammer-tls-key \
--project=<PROJECT_ID> \
--replication-policy=automatic \
--data-file=./wildcard.key
- Create the Kubernetes TLS secret in
bh-control-plane:
kubectl create secret tls <env>-gcp-bighammer-tls \
--cert=wildcard.crt \
--key=wildcard.key \
-n bh-control-plane
- Point the active edge mode at that secret:
| Mode | Configure |
|---|---|
| Ingress | global.ingress.tlsSecretName: <env>-gcp-bighammer-tls |
| Gateway | bh-gateway → gateway.listeners.https.tlsSecretName + tlsSecretNamespace |
- Re-run the relevant Helm upgrade (
bighammerand/orbh-gateway).
Option B skips cert-manager manifests; you maintain renewal and GSM uploads manually.
Keycloak DB private key (DER)
Keycloak requires the DB client key in DER format in Secret Manager (db-client-private-key-der):
openssl pkcs8 -topk8 -inform PEM -outform DER -in client-key.pem -out client-key.der -nocrypt
gcloud secrets versions add db-client-private-key-der --data-file=client-key.der --project=<PROJECT_ID>
Comparison
| Option A (Let's Encrypt) | Option B (Private cert) | |
|---|---|---|
| Cost | Free | CA-dependent |
| Automation | Full (cert-manager + PushSecret) | Manual renewal |
| DNS | Public Cloud DNS + DNS-01 | Any DNS |
| K8s secret | cert-manager creates <env>-gcp-bighammer-tls | You create with kubectl |
| Secret Manager | PushSecret populates crt/key secrets | You upload with gcloud |
| Manifests | 3 files in chart cert-manager/ | None |
| Prod fit | Dev / staging | Recommended for prod |
Verify TLS
# Kubernetes TLS secret
kubectl get secret <env>-gcp-bighammer-tls -n bh-control-plane
# cert-manager (Option A)
kubectl get issuer,certificate,pushsecret -n bh-control-plane
# GSM (Option A or B)
gcloud secrets list --filter="name:<env>-gcp-bighammer-tls" --project=<PROJECT_ID>
# Ingress mode
kubectl get ingress -n bh-control-plane -o yaml | grep -A2 tls
# Gateway mode
kubectl describe gateway bh-gateway -n envoy-gateway-system
# External
curl -vI https://<your-hostname>
For detailed diagnosis, see Cert troubleshooting below.
DNS records
TLS certificates prove hostname identity; clients still need DNS A records pointing to your load balancer:
| Mode | Get LB IP |
|---|---|
| NGINX Ingress | kubectl get svc -n ingress-nginx |
| Gateway API | Data-plane Service in envoy-gateway-system — bh-gateway |
Hostnames must match certificate SANs (dnsNames in wildcard-cert.yaml or your private cert).
Cert troubleshooting
Use this section when HTTPS fails, certificates stay Pending, PushSecret does not sync, or browsers report certificate errors. Work top to bottom — edge issues often look like app issues when the K8s TLS secret is missing or misnamed.
Quick diagnosis
# 1. Issuance (Option A)
kubectl describe certificate <env>-gcp-bighammer-wildcard -n bh-control-plane
kubectl describe issuer letsencrypt-prod -n bh-control-plane
kubectl get challenges -n bh-control-plane
# 2. Kubernetes TLS secret (both options)
kubectl get secret <env>-gcp-bighammer-tls -n bh-control-plane -o jsonpath='{.type}{"\n"}{.data.tls\.crt}{"\n"}' | head -2
# 3. PushSecret → GSM (Option A)
kubectl describe pushsecret push-wildcard-cert -n bh-control-plane
gcloud secrets versions list <env>-gcp-bighammer-tls-crt --project=<PROJECT_ID> --limit=3
# 4. Edge reference
kubectl get ingress bh-ingress -n bh-control-plane -o yaml | grep -A5 'tls:'
kubectl describe gateway bh-gateway -n envoy-gateway-system | grep -A10 'Listeners:'
# 5. External check
openssl s_client -connect <hostname>:443 -servername <hostname> </dev/null 2>/dev/null | openssl x509 -noout -subject -dates -ext subjectAltName
| Check | Healthy signal |
|---|---|
Certificate | Ready=True |
| K8s secret | Type kubernetes.io/tls; keys tls.crt, tls.key |
PushSecret | Status Synced / Ready |
| GSM | Latest version exists for crt and key secrets |
| Ingress | spec.tls[].secretName = global.ingress.tlsSecretName |
| Gateway | HTTPS listener certificateRefs point to secret in tlsSecretNamespace |
Browser / openssl | Subject/SAN includes your hostname; dates valid |
cert-manager — certificate not Ready
| Symptom | Likely cause | What to do |
|---|---|---|
Certificate Pending indefinitely | Issuer not ready or ACME account issue | kubectl describe issuer letsencrypt-prod -n bh-control-plane; confirm Ready=True and ACME account key secret exists |
Challenge stuck pending | DNS-01 cannot create TXT records | See DNS-01 and Cloud DNS |
Challenge invalid | Wrong zone, rate limit, or bad dnsNames | kubectl describe challenge -n bh-control-plane; fix dnsNames in wildcard-cert.yaml to match your Cloud DNS zone |
Certificate Ready but no secret | Wrong secretName or namespace mismatch | Certificate and secret must both be in bh-control-plane |
| Let's Encrypt rate limit error | Too many failed or duplicate requests | Use staging ACME server in cluster-issuer.yaml for testing; wait for limit reset in prod |
kubectl get certificate,order,challenge -n bh-control-plane
kubectl logs -n cert-manager deploy/cert-manager --tail=100
DNS-01 and Cloud DNS
| Symptom | Likely cause | What to do |
|---|---|---|
| TXT record never appears | cert-manager lacks roles/dns.admin | Bind cert-manager KSA (cert-manager/cert-manager) to a GCP SA with roles/dns.admin on the DNS project — Workload Identity |
| TXT in wrong project | cloudDNS.project mismatch | Set spec.acme.solvers[].dns01.cloudDNS.project in cluster-issuer.yaml to the project that hosts the DNS zone |
| Zone not found | Domain not in Cloud DNS | Create or delegate a public managed zone; apex/wildcard names must live under that zone |
| WI still using JSON key path | serviceAccountSecretRef set incorrectly | On GKE, omit serviceAccountSecretRef under cloudDNS; use Workload Identity only |
| Challenge succeeds then fails on renew | IAM or zone changed | Re-verify cert-manager SA binding; check Cloud DNS audit logs |
Verify TXT records during an active challenge:
gcloud dns record-sets list --zone=<ZONE_NAME> --project=<PROJECT_ID> --filter="_acme-challenge"
PushSecret and Secret Manager
| Symptom | Likely cause | What to do |
|---|---|---|
PushSecret not found | Applied before ESO or wrong namespace | Apply push-secret.yaml in bh-control-plane; ESO must be installed |
| PushSecret not ready | SecretStore/gcp-secret-store missing | Install or upgrade bighammer so SecretStore exists — External Secrets Operator |
| Permission denied on push | ESO GCP SA lacks Secret Manager write | Grant roles/secretmanager.admin or secret-level access to the ESO Workload Identity SA |
| GSM secrets empty or stale | Source K8s secret missing or PushSecret not synced | Confirm <env>-gcp-bighammer-tls exists first; kubectl describe pushsecret push-wildcard-cert -n bh-control-plane |
| Keycloak TLS errors but edge HTTPS works | GSM crt/key not updated | PushSecret only runs on interval (refreshInterval: 12h); after renewal, wait or temporarily lower interval; confirm remoteKey names match global.remoteSecrets.tls |
| Wrong GSM secret names | remoteKey ≠ chart values | Align push-secret.yaml remoteKey with global.remoteSecrets.tls.cert / .key |
kubectl get secretstore gcp-secret-store -n bh-control-plane
kubectl get externalsecret -n bh-control-plane | grep -i tls
gcloud secrets describe <env>-gcp-bighammer-tls-crt --project=<PROJECT_ID>
Edge proxy — Ingress or Gateway
| Symptom | Likely cause | What to do |
|---|---|---|
| HTTP works, HTTPS fails | TLS secret missing or wrong name | Create or fix <env>-gcp-bighammer-tls; set global.ingress.tlsSecretName (Ingress) or bh-gateway tlsSecretName + tlsSecretNamespace |
| default backend certificate / wrong cert in browser | Hostname not on cert SAN | Add hostname to dnsNames (Option A) or re-issue private cert (Option B); wildcard must cover subdomain |
| Ingress serves HTTP only | tls block absent on Ingress | Upgrade bighammer with global.ingress.tlsSecretName set; kubectl get ingress bh-ingress -n bh-control-plane -o yaml |
| Gateway listener not programmed | Secret in wrong namespace | Gateway cannot read secrets across namespaces without reference grant; set tlsSecretNamespace: bh-control-plane on bh-gateway |
| 502 / connection reset on 443 | Edge up but backends unhealthy | Separate from TLS — check pod readiness; see Troubleshooting |
Ingress mode
kubectl get ingress bh-ingress -n bh-control-plane -o jsonpath='{.spec.tls}'
helm get values bighammer -n bh-control-plane | grep -A2 tlsSecretName
Gateway mode
kubectl get gateway bh-gateway -n envoy-gateway-system -o yaml | grep -A15 certificateRefs
helm get values bh-gateway -n envoy-gateway-system | grep -A5 tlsSecret
Certificate name mismatch (browser errors)
Browser / curl message | Meaning | Fix |
|---|---|---|
NET::ERR_CERT_COMMON_NAME_INVALID | Hostname not in certificate SAN | Match DNS hostname to dnsNames or cert SANs |
certificate has expired | Past notAfter | Option A: check cert-manager renewal; Option B: upload new PEM to GSM + K8s secret |
unable to get local issuer certificate | Incomplete chain | Ensure tls.crt contains full chain (leaf + intermediates) |
SSL certificate problem: self signed | Wrong secret mounted or staging cert left in place | Confirm secret contents; use production Issuer URL for real traffic |
Compare served cert vs expected secret:
# Cert presented on the wire
echo | openssl s_client -connect <hostname>:443 -servername <hostname> 2>/dev/null | openssl x509 -noout -text | grep -A1 'Subject Alternative Name'
# Cert in cluster secret (decode first line only)
kubectl get secret <env>-gcp-bighammer-tls -n bh-control-plane -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text | grep -A1 'Subject Alternative Name'
Option B — private certificate
| Symptom | Likely cause | What to do |
|---|---|---|
| Edge HTTPS broken after upload to GSM only | K8s TLS secret not created | kubectl create secret tls <env>-gcp-bighammer-tls --cert=... --key=... -n bh-control-plane |
| Keycloak fails, edge OK | GSM names or format wrong | Upload PEM to <env>-gcp-bighammer-tls-crt / -key; names must match global.remoteSecrets.tls |
| Renewed cert not picked up | Old secret version cached | Create new GSM version; restart affected pods or wait for ExternalSecret refresh |
| Mixed old/new material | Cert and key from different issuances | Re-upload matching pair to GSM and recreate K8s secret |
Apply order mistakes
| Mistake | Effect | Recovery |
|---|---|---|
push-secret before wildcard-cert | PushSecret errors (no source secret) | Apply wildcard-cert.yaml; wait for Ready; re-apply or wait for PushSecret reconcile |
wildcard-cert before cluster-issuer | Certificate cannot issue | Apply cluster-issuer.yaml first |
Changed secretName in Certificate only | Ingress/Gateway still point at old name | Update global.ingress.tlsSecretName and bh-gateway listener to match or revert secretName |
| Deleted K8s TLS secret manually | Edge HTTPS breaks until re-issued | Option A: cert-manager recreates on reconcile; Option B: re-run kubectl create secret tls |
Still stuck?
- Collect:
kubectl describe certificate,issuer,pushsecret -n bh-control-plane, cert-manager logs, and ESO controller logs. - Confirm the full chain: Issuer Ready → Certificate Ready → K8s secret → PushSecret Synced → GSM versions → Helm values → edge listener.
- See also Troubleshooting — TLS for platform-wide symptoms.
Related
| Topic | Link |
|---|---|
| GSM secret names | Secret Manager Secrets |
| NGINX edge | NGINX Ingress Deployment |
| Gateway edge | bh-gateway |
| ESO + SecretStore | External Secrets Operator |
| cert-manager add-on | Prerequisites |