Skip to main content

SSL/TLS encryption

BigHammer uses SSL/TLS (HTTPS) at the cluster edge to encrypt traffic between clients and the platform. TLS terminates on the edge proxy — either nginx Ingress or Envoy Gateway — not on the GCP external load balancer (L4 passthrough).

This page covers edge HTTPS certificates. Cloud SQL client SSL is separate — see postgresSSLMode in Platform Deployment and Secret Manager Secrets.


Certificate lifecycle (Option A — Let's Encrypt)

  1. cert-manager proves domain control via DNS-01 (TXT records in Cloud DNS).
  2. Let's Encrypt issues a certificate; cert-manager stores it in Kubernetes secret <env>-gcp-bighammer-tls.
  3. PushSecret (External Secrets Operator) copies tls.crt / tls.key to Secret Manager.
  4. Edge (Ingress or Gateway) mounts the Kubernetes TLS secret for HTTPS.
  5. Application pods (for example Keycloak) read PEM material from GSM via global.remoteSecrets.tls and ExternalSecrets.

How TLS fits the two edge modes

The same Kubernetes TLS secret can serve both modes. What changes is which chart reads the secret name:

Edge modeChartValues pathDoc
NGINX Ingressbighammerglobal.ingress.tlsSecretNameNGINX Ingress Deployment
Gateway APIbh-gatewaygateway.listeners.https.tlsSecretName + tlsSecretNamespacebh-gateway — TLS
Gateway mode

global.gateway.* in the bighammer chart controls HTTPRoutes only — not TLS. Configure the HTTPS listener on the bh-gateway release (install).


Where certificates are stored

StoreNameKeys / formatWritten byRead by
Kubernetes Secret<env>-gcp-bighammer-tls in bh-control-planetls.crt, tls.key (kubernetes.io/tls)cert-manager (Certificate) or kubectl create secret tlsnginx Ingress, bh-gateway HTTPS listener
Secret Manager<env>-gcp-bighammer-tls-crtPEM certificatePushSecret (Option A) or gcloud (Option B)bighammer ExternalSecrets → Keycloak and other consumers
Secret Manager<env>-gcp-bighammer-tls-keyPEM private keyPushSecret (Option A) or gcloud (Option B)Same

Default GSM secret names are configured in bighammer values:

global:
remoteSecrets:
tls:
cert: <env>-gcp-bighammer-tls-crt
key: <env>-gcp-bighammer-tls-key

See Secret Manager Secrets.


How the bighammer chart references TLS

ConsumerMechanismValues / template
nginx Ingressspec.tls.secretName on bh-ingressglobal.ingress.tlsSecretNamecharts/bhui/templates/ingress.yaml
Gateway HTTPScertificateRefs on bh-gateway listenerbh-gateway values: gateway.listeners.https.tlsSecretName + tlsSecretNamespace
KeycloakExternalSecret pulls GSM PEM into pod secretglobal.remoteSecrets.tls.cert / .keycharts/bhkeycloak/templates/secret.yaml

Ingress mode

global:
ingress:
enabled: true
tlsSecretName: <env>-gcp-bighammer-tls
gateway:
enabled: false

All hostnames listed under global.ingress.*HostName use this TLS secret on the single bh-ingress resource.

Gateway mode

# bh-gateway release (envoy-gateway-system)
gateway:
listeners:
https:
tlsSecretName: <env>-gcp-bighammer-tls
tlsSecretNamespace: bh-control-plane

# bighammer release
global:
ingress:
enabled: false
gateway:
enabled: true

Keycloak (GSM path)

Keycloak does not read the Ingress TLS secret directly. It syncs certificate and key from Secret Manager using the same remote keys PushSecret populates:

# ExternalSecret data keys (from chart template)
KC_HTTPS_CERTIFICATE_FILE → GSM: global.remoteSecrets.tls.cert
KC_HTTPS_CERTIFICATE_KEY_FILE → GSM: global.remoteSecrets.tls.key

Therefore Option A requires both the Kubernetes TLS secret (edge) and GSM copies (apps).


Option A — Let's Encrypt (cert-manager + DNS-01)

Best for: Dev / non-prod with a public Cloud DNS zone.

Sample manifests in the Helm chart

The bighammer chart ships reference manifests under cert-manager/ (applied with kubectl, not via helm install):

FileKubernetes resource
cert-manager/cluster-issuer.yamlIssuer — Let's Encrypt + Cloud DNS
cert-manager/wildcard-cert.yamlCertificate — wildcard request
cert-manager/push-secret.yamlPushSecret — sync to Secret Manager

Obtain the files from your chart package:

# After helm pull of bighammer OCI chart
tar -xzf bighammer-<version>.tgz
ls bighammer/cert-manager/

Prerequisites

Complete before applying manifests:

#RequirementDoc
1cert-manager cert-manager-v1.19.2 in clusterPrerequisites — add-ons
2External Secrets OperatorExternal Secrets Operator
3Workload Identity — ESO + cert-manager DNS accessWorkload Identity
4Cloud DNS public zone for your domainInfrastructure
5cert-manager controller SA with roles/dns.adminDNS-01 challenge (GCP SA bound to cert-manager/cert-manager KSA)
6SecretStore/gcp-secret-store in bh-control-planeCreated by bighammer chart or platform bootstrap
7PushSecret-capable ESO versionReference: ESO 0.18.x (PushSecret is v1alpha1)

Install bighammer once (or apply equivalent bootstrap) so SecretStore/gcp-secret-store exists before push-secret.yaml.

Provision — apply order

kubectl apply -f cluster-issuer.yaml
kubectl apply -f wildcard-cert.yaml
# Wait until Certificate is Ready and secret <env>-gcp-bighammer-tls exists
kubectl apply -f push-secret.yaml

Manifest 1 — cluster-issuer.yaml

Defines a namespace-scoped Issuer that uses Let's Encrypt production ACME with DNS-01 via Cloud DNS. cert-manager creates temporary TXT records to prove you control the zone.

Customize before apply: email, spec.acme.solvers[].dns01.cloudDNS.project (GCP project hosting the DNS zone).

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-prod
namespace: bh-control-plane
spec:
acme:
# Let's Encrypt production directory (use staging for tests)
server: https://acme-v02.api.letsencrypt.org/directory
email: admin@example.com # ← your operations contact
privateKeySecretRef:
name: letsencrypt-prod-account-key # ACME account key (auto-created)
solvers:
- dns01:
cloudDNS:
project: <PROJECT_ID> # ← GCP project with Cloud DNS zone
# Workload Identity: omit serviceAccountSecretRef.
# Bind cert-manager/cert-manager KSA → GCP SA with roles/dns.admin
FieldPurpose
kind: IssuerNamespace-scoped; lives in bh-control-plane with the Certificate
serverACME directory URL (staging URL for testing)
emailLet's Encrypt account / expiry notifications
privateKeySecretRefStores ACME account private key in the cluster
dns01.cloudDNS.projectProject where Cloud DNS zone exists
WI noteOn GKE, cert-manager uses Workload Identity — no JSON key secret

Manifest 2 — wildcard-cert.yaml

Requests a wildcard certificate from the Issuer above. cert-manager writes the issued cert and key into the Kubernetes TLS secret named secretName.

Customize: dnsNames to match your zone (must be covered by the Cloud DNS zone).

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: <env>-gcp-bighammer-wildcard
namespace: bh-control-plane
spec:
secretName: <env>-gcp-bighammer-tls # ← K8s TLS secret Ingress/Gateway use
issuerRef:
name: letsencrypt-prod
kind: Issuer
dnsNames:
- "*.<env>.<your-domain>" # ← wildcard for app hostnames
- "<env>.<your-domain>" # ← apex (optional, if needed)
FieldPurpose
metadata.nameCertificate resource name (not the secret name)
spec.secretNameOutput — must match global.ingress.tlsSecretName and bh-gateway tlsSecretName
issuerRefLinks to Issuer/letsencrypt-prod
dnsNamesSANs on the certificate; align with global.ingress.*HostName / gateway hostnames

After apply, wait for readiness:

kubectl get certificate <env>-gcp-bighammer-wildcard -n bh-control-plane
kubectl get secret <env>-gcp-bighammer-tls -n bh-control-plane

Expected: Certificate status Ready=True; secret type kubernetes.io/tls with tls.crt and tls.key.


Manifest 3 — push-secret.yaml

Uses External Secrets Operator PushSecret to copy the Kubernetes TLS secret into GCP Secret Manager so bighammer ExternalSecrets (Keycloak, etc.) can read PEM files.

Requires: SecretStore/gcp-secret-store in bh-control-plane (from bighammer chart).

# PushSecret is v1alpha1 on ESO 0.18.x
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
name: push-wildcard-cert
namespace: bh-control-plane
spec:
refreshInterval: 12h
updatePolicy: Replace
secretStoreRefs:
- name: gcp-secret-store
kind: SecretStore
selector:
secret:
name: <env>-gcp-bighammer-tls # ← source K8s secret from Certificate
data:
- match:
secretKey: tls.crt
remoteRef:
remoteKey: <env>-gcp-bighammer-tls-crt # ← GSM name (global.remoteSecrets.tls.cert)
- match:
secretKey: tls.key
remoteRef:
remoteKey: <env>-gcp-bighammer-tls-key # ← GSM name (global.remoteSecrets.tls.key)
FieldPurpose
secretStoreRefsMust match global.secretStore.name (gcp-secret-store)
selector.secret.nameSource — cert-manager TLS secret
data[].match.secretKeyStandard TLS secret keys
remoteRef.remoteKeyTarget GSM secret names — must match global.remoteSecrets.tls in values
refreshIntervalHow often ESO checks for changes (renewal updates GSM when K8s secret rotates)
updatePolicy: ReplaceOverwrite GSM secret versions when cert renews

Verify:

kubectl get pushsecret push-wildcard-cert -n bh-control-plane
gcloud secrets versions access latest --secret=<env>-gcp-bighammer-tls-crt --project=<PROJECT_ID>

Wire up bighammer values

After all three manifests succeed:

global:
remoteSecrets:
tls:
cert: <env>-gcp-bighammer-tls-crt
key: <env>-gcp-bighammer-tls-key
ingress:
enabled: true # or gateway via bh-gateway
tlsSecretName: <env>-gcp-bighammer-tls

Deploy or upgrade bighammer and confirm edge + Keycloak:

helm upgrade --install bighammer <chart> -f values.yaml -f values-<env>.yaml -n bh-control-plane

Renewal

Automatic — cert-manager renews ~30 days before expiry and updates <env>-gcp-bighammer-tls. PushSecret propagates new PEM to GSM on its refreshInterval. No Helm value changes needed if secret names stay the same.


Option B — Private SSL/TLS certificate

Best for: Production, corporate CA, or environments without public DNS-01.

Steps

  1. Obtain wildcard or SAN certificate from your CA (PEM cert + key).
  2. Store in Secret Manager (names must match global.remoteSecrets.tls):
gcloud secrets create <env>-gcp-bighammer-tls-crt \
--project=<PROJECT_ID> \
--replication-policy=automatic \
--data-file=./wildcard.crt

gcloud secrets create <env>-gcp-bighammer-tls-key \
--project=<PROJECT_ID> \
--replication-policy=automatic \
--data-file=./wildcard.key
  1. Create the Kubernetes TLS secret in bh-control-plane:
kubectl create secret tls <env>-gcp-bighammer-tls \
--cert=wildcard.crt \
--key=wildcard.key \
-n bh-control-plane
  1. Point the active edge mode at that secret:
ModeConfigure
Ingressglobal.ingress.tlsSecretName: <env>-gcp-bighammer-tls
Gatewaybh-gatewaygateway.listeners.https.tlsSecretName + tlsSecretNamespace
  1. Re-run the relevant Helm upgrade (bighammer and/or bh-gateway).

Option B skips cert-manager manifests; you maintain renewal and GSM uploads manually.

Keycloak DB private key (DER)

Keycloak requires the DB client key in DER format in Secret Manager (db-client-private-key-der):

openssl pkcs8 -topk8 -inform PEM -outform DER -in client-key.pem -out client-key.der -nocrypt
gcloud secrets versions add db-client-private-key-der --data-file=client-key.der --project=<PROJECT_ID>

Comparison

Option A (Let's Encrypt)Option B (Private cert)
CostFreeCA-dependent
AutomationFull (cert-manager + PushSecret)Manual renewal
DNSPublic Cloud DNS + DNS-01Any DNS
K8s secretcert-manager creates <env>-gcp-bighammer-tlsYou create with kubectl
Secret ManagerPushSecret populates crt/key secretsYou upload with gcloud
Manifests3 files in chart cert-manager/None
Prod fitDev / stagingRecommended for prod

Verify TLS

# Kubernetes TLS secret
kubectl get secret <env>-gcp-bighammer-tls -n bh-control-plane

# cert-manager (Option A)
kubectl get issuer,certificate,pushsecret -n bh-control-plane

# GSM (Option A or B)
gcloud secrets list --filter="name:<env>-gcp-bighammer-tls" --project=<PROJECT_ID>

# Ingress mode
kubectl get ingress -n bh-control-plane -o yaml | grep -A2 tls

# Gateway mode
kubectl describe gateway bh-gateway -n envoy-gateway-system

# External
curl -vI https://<your-hostname>

For detailed diagnosis, see Cert troubleshooting below.


DNS records

TLS certificates prove hostname identity; clients still need DNS A records pointing to your load balancer:

ModeGet LB IP
NGINX Ingresskubectl get svc -n ingress-nginx
Gateway APIData-plane Service in envoy-gateway-systembh-gateway

Hostnames must match certificate SANs (dnsNames in wildcard-cert.yaml or your private cert).


Cert troubleshooting

Use this section when HTTPS fails, certificates stay Pending, PushSecret does not sync, or browsers report certificate errors. Work top to bottom — edge issues often look like app issues when the K8s TLS secret is missing or misnamed.

Quick diagnosis

# 1. Issuance (Option A)
kubectl describe certificate <env>-gcp-bighammer-wildcard -n bh-control-plane
kubectl describe issuer letsencrypt-prod -n bh-control-plane
kubectl get challenges -n bh-control-plane

# 2. Kubernetes TLS secret (both options)
kubectl get secret <env>-gcp-bighammer-tls -n bh-control-plane -o jsonpath='{.type}{"\n"}{.data.tls\.crt}{"\n"}' | head -2

# 3. PushSecret → GSM (Option A)
kubectl describe pushsecret push-wildcard-cert -n bh-control-plane
gcloud secrets versions list <env>-gcp-bighammer-tls-crt --project=<PROJECT_ID> --limit=3

# 4. Edge reference
kubectl get ingress bh-ingress -n bh-control-plane -o yaml | grep -A5 'tls:'
kubectl describe gateway bh-gateway -n envoy-gateway-system | grep -A10 'Listeners:'

# 5. External check
openssl s_client -connect <hostname>:443 -servername <hostname> </dev/null 2>/dev/null | openssl x509 -noout -subject -dates -ext subjectAltName
CheckHealthy signal
CertificateReady=True
K8s secretType kubernetes.io/tls; keys tls.crt, tls.key
PushSecretStatus Synced / Ready
GSMLatest version exists for crt and key secrets
Ingressspec.tls[].secretName = global.ingress.tlsSecretName
GatewayHTTPS listener certificateRefs point to secret in tlsSecretNamespace
Browser / opensslSubject/SAN includes your hostname; dates valid

cert-manager — certificate not Ready

SymptomLikely causeWhat to do
Certificate Pending indefinitelyIssuer not ready or ACME account issuekubectl describe issuer letsencrypt-prod -n bh-control-plane; confirm Ready=True and ACME account key secret exists
Challenge stuck pendingDNS-01 cannot create TXT recordsSee DNS-01 and Cloud DNS
Challenge invalidWrong zone, rate limit, or bad dnsNameskubectl describe challenge -n bh-control-plane; fix dnsNames in wildcard-cert.yaml to match your Cloud DNS zone
Certificate Ready but no secretWrong secretName or namespace mismatchCertificate and secret must both be in bh-control-plane
Let's Encrypt rate limit errorToo many failed or duplicate requestsUse staging ACME server in cluster-issuer.yaml for testing; wait for limit reset in prod
kubectl get certificate,order,challenge -n bh-control-plane
kubectl logs -n cert-manager deploy/cert-manager --tail=100

DNS-01 and Cloud DNS

SymptomLikely causeWhat to do
TXT record never appearscert-manager lacks roles/dns.adminBind cert-manager KSA (cert-manager/cert-manager) to a GCP SA with roles/dns.admin on the DNS project — Workload Identity
TXT in wrong projectcloudDNS.project mismatchSet spec.acme.solvers[].dns01.cloudDNS.project in cluster-issuer.yaml to the project that hosts the DNS zone
Zone not foundDomain not in Cloud DNSCreate or delegate a public managed zone; apex/wildcard names must live under that zone
WI still using JSON key pathserviceAccountSecretRef set incorrectlyOn GKE, omit serviceAccountSecretRef under cloudDNS; use Workload Identity only
Challenge succeeds then fails on renewIAM or zone changedRe-verify cert-manager SA binding; check Cloud DNS audit logs

Verify TXT records during an active challenge:

gcloud dns record-sets list --zone=<ZONE_NAME> --project=<PROJECT_ID> --filter="_acme-challenge"

PushSecret and Secret Manager

SymptomLikely causeWhat to do
PushSecret not foundApplied before ESO or wrong namespaceApply push-secret.yaml in bh-control-plane; ESO must be installed
PushSecret not readySecretStore/gcp-secret-store missingInstall or upgrade bighammer so SecretStore exists — External Secrets Operator
Permission denied on pushESO GCP SA lacks Secret Manager writeGrant roles/secretmanager.admin or secret-level access to the ESO Workload Identity SA
GSM secrets empty or staleSource K8s secret missing or PushSecret not syncedConfirm <env>-gcp-bighammer-tls exists first; kubectl describe pushsecret push-wildcard-cert -n bh-control-plane
Keycloak TLS errors but edge HTTPS worksGSM crt/key not updatedPushSecret only runs on interval (refreshInterval: 12h); after renewal, wait or temporarily lower interval; confirm remoteKey names match global.remoteSecrets.tls
Wrong GSM secret namesremoteKey ≠ chart valuesAlign push-secret.yaml remoteKey with global.remoteSecrets.tls.cert / .key
kubectl get secretstore gcp-secret-store -n bh-control-plane
kubectl get externalsecret -n bh-control-plane | grep -i tls
gcloud secrets describe <env>-gcp-bighammer-tls-crt --project=<PROJECT_ID>

Edge proxy — Ingress or Gateway

SymptomLikely causeWhat to do
HTTP works, HTTPS failsTLS secret missing or wrong nameCreate or fix <env>-gcp-bighammer-tls; set global.ingress.tlsSecretName (Ingress) or bh-gateway tlsSecretName + tlsSecretNamespace
default backend certificate / wrong cert in browserHostname not on cert SANAdd hostname to dnsNames (Option A) or re-issue private cert (Option B); wildcard must cover subdomain
Ingress serves HTTP onlytls block absent on IngressUpgrade bighammer with global.ingress.tlsSecretName set; kubectl get ingress bh-ingress -n bh-control-plane -o yaml
Gateway listener not programmedSecret in wrong namespaceGateway cannot read secrets across namespaces without reference grant; set tlsSecretNamespace: bh-control-plane on bh-gateway
502 / connection reset on 443Edge up but backends unhealthySeparate from TLS — check pod readiness; see Troubleshooting

Ingress mode

kubectl get ingress bh-ingress -n bh-control-plane -o jsonpath='{.spec.tls}'
helm get values bighammer -n bh-control-plane | grep -A2 tlsSecretName

Gateway mode

kubectl get gateway bh-gateway -n envoy-gateway-system -o yaml | grep -A15 certificateRefs
helm get values bh-gateway -n envoy-gateway-system | grep -A5 tlsSecret

Certificate name mismatch (browser errors)

Browser / curl messageMeaningFix
NET::ERR_CERT_COMMON_NAME_INVALIDHostname not in certificate SANMatch DNS hostname to dnsNames or cert SANs
certificate has expiredPast notAfterOption A: check cert-manager renewal; Option B: upload new PEM to GSM + K8s secret
unable to get local issuer certificateIncomplete chainEnsure tls.crt contains full chain (leaf + intermediates)
SSL certificate problem: self signedWrong secret mounted or staging cert left in placeConfirm secret contents; use production Issuer URL for real traffic

Compare served cert vs expected secret:

# Cert presented on the wire
echo | openssl s_client -connect <hostname>:443 -servername <hostname> 2>/dev/null | openssl x509 -noout -text | grep -A1 'Subject Alternative Name'

# Cert in cluster secret (decode first line only)
kubectl get secret <env>-gcp-bighammer-tls -n bh-control-plane -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text | grep -A1 'Subject Alternative Name'

Option B — private certificate

SymptomLikely causeWhat to do
Edge HTTPS broken after upload to GSM onlyK8s TLS secret not createdkubectl create secret tls <env>-gcp-bighammer-tls --cert=... --key=... -n bh-control-plane
Keycloak fails, edge OKGSM names or format wrongUpload PEM to <env>-gcp-bighammer-tls-crt / -key; names must match global.remoteSecrets.tls
Renewed cert not picked upOld secret version cachedCreate new GSM version; restart affected pods or wait for ExternalSecret refresh
Mixed old/new materialCert and key from different issuancesRe-upload matching pair to GSM and recreate K8s secret

Apply order mistakes

MistakeEffectRecovery
push-secret before wildcard-certPushSecret errors (no source secret)Apply wildcard-cert.yaml; wait for Ready; re-apply or wait for PushSecret reconcile
wildcard-cert before cluster-issuerCertificate cannot issueApply cluster-issuer.yaml first
Changed secretName in Certificate onlyIngress/Gateway still point at old nameUpdate global.ingress.tlsSecretName and bh-gateway listener to match or revert secretName
Deleted K8s TLS secret manuallyEdge HTTPS breaks until re-issuedOption A: cert-manager recreates on reconcile; Option B: re-run kubectl create secret tls

Still stuck?

  1. Collect: kubectl describe certificate,issuer,pushsecret -n bh-control-plane, cert-manager logs, and ESO controller logs.
  2. Confirm the full chain: Issuer Ready → Certificate Ready → K8s secret → PushSecret Synced → GSM versions → Helm values → edge listener.
  3. See also Troubleshooting — TLS for platform-wide symptoms.

TopicLink
GSM secret namesSecret Manager Secrets
NGINX edgeNGINX Ingress Deployment
Gateway edgebh-gateway
ESO + SecretStoreExternal Secrets Operator
cert-manager add-onPrerequisites