GCP Infrastructure
The BigHammer platform can be deployed on Google Cloud Platform (GCP) using various provisioning methods, including:
- GCP Console
- GCP CLI (
gcloud) - Terraform
- Infrastructure-as-Code (IaC) tools approved by the customer
Regardless of the provisioning method used, the following reference architecture, resource requirements, and configuration values should be used as the baseline for setting up the infrastructure required for the BigHammer platform.
The sections below provide the recommended GCP services, networking requirements, IAM configuration, storage, compute resources, and other foundational components necessary for a successful deployment.
The exact implementation may vary based on customer-specific security, networking, compliance, and operational requirements. The reference values below are the minimum recommended configuration for deploying and operating the BigHammer platform in GCP.
After foundation is provisioned, use Prerequisites for the Helm deploy checklist (add-ons, firewall, tools).
GCP services recommended configuration
| Service | Description |
|---|---|
| GKE | Private GKE cluster |
| Cloud SQL | Cloud SQL PostgreSQL — bighammer_db_<env>, keycloak_db_<env>, users, SSL certs → passwords in Secret Manager (incl. bh-kcdbpwd) |
| Secret Manager | Bootstrap secrets; otherwise use gcloud / Console to create reference secrets (DB passwords, client certs) |
| GCS | GCS buckets — placeholder (core Helm chart does not require GCS; Composer needs buckets for DAGs and related jobs) |
| Cloud DNS | Cloud DNS zone + application A records |
| IAM | CI/CD SA, GAR reader for GKE node SA |
| Workload Identity | CI/CD or GitHub Actions + GKE Workload Identity bindings |
| KMS | KMS for Secret Manager encryption |
| Cloud Composer | Cloud Composer (Airflow) environment |
| Dataproc | Dataproc service accounts (VM-backed clusters) |
| Vertex AI | Chat LLMs (Claude, Gemini) for AI Agent — APIs, SA, Model Garden, tenant config |
Project & networking (reference dev)
| Setting | Reference value |
|---|---|
| Project ID | Customer-specific (example: place-holder) |
| Region / zone | Customer-specific (example: us-east1 / us-east1-b) |
| Network | Customer-specific VPC (example: place-holder) |
| Host VPC project | Dedicated or Shared VPC host project (example: place-holder) |
| Subnet | Customer-specific primary subnet (example: place-holder) |
| Pod / service secondary ranges | Customer-specific secondary ranges for pods and services |
GKE
| Setting | Reference value |
|---|---|
| Cluster name prefix | Standard naming convention (+ random suffix) |
| Kubernetes version | BigHammer tested version: 1.34.6-gke.1307000 (There is no specific constraint for other GKE versions, It should work without any issues.) |
| Release channel | STABLE |
| Node service account | Customer-specific; example: <your-gke-node-sa>@<project>.iam.gserviceaccount.com |
| Workload Identity | Enabled |
| GCE PD CSI Driver | Enabled |
| GCS FUSE CSI driver | Enabled [Optional] |
| Node IAM (additional) | Add roles/artifactregistry.reader on node SA to pull images from Google Artifact Registry |
Node pool — platform / AI Agent (required)
| Setting | Reference value |
|---|---|
| Pool name | Customer-specific; example: <your-node-pool> |
| Machine type | e2-standard-2 (2 vCPU, 8 GB RAM). There is no specific constraint for machine type, It should work without any issues with other machine types as well. |
| Node locations | Regional — zones supported in the target region; example: us-east1-b, us-east1-c |
| Autoscaling | min 1, max 4 (Since workloads are HPA enabled, and GKE will auto scaling enabled these values can be decided based on requirements ) |
| Boot disk | 50 GB pd-standard |
| Node image | COS_CONTAINERD |
| Secure Boot | Enabled |
Additional node pools (optional)
No Specific requirement for Bighammer.
Cloud SQL
| Setting | Reference value |
|---|---|
| Instance name | Customer-specific; example: <your-postgres-instance> |
| PostgreSQL version | 16 (POSTGRES_16) — tested |
| Machine tier | db-custom-2-3840 (2 vCPU, 3.75 GB RAM) |
| Storage | 20 GB (minimum; scale per workload) |
| Connectivity | Private IP on VPC (no authorized networks) |
| SSL mode | Per customer security policy (recommended: TRUSTED_CLIENT_CERTIFICATE_REQUIRED) |
Databases
Provision on the Cloud SQL instance. Names must match Helm values (POSTGRES_DB, KC_DB_URL).
| Database | Purpose | Helm / chart |
|---|---|---|
bighammer_db_<env> | Application data (APIs, workers, keycloakapi_db schema) | POSTGRES_DB on app services |
keycloak_db_<env> | Keycloak server database (SSO runtime tables) | KC_DB_URL → .../keycloak_db_<env>?... on bhkeycloak |
| Application | Keycloak server | |
|---|---|---|
| Database | bighammer_db_<env> | keycloak_db_<env> |
| Provisioned by | Terraform + schemas via bighammer_db_setup.sql | Terraform only (not in SQL script) |
| Chart service | APIs, workers, bhkeycloakapi | bhkeycloak only |
| Env vars | POSTGRES_DB, dual-role users | KC_DB_URL, KC_DB_USERNAME, KC_DB_PASSWORD |
| DB user | bh_dev_user / bh_app_user | keycloak_db_user |
| GSM password | bh-dev-user-password, bh-app-user-password | bh-kcdbpwd |
bhkeycloakapi uses bighammer_db_<env> (schema keycloakapi_db), not keycloak_db_<env>.
Reference Terraform creates both databases via additional_databases on the PostgreSQL module (for example bighammer_db_<env> and keycloak_db_<env> in <env>.auto.tfvars). Customer equivalents use the same naming pattern or override names consistently — see Renaming databases or users.
Database users
Users are created on the instance together with passwords stored in Secret Manager (reference: Terraform random_password → GSM). See Secret Manager Secrets for chart mapping.
| User | Purpose | Secret Manager password |
|---|---|---|
postgres | Admin / bootstrap; not used by Helm at runtime | db-postgres-password |
bh_dev_user | Migrations / Alembic — CREATE on schemas, DDL on tables | bh-dev-user-password |
bh_app_user | Runtime — DML only (BH_APP_USER in chart) | bh-app-user-password |
keycloak_db_user | Keycloak server JDBC (KC_DB_USERNAME on bhkeycloak) | bh-kcdbpwd |
Terraform-managed (reference): Cloud SQL instance, databases (bighammer_db_<env>, keycloak_db_<env>), users above, client SSL certificates, and GSM secrets db-postgres-password, bh-dev-user-password, bh-app-user-password, bh-kcdbpwd, db-client-cert, db-client-private-key, db-server-ca-cert.
After Terraform (or equivalent provisioning):
- Application DB — run
bighammer_db_setup.sqlfrom the chartscripts/folder aspostgresto create schemas and grants insidebighammer_db_<env>(skip or commentCREATE DATABASE; use\connect bighammer_db_<env>). See Scripts Reference. Troubleshooting: Database Bootstrap Troubleshooting. - Keycloak DB — database and user already exist; ensure
keycloak_db_userhasCONNECTandCREATEonkeycloak_db_<env>andCREATEon schemapublic(Keycloak creates its own tables). Password is already in GSM asbh-kcdbpwd. - One-time secrets — run
bh_gsm_onetime_setup.shfor secrets not created by Terraform (admin password, RabbitMQ, license, etc.).
Cloud DNS
| Setting | Reference value |
|---|---|
| Zone name | Customer-specific; example: <your-dns-zone> |
| Domain | Customer-specific; example: <your-domain> |
| Zone type | Per customer security policy — public or private hosted zone |
Record examples: ui.<env>.<your-domain>, catalog.<env>.<your-domain>, kc.<env>.<your-domain> (A records → ingress / Gateway load balancer IP).
Secret Manager & KMS
| Setting | Reference value |
|---|---|
| KMS key ring | Customer-specific; example: <your-kms-keyring> (or default Secret Manager encryption) |
| KMS crypto key | Customer-specific; example: <your-kms-crypto-key> (or default Secret Manager encryption) |
Secret names and chart mapping: Secret Manager Secrets.
GCS buckets (reference)
| Bucket | Purpose |
|---|---|
| Customer-defined | Composer DAG / plugin storage (required for Composer) |
| Customer-defined | Spark / Dataproc event logs (optional analytics path) |
Artifact Registry (reference)
| Setting | Reference value |
|---|---|
| Repository | Docker images — example: <bh-catalog-docker-repo> |
| Region | Customer-specific; example: us-east1 |
| Image path prefix | us-docker.pkg.dev/{customer-project}/{artifact-registry-name}/ |
Cloud Composer (reference)
| Setting | Reference value |
|---|---|
| Environment | Customer-specific; example: <your-composer-environment> |
| Image | Composer image version from your release channel (example: composer-3-airflow-3.x) |
| Size | Per workload (example: ENVIRONMENT_SIZE_SMALL) |
| Service account | Customer-specific; example: <your-composer-sa> |
| GCS bucket (DAG sync) | Customer-specific; example: <your-composer-dag-bucket> |
Composer SA includes roles/dataproc.editor for submitting Spark jobs to Dataproc.
Dataproc (reference)
| Setting | Reference value |
|---|---|
| Cluster key | Customer-specific; example: <your-dataproc-cluster> |
| Service account | Customer-specific; example: <your-dataproc-sa> |
| IAM role | roles/dataproc.worker |
Vertex AI (LLM)
Optional - Required only when AI Agent tenants use Vertex-hosted chat models (anthropic_vertex or google_vertex providers).
APIs
| API | Purpose |
|---|---|
aiplatform.googleapis.com | Vertex AI model invocation (Claude partner models, Gemini) |
Enable via Console, gcloud services enable, or your IaC (Terraform api_names).
Service account (Terraform-provisioned)
| Setting | Dev example | Placeholder |
|---|---|---|
| Account ID | bh-nprd-vertexai-user | bh-<env>-vertexai-user |
bh-<env>-vertexai-user@<project-id>.iam.gserviceaccount.com | same | |
| Display name | Vertex AI user for AI Agent (JSON key auth) | same |
IAM roles (project scope, assigned by Terraform):
| Role | Purpose |
|---|---|
roles/aiplatform.user | Invoke Vertex AI chat models |
This SA is provisioning is required when tenant is using without Workload Identity bindings. Download a JSON key manually and configure tenant LLM with wif_enabled: false [ Setting can be seen in BHUI].
Model Garden
Before first inference, enable target models in Vertex AI → Model Garden for <project-id>:
| Model family | Example model ID | Provider key in AI Agent |
|---|---|---|
| Anthropic Claude | claude-<version> | anthropic_vertex |
| Google Gemini | gemini-<version> | google_vertex |
| Setting | Notes |
|---|---|
| Region / location | e.g. global, us-central1 — must match where the model is enabled |
| Terms | Partner models (Claude) may require one-time acceptance in Model Garden |
When enabling Workload Identity in Admin Console, you must confirm that the Vertex AI model and AI Agent run on the same cloud platform. On mismatched platforms, disable WIF and use a service account key.
Post-provision verification
gcloud iam service-accounts describe bh-<env>-vertexai-user@<project-id>.iam.gserviceaccount.com
gcloud iam service-accounts keys create ./bh-<env>-vertexai-user-key.json \
--iam-account=bh-<env>-vertexai-user@<project-id>.iam.gserviceaccount.com
Workload Identity service accounts
| GCP service account | Kubernetes service account(s) | IAM role |
|---|---|---|
external-secrets-operator | bh-control-plane/external-secrets-operator | secretmanager.secretAccessor |
<your-app-gsa> | All app KSAs in bh-control-plane | secretmanager.admin |
Full bindings and chart config: Workload Identity.
Helm chart inputs (after provisioning)
Map provisioned resources into your environment values file (for example values-prod.yaml):
| Helm path | Source |
|---|---|
global.secretStore.projectId | GCP project ID |
global.gcpProject / global.gcpRegion | Project and region |
global.image.repository | GAR or customer registry path |
POSTGRES_HOST | Cloud SQL private IP |
POSTGRES_USER / POSTGRES_DB | bh_dev_user / bighammer_db_<env> (migrations — Alembic) |
BH_APP_USER | bh_app_user (runtime SQLAlchemy URI) |
global.remoteSecrets.postgres.password | GSM key for POSTGRES_PASSWORD → use bh-dev-user-password |
global.remoteSecrets.app.userPassword | GSM key for BH_APP_USER_PASSWORD → bh-app-user-password |
global.bhkeycloak.env.KC_DB_USERNAME | keycloak_db_user |
global.bhkeycloak.env.KC_DB_URL | JDBC URL with database keycloak_db_<env> |
global.remoteSecrets.keycloak.dbPassword | bh-kcdbpwd (Terraform → GSM) |
global.remoteSecrets.* | Other Secret Manager secret names |
Renaming databases or users
If Terraform (or manual provisioning) uses different database or user names than the reference defaults, update every consumer below. Chart templates do not hardcode database names — all runtime values come from your values-<env>.yaml overlay.
bighammer_db_<env> → custom application database name
| # | Helm values path (values-<env>.yaml) | Field | Services affected |
|---|---|---|---|
| 1 | global.bhauditapi.env | POSTGRES_DB | bhauditapi |
| 2 | global.bhauditworker.env | POSTGRES_DB | bhauditworker |
| 3 | global.bhcatalogapi.env | POSTGRES_DB | bhcatalogapi |
| 4 | global.bhcatalogworker.env | POSTGRES_DB | bhcatalogworker |
| 5 | global.bhaiagent.env | POSTGRES_DB | bhaiagent |
| 6 | global.bhaiagent.env | SYNC_DB_URL | bhaiagent (path segment after host/port) |
| 7 | global.bhaiagentworker.env | POSTGRES_DB | bhaiagentworker |
| 8 | global.bhaiagentworker.env | SYNC_DB_URL | bhaiagentworker |
| 9 | global.bhkeycloakapi.env | POSTGRES_DB | bhkeycloakapi |
| 10 | global.bhcelerybeat.env | POSTGRES_DB | bhcelerybeat |
Total in chart values: 10 fields across 8 services (8× POSTGRES_DB + 2× SYNC_DB_URL).
Also update outside the Helm values file (not chart templates):
| Location | What to change |
|---|---|
| Terraform / Cloud SQL | additional_databases entry for the application DB |
bighammer_db_setup.sql | \connect, GRANT CONNECT ON DATABASE, and CREATE DATABASE (if used) |
| GSM | No secret name change for app users; passwords must still match bh_dev_user / bh_app_user in Cloud SQL |
keycloak_db_<env> → custom Keycloak database name
| # | Helm values path | Field | Service |
|---|---|---|---|
| 1 | global.bhkeycloak.env | KC_DB_URL | bhkeycloak — database name is the path segment after host/port in the JDBC URL (.../keycloak_db_<env>?...) |
Total in chart values: 1 field (KC_DB_URL). KC_DB_HOST / KC_DB_PORT are separate and do not embed the database name.
Also update outside Helm values:
| Location | What to change |
|---|---|
| Terraform / Cloud SQL | additional_databases entry for Keycloak DB |
GSM bh-kcdbpwd | Secret name unchanged; value must match the Keycloak DB user password in Cloud SQL |
bighammer_db_setup.sql does not touch the Keycloak database.
keycloak_db_user → custom Keycloak DB user name
| # | Helm values path | Field | Service |
|---|---|---|---|
| 1 | global.bhkeycloak.env | KC_DB_USERNAME | bhkeycloak |
Total in chart values: 1 field.
Also update outside Helm values:
| Location | What to change |
|---|---|
| Terraform / Cloud SQL | additional_users entry for Keycloak JDBC user |
GSM bh-kcdbpwd | Password for the new username |
global.remoteSecrets.keycloak.dbPassword | Only if you rename the GSM secret itself (default stays bh-kcdbpwd) |
Quick reference — reference defaults
| Object | Terraform (reference) | Chart env var | GSM |
|---|---|---|---|
| Application DB | bighammer_db_<env> | POSTGRES_DB (8 services) | — |
| Keycloak DB | keycloak_db_<env> | KC_DB_URL (bhkeycloak) | — |
| Keycloak user | keycloak_db_user | KC_DB_USERNAME | bh-kcdbpwd |
See also Scripts Reference — Keycloak database and Secret Manager — Keycloak.
Customer artifact mirror workflow: Release Manifest & Distribution.