Cloud Architecture on GCP
High-level cloud architecture for deploying the BigHammer platform on Google Kubernetes Engine (GKE) using the platform umbrella Helm chart.
This document describes components and data flows in a cloud-agnostic, environment-neutral way. Replace placeholders such as <project-id>, <your-domain>, and <env> with values from your deployment.
Architecture overview
Deployment layers
| Layer | Responsibility |
|---|---|
| GCP foundation | GKE, Cloud SQL, Secret Manager, Cloud DNS, GCS, Artifact Registry |
| Orchestration & analytics | Cloud Composer (Airflow), Dataproc on VM, Vertex AI |
| Cluster add-ons | ESO, ingress-nginx or Envoy Gateway + bh-gateway, RabbitMQ |
| Platform chart | APIs, UI, workers, Keycloak, HTTPRoutes / Ingress |
| Edge | NGINX Ingress or Gateway API (bh-gateway) |
Orchestration & ML connections
| From | To | Purpose |
|---|---|---|
GKE (bhaiagent) | Vertex AI | LLM inference and embeddings for AI Agent workloads |
| Cloud Composer | Dataproc (VM) | Airflow DAGs submit Spark / batch jobs |
| Dataproc (VM) | Secret Manager | Job credentials and runtime secrets |
| Dataproc (VM) | GCS | Input/output data, staging, event logs |
| Dataproc (VM) | Artifact Registry | Spark container images and job dependencies |
Reference Terraform modules may define Composer and Dataproc resources; the Composer service account typically includes roles/dataproc.editor for job submission.
Data flow
Platform chart components
| Sub-chart | Role |
|---|---|
bhui, bhadminui | Web UIs |
bhcatalogapi, bhauditapi, bhaiagent, bhkeycloakapi | APIs |
bhcatalogworker, bhauditworker, bhaiagentworker, bhcelerybeat | Celery workers |
bhkeycloak | Identity (SSO) |
bhtransformationutils | Transformation API |
Deploy separately: ESO, edge controller, RabbitMQ, Redis (optional).
Authentication model
| Component | Auth method |
|---|---|
| External Secrets Operator | GKE Workload Identity → external-secrets-operator@<project>.iam.gserviceaccount.com |
| Application pods | Workload Identity → shared app GSA (<your-app-gsa>) |
| Cloud SQL | Private IP + client certs; postgresSSLMode: require in chart values |
| Image pull | GAR reader on node SA, or imagePullSecrets |
Hostnames (example: <env> = dev)
Configured under global.ingress.*HostName in values-<env>.yaml:
| Host | Example |
|---|---|
| UI | ui.<env>.<your-domain> |
| Catalog | catalog.<env>.<your-domain> |
| Audit | audit.<env>.<your-domain> |
| Keycloak | kc.<env>.<your-domain> |
| Keycloak API | kcapi.<env>.<your-domain> |
| AI Agent | aiagent.<env>.<your-domain> |
| Admin UI | adminui.<env>.<your-domain> |
DNS A records point to the ingress / Gateway load balancer IP.