RabbitMQ Broker

BigHammer workers and APIs use RabbitMQ for Celery task queues. The broker is not part of the bighammer umbrella chart — deploy it as a separate Helm release (or point apps at an existing broker).

Prerequisites

External Secrets Operator installed
Workload Identity for ESO → Secret Manager
GCP Secret Manager secret bh-rabbitmq-password (same password used by the broker and app pods)
SecretStore gcp-secret-store in bh-control-plane (created by bighammer platform chart or your bootstrap)

Choose an installation path

Option	When to use
A — BigHammer `bhrabbitmq` chart (recommended)	Standard BigHammer on GKE; chart published to ECR OCI
B — Upstream Bitnami chart	You manage Bitnami RabbitMQ yourself; align service name and secrets with `bighammer`
C — Existing broker	Corporate or shared RabbitMQ already running; configure `bighammer` workers only

All paths must expose AMQP on port 5672 and use a password synced from bh-rabbitmq-password in Secret Manager (unless you override global.remoteSecrets.rabbitmq).

Option A — BigHammer `bhrabbitmq` chart (OCI)

BigHammer publishes the bhrabbitmq chart to AWS ECR. Mirror to your GAR per Release Manifest & Distribution.

Item	Value
Release name	`bhrabbitmq` (fixed — matches `fullnameOverride`)
Namespace	`bh-control-plane`
Service DNS	`bhrabbitmq.bh-control-plane.svc.cluster.local:5672`
OCI chart	`oci://<bh-aws-account-id>.dkr.ecr.<bh-catalog-aws-region>.amazonaws.com/gcp/bh-helm-release/bhrabbitmq`
Chart version	Pin to published tag from your release manifest

1. Create the password in Secret Manager

gcloud secrets create bh-rabbitmq-password \
  --project=<PROJECT_ID> \
  --replication-policy=automatic

gcloud secrets versions add bh-rabbitmq-password \
  --project=<PROJECT_ID> \
  --data-file=- <<< 'your-secure-password'

See Secret Manager Secrets for the full global.remoteSecrets mapping.

2. Authenticate to ECR

export ECR_REGISTRY=<bh-aws-account-id>.dkr.ecr.<bh-catalog-aws-region>.amazonaws.com
export ECR_OCI_CHART=oci://${ECR_REGISTRY}/gcp/bh-helm-release/bhrabbitmq
export CHART_VERSION=<chart-version>

aws ecr get-login-password --region <bh-catalog-aws-region> | \
  helm registry login --username AWS --password-stdin "${ECR_REGISTRY}"

3. Prepare GKE values overlay

Create values-gcp-rabbitmq.yaml (or pull defaults from the chart tarball):

global:
  cloudProvider: gcp
  secretStore:
    kind: SecretStore
    name: gcp-secret-store

externalSecrets:
  enabled: true
  passwordRemoteKey: bh-rabbitmq-password

auth:
  existingPasswordSecret: bhrabbitmq-credentials
  existingSecretPasswordKey: rabbitmq-password

storageClass:
  enabled: true
  name: bh-rabbitmq-sc
  provisioner: pd.csi.storage.gke.io
  parameters:
    type: pd-standard

persistence:
  storageClass: bh-rabbitmq-sc

replicaCount: 1
ingress:
  enabled: false

4. Install

helm pull "${ECR_OCI_CHART}" --version "${CHART_VERSION}"
tar -xzf "bhrabbitmq-${CHART_VERSION}.tgz"

helm upgrade --install bhrabbitmq ./bhrabbitmq \
  -f bhrabbitmq/values.yaml \
  -f values-gcp-rabbitmq.yaml \
  -n bh-control-plane \
  --create-namespace

Or install directly from OCI if your values files are on disk:

helm upgrade --install bhrabbitmq "${ECR_OCI_CHART}" \
  --version "${CHART_VERSION}" \
  -f values-gcp-rabbitmq.yaml \
  -n bh-control-plane \
  --create-namespace

5. Verify

kubectl get externalsecret,secret bhrabbitmq-credentials -n bh-control-plane
kubectl get svc,statefulset bhrabbitmq -n bh-control-plane

kubectl run -it --rm nettest --image=busybox:1.36 --restart=Never -n bh-control-plane -- \
  nc -zv bhrabbitmq.bh-control-plane.svc.cluster.local 5672

Credential flow

Broker: ESO syncs GSM → bhrabbitmq-credentials → RabbitMQ auth.existingPasswordSecret
Apps: bighammer ExternalSecrets read the same GSM key into per-app secrets as RABBITMQ_PASSWORD

# bighammer values
global:
  remoteSecrets:
    rabbitmq:
      password: bh-rabbitmq-password

Management UI — Gateway API (HTTPRoute in `bighammer`)

Applies to Options A, B, and C when the broker Service is in-cluster and you use Gateway API.

The bighammer chart already defines the RabbitMQ management HTTPRoute in templates/gateway-httproutes.yaml. You do not enable ingress on the bhrabbitmq chart — routing is owned by the platform chart when global.gateway.enabled: true.

Enable and set the hostname in values-<env>.yaml:

global:
  gateway:
    enabled: true
    routes:
      bhrabbitmq:
        enabled: true
        name: bhrabbitmq-route
        hostname: rabbitmq.<env>.<your-domain>
        match:
          path:
            type: PathPrefix
            value: /
        backendRef:
          name: bhrabbitmq
          port: 15672
          # namespace: messaging   # only if broker Service is not in global.namespace

Field	Default	Purpose
`match.path`	`PathPrefix` `/`	Management UI root
`backendRef.port`	`15672`	RabbitMQ management plugin
`backendRef.name`	`bhrabbitmq`	Kubernetes Service name
`backendRef.namespace`	(empty → app namespace)	Set when broker runs in another namespace

Requires Gateway API + bh-gateway and DNS for hostname.

Deploy order (Gateway API + management UI)

Update bh-gateway values if the broker namespace differs from bh-control-plane — see Cross-namespace broker.
Redeploy bh-gateway — helm upgrade bh-gateway (OCI) in envoy-gateway-system.
Deploy bhrabbitmq — broker Service must exist before routes reconcile.
Deploy bighammer — HTTPRoute created with other app routes.

Management UI — NGINX Ingress

With global.ingress.enabled: true, bh-ingress routes global.ingress.rabbitmqHostName → bhrabbitmq:15672 (see charts/bhui/templates/ingress.yaml). Set:

global:
  ingress:
    enabled: true
    rabbitmqHostName: rabbitmq.<env>.<your-domain>

No bh-gateway changes required for nginx mode.

Option B — Upstream Bitnami RabbitMQ chart

Use the official Bitnami RabbitMQ Helm chart if your team already standardizes on Bitnami installs.

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

helm upgrade --install bhrabbitmq bitnami/rabbitmq \
  --namespace bh-control-plane \
  --create-namespace \
  --set fullnameOverride=bhrabbitmq \
  --set auth.username=admin \
  --set auth.existingPasswordSecret=bhrabbitmq-credentials \
  --set auth.existingSecretPasswordKey=rabbitmq-password

You must still:

Create bh-rabbitmq-password in Secret Manager
Sync password to bhrabbitmq-credentials (ExternalSecret or manual kubectl create secret)
Set fullnameOverride: bhrabbitmq so the Service DNS matches bighammer defaults
Configure global.remoteSecrets.rabbitmq.password in bighammer to the same GSM key

The BigHammer bhrabbitmq chart (Option A) adds GCP SecretStore integration, storage class, and tested defaults — prefer it unless you have a strong reason to run raw Bitnami.

Cross-namespace broker (Gateway API)

Use this when the broker Service runs in a different namespace than bighammer (for example messaging), but HTTPRoutes stay in bh-control-plane.

1. Update `bh-gateway` ReferenceGrants

Edit your bh-gateway values overlay, then redeploy bh-gateway before installing the broker:

referenceGrants:
  routeNamespaces:
    - bh-control-plane    # HTTPRoute namespace (global.namespace)
    - messaging           # optional: only if HTTPRoutes are created in messaging
  crossNamespace:
    - routeNamespace: bh-control-plane
      backendNamespace: messaging

Grant	Purpose
`routeNamespaces`	Namespaces allowed to attach HTTPRoutes to `bh-gateway`
`crossNamespace`	HTTPRoute in `routeNamespace` may target Services in `backendNamespace`

helm upgrade --install bh-gateway "${BH_GATEWAY_CHART}" \
  --version "${CHART_VERSION}" \
  -f values-gcp.yaml \
  -n envoy-gateway-system \
  --force-conflicts

Details: bh-gateway — HTTPRoute namespaces.

2. Deploy broker in the target namespace

helm upgrade --install bhrabbitmq "${ECR_OCI_CHART}" \
  --version "${CHART_VERSION}" \
  -f values-gcp-rabbitmq.yaml \
  -n messaging \
  --create-namespace

Ensure the Service name matches backendRef.name (default bhrabbitmq).

3. Point `bighammer` HTTPRoute at the remote Service

global:
  namespace: bh-control-plane
  gateway:
    routes:
      bhrabbitmq:
        enabled: true
        hostname: rabbitmq.<env>.<your-domain>
        backendRef:
          name: bhrabbitmq
          port: 15672
          namespace: messaging

Workers still use AMQP DNS for the broker namespace, for example bhrabbitmq.messaging.svc.cluster.local.

Option C — Existing RabbitMQ broker

If a RabbitMQ cluster already exists (in-cluster or reachable from GKE), skip the bhrabbitmq Helm release and configure bighammer worker/API env vars.

1. Secret Manager

Store the broker password under the name your chart expects (default bh-rabbitmq-password):

gcloud secrets versions add bh-rabbitmq-password \
  --project=<PROJECT_ID> \
  --data-file=- <<< 'existing-broker-password'

Or override the remote key:

global:
  remoteSecrets:
    rabbitmq:
      password: <your-gsm-secret-name>

2. `bighammer` connection settings

Update each worker/API block in values-<env>.yaml (pattern used in reference dev):

global:
  remoteSecrets:
    rabbitmq:
      password: bh-rabbitmq-password

# Per service env (example: bhcatalogworker)
bhcatalogworker:
  env:
    normal:
      RABBITMQ_USER: admin
      RABBITMQ_HOST: my-rabbitmq.my-namespace.svc.cluster.local
      RABBITMQ_PORT: "5672"
      RABBITMQ_PASSWORD: ""   # populated by ExternalSecret

Variable	In-cluster default	Existing broker
`RABBITMQ_HOST`	`bhrabbitmq.bh-control-plane.svc.cluster.local`	Your broker hostname or K8s Service FQDN
`RABBITMQ_PORT`	`5672`	Broker AMQP port
`RABBITMQ_USER`	`admin`	Your broker username
`RABBITMQ_PASSWORD`	From ESO → GSM	Same GSM secret as broker password

Apply the same RABBITMQ_* values on all components that use Celery: bhcatalogworker, bhauditworker, bhaiagentworker, bhcelerybeat, and related APIs.

3. Network access

Ensure GKE pods can reach the broker (VPC, firewall, or cluster DNS). For brokers outside the cluster, use private IP or internal load balancer hostnames allowed by your network policy.

4. Management UI (existing in-cluster broker)

If the broker is reachable in-cluster and you use Gateway API, enable the built-in HTTPRoute in bighammer and set backendRef to your Service name (and namespace if not bh-control-plane). Update bh-gateway ReferenceGrants and redeploy bh-gateway before bighammer when using a cross-namespace backend — see Cross-namespace broker.

For nginx mode, set global.ingress.rabbitmqHostName if the management UI should be exposed (Service must be in the same namespace as the Ingress backend unless you use a custom Ingress).

Disable the route if the UI is not needed:

global:
  gateway:
    routes:
      bhrabbitmq:
        enabled: false

Install order

Step	Action
1	ESO + Workload Identity
2	Create `bh-rabbitmq-password` in GSM
3	Gateway API edge: bh-gateway (update ReferenceGrants if broker ≠ app namespace)
4	Deploy broker — Option A, B, or validate Option C
5	Deploy `bighammer` — Platform Deployment

Gateway API with management UI

Order	Release	Notes
1	`bh-gateway`	Add broker namespace to `referenceGrants`; `crossNamespace` if HTTPRoute and Service differ
2	`bhrabbitmq`	Service must exist before HTTPRoute reconciles
3	`bighammer`	`global.gateway.routes.bhrabbitmq.enabled: true`

AMQP (port 5672) must be reachable before workers start. Install the broker before or alongside bighammer; management HTTPRoute can ship with the same bighammer upgrade once the backend Service is up.

Troubleshooting

Symptom	Check
`ExternalSecret` not ready	`SecretStore` `gcp-secret-store` exists; ESO WI binding
Auth failed	GSM password matches broker; same key in `global.remoteSecrets.rabbitmq`
Connection refused	`kubectl get svc bhrabbitmq -n bh-control-plane`; network policies
Workers idle	RabbitMQ pod running; `RABBITMQ_HOST` correct in worker env
Management UI 404 / no route	`global.gateway.routes.bhrabbitmq.enabled: true`; broker Service up; DNS set
HTTPRoute not accepted	Broker NS in `bh-gateway` `referenceGrants.routeNamespaces`
Cross-ns backend blocked	`referenceGrants.crossNamespace` in `bh-gateway`; redeploy `bh-gateway` before `bighammer`

Topic	Link
GSM secret names	Secret Manager Secrets
Celery / RabbitMQ overview	RabbitMQ & Celery
Mirror chart to GAR	Release Manifest & Distribution
Platform deploy	Platform Deployment
Gateway ReferenceGrants	bh-gateway

Choose an installation path​

Option A — BigHammer bhrabbitmq chart (OCI)​

1. Create the password in Secret Manager​

2. Authenticate to ECR​

3. Prepare GKE values overlay​

4. Install​

5. Verify​

Credential flow​

Management UI — Gateway API (HTTPRoute in bighammer)​

Deploy order (Gateway API + management UI)​

Management UI — NGINX Ingress​

Option B — Upstream Bitnami RabbitMQ chart​

Cross-namespace broker (Gateway API)​

1. Update bh-gateway ReferenceGrants​

2. Deploy broker in the target namespace​

3. Point bighammer HTTPRoute at the remote Service​

Option C — Existing RabbitMQ broker​

1. Secret Manager​

2. bighammer connection settings​

3. Network access​

4. Management UI (existing in-cluster broker)​

Install order​

Gateway API with management UI​

Troubleshooting​

Related​