RabbitMQ Broker

BigHammer workers and APIs use RabbitMQ for Celery task queues. The broker is not part of the bighammer umbrella chart — deploy it as a separate Helm release (or point apps at an existing broker).

Prerequisites

External Secrets Operator (ESO) installed on the cluster
IRSA (or equivalent) so ESO can read AWS Secrets Manager
Secrets Manager secret for the broker password (see Option A)
SecretStore aws-secret-store in bh-control-plane (created by the bighammer platform chart or your bootstrap)
Entitled chart and custom RabbitMQ image mirrored per Release Manifest & Distribution

Choose an installation path

Option	When to use
A — BigHammer `bhrabbitmq` chart (recommended)	Standard BigHammer on EKS; chart published to ECR OCI
B — Upstream Bitnami chart	You manage Bitnami RabbitMQ yourself; align service name and secrets with `bighammer`
C — Existing broker	Corporate or shared RabbitMQ already running; configure `bighammer` workers only

All paths must expose AMQP on port 5672 and use a password synced from Secrets Manager (unless you override global.remoteSecrets.rabbitmq).

Helm versions

Chart version	App version	Custom image tag	Status	Notes
16.0.18	4.3.6	`4.3.6-2-rc`	Current	Initial AWS catalog release — release notes

Pin the chart version from your release manifest. Do not use floating tags.

Option A — BigHammer `bhrabbitmq` chart (OCI)

BigHammer publishes the bhrabbitmq chart to AWS ECR. Mirror to your ECR per Release Manifest & Distribution.

Item	Value
Release name	`bhrabbitmq` (fixed — matches `fullnameOverride`)
Namespace	`bh-control-plane`
Service DNS	`bhrabbitmq.bh-control-plane.svc.cluster.local:5672`
OCI chart (BH source)	`oci://<bh-aws-account-id>.dkr.ecr.<bh-catalog-aws-region>.amazonaws.com/aws/bh-helm-release/bhrabbitmq`
Your ECR OCI	`oci://<your-aws-account-id>.dkr.ecr.<your-aws-region>.amazonaws.com/bh-helm-release/bhrabbitmq`
Chart version	Pin from your release manifest (current: 16.0.18)

Mirror the chart and the custom image

This chart uses a BigHammer custom RabbitMQ image, not the public Bitnami Docker Hub image.

After you download/mirror the Helm chart via the artifact sharing process, you must also mirror the Docker image referenced by the chart (see Custom image below). Installing the chart alone is not enough — pods will fail to pull if the image is missing from your registry.

Custom image

Item	BH catalog (mirror from)
Image	`<bh-aws-account-id>.dkr.ecr.<bh-catalog-aws-region>.amazonaws.com/bh-docker-public/bh-rabbitmq:<image-tag>`
Tag for chart 16.0.18	`4.3.6-2-rc`
Full URI (16.0.18)	`<bh-aws-account-id>.dkr.ecr.<bh-catalog-aws-region>.amazonaws.com/bh-docker-public/bh-rabbitmq:4.3.6-2-rc`

Mirror into your ECR (example destination):

<your-aws-account-id>.dkr.ecr.<your-aws-region>.amazonaws.com/bh-docker-public/bh-rabbitmq:4.3.6-2-rc

Then set image.registry / image.repository / image.tag in your values overlay to your mirrored ECR (or rely on chart defaults after retagging to the same path layout).

1. Create the password in Secrets Manager

Create a Secrets Manager secret that ESO can sync. The AWS chart overlay expects:

Remote secret name	JSON key	Kubernetes secret
`rabbitmq-credentials`	`rabbitmq-password`	`bhrabbitmq-credentials`

aws secretsmanager create-secret \
  --name rabbitmq-credentials \
  --secret-string '{"rabbitmq-password":"your-secure-password"}' \
  --region <your-aws-region>

Use the same password for bighammer worker/API remote secrets (global.remoteSecrets.rabbitmq). See Secrets Manager Secrets.

2. Authenticate and pull the chart

After assuming the BH reader role (or using your mirrored ECR):

export ECR_REGISTRY=<bh-aws-account-id>.dkr.ecr.<bh-catalog-aws-region>.amazonaws.com
export ECR_OCI_CHART=oci://${ECR_REGISTRY}/aws/bh-helm-release/bhrabbitmq
export CHART_VERSION=16.0.18

aws ecr get-login-password --region <bh-catalog-aws-region> | \
  helm registry login --username AWS --password-stdin "${ECR_REGISTRY}"

helm pull "${ECR_OCI_CHART}" --version "${CHART_VERSION}"
tar -xzf "bhrabbitmq-${CHART_VERSION}.tgz"

From your mirrored ECR (recommended in production):

export MY_ECR=<your-aws-account-id>.dkr.ecr.<your-aws-region>.amazonaws.com
export CHART_VERSION=16.0.18

aws ecr get-login-password --region <your-aws-region> | \
  helm registry login --username AWS --password-stdin "${MY_ECR}"

helm pull "oci://${MY_ECR}/bh-helm-release/bhrabbitmq" --version "${CHART_VERSION}"
tar -xzf "bhrabbitmq-${CHART_VERSION}.tgz"

3. Mirror the custom Docker image

BH_REG=<bh-aws-account-id>.dkr.ecr.<bh-catalog-aws-region>.amazonaws.com
MY_REG=<your-aws-account-id>.dkr.ecr.<your-aws-region>.amazonaws.com
IMG=bh-docker-public/bh-rabbitmq
TAG=4.3.6-2-rc   # pin from release notes / chart values for this chart version

# Login to BH ECR (reader role) and your ECR (your account)
aws ecr get-login-password --region <bh-catalog-aws-region> | \
  docker login --username AWS --password-stdin "${BH_REG}"
aws ecr get-login-password --region <your-aws-region> | \
  docker login --username AWS --password-stdin "${MY_REG}"

docker pull "${BH_REG}/${IMG}:${TAG}"
docker tag  "${BH_REG}/${IMG}:${TAG}" "${MY_REG}/${IMG}:${TAG}"
docker push "${MY_REG}/${IMG}:${TAG}"

Confirm the image tag inside the downloaded chart (values.yaml / Chart.yaml annotations) matches what you mirrored.

4. Prepare EKS values overlay

Create values-aws-rabbitmq.yaml (chart also ships values-aws.yaml defaults):

global:
  cloudProvider: aws
  storageClass: gp3
  secretStore:
    kind: SecretStore
    name: aws-secret-store
  # After mirror — point at your ECR:
  # imageRegistry: "<your-aws-account-id>.dkr.ecr.<your-aws-region>.amazonaws.com"

externalSecrets:
  enabled: true
  passwordRemoteKey: rabbitmq-credentials
  passwordRemoteProperty: rabbitmq-password
  k8sSecretName: bhrabbitmq-credentials

image:
  registry: <your-aws-account-id>.dkr.ecr.<your-aws-region>.amazonaws.com
  repository: bh-docker-public/bh-rabbitmq
  tag: "4.3.6-2-rc"

auth:
  existingPasswordSecret: bhrabbitmq-credentials
  existingSecretPasswordKey: rabbitmq-password

storageClass:
  enabled: true
  name: gp3
  provisioner: ebs.csi.aws.com
  parameters:
    type: gp3
    iops: "3000"
    throughput: "125"
    encrypted: "true"

persistence:
  storageClass: gp3

replicaCount: 1
ingress:
  enabled: false

5. Install

helm upgrade --install bhrabbitmq ./bhrabbitmq \
  -f bhrabbitmq/values.yaml \
  -f values-aws-rabbitmq.yaml \
  -n bh-control-plane \
  --create-namespace

Or install from your mirrored OCI URI:

helm upgrade --install bhrabbitmq "oci://${MY_ECR}/bh-helm-release/bhrabbitmq" \
  --version "${CHART_VERSION}" \
  -f values-aws-rabbitmq.yaml \
  -n bh-control-plane \
  --create-namespace

Upgrades

Bitnami RabbitMQ may require the current password on upgrade even when using existingPasswordSecret. Export it from the synced secret and pass --set auth.password=... if the upgrade fails password validation.

6. Verify

kubectl get externalsecret,secret bhrabbitmq-credentials -n bh-control-plane
kubectl get svc,statefulset bhrabbitmq -n bh-control-plane

kubectl run -it --rm nettest --image=busybox:1.36 --restart=Never -n bh-control-plane -- \
  nc -zv bhrabbitmq.bh-control-plane.svc.cluster.local 5672

Credential flow

Broker: ESO syncs Secrets Manager → bhrabbitmq-credentials → RabbitMQ auth.existingPasswordSecret
Apps: bighammer ExternalSecrets read the same password into per-app secrets as RABBITMQ_PASSWORD

# bighammer values
global:
  remoteSecrets:
    rabbitmq:
      password: rabbitmq-credentials   # or your mapped remote key name

Management UI — ALB / Ingress

Keep chart ingress.enabled: false unless you intentionally expose management from the bhrabbitmq chart. Prefer routing from the bighammer platform ingress (ALB) when enabled — set the RabbitMQ hostname in your platform values (see Platform Deployment).

Management port: 15672. AMQP remains in-cluster on 5672.

Option B — Upstream Bitnami RabbitMQ chart

Use the official Bitnami RabbitMQ Helm chart if your team already standardizes on Bitnami installs.

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

helm upgrade --install bhrabbitmq bitnami/rabbitmq \
  --namespace bh-control-plane \
  --create-namespace \
  --set fullnameOverride=bhrabbitmq \
  --set auth.username=admin \
  --set auth.existingPasswordSecret=bhrabbitmq-credentials \
  --set auth.existingSecretPasswordKey=rabbitmq-password

You must still:

Create the password in Secrets Manager and sync to bhrabbitmq-credentials
Set fullnameOverride: bhrabbitmq so Service DNS matches bighammer defaults
Configure global.remoteSecrets.rabbitmq in bighammer to the same secret

Prefer Option A (BigHammer bhrabbitmq) unless you have a strong reason to run raw Bitnami — Option A includes AWS SecretStore integration, EBS gp3 defaults, and the supported custom image.

Option C — Existing RabbitMQ broker

If a RabbitMQ cluster already exists (in-cluster or reachable from EKS), skip the bhrabbitmq Helm release and configure bighammer worker/API env vars.

1. Secrets Manager

Store the broker password under the name your chart expects, then wire global.remoteSecrets.rabbitmq.

2. `bighammer` connection settings

global:
  remoteSecrets:
    rabbitmq:
      password: rabbitmq-credentials

bhcatalogworker:
  env:
    normal:
      RABBITMQ_USER: admin
      RABBITMQ_HOST: my-rabbitmq.my-namespace.svc.cluster.local
      RABBITMQ_PORT: "5672"
      RABBITMQ_PASSWORD: ""   # populated by ExternalSecret

Variable	In-cluster default	Existing broker
`RABBITMQ_HOST`	`bhrabbitmq.bh-control-plane.svc.cluster.local`	Your broker hostname or K8s Service FQDN
`RABBITMQ_PORT`	`5672`	Broker AMQP port
`RABBITMQ_USER`	`admin`	Your broker username
`RABBITMQ_PASSWORD`	From ESO → Secrets Manager	Same secret as broker password

Apply the same RABBITMQ_* values on all Celery components (bhcatalogworker, bhauditworker, bhaiagentworker, bhcelerybeat, and related APIs).

3. Network access

Ensure EKS pods can reach the broker (VPC, security groups, network policies). For brokers outside the cluster, use private IP or internal NLB/ALB hostnames allowed by your network policy.

Install order

Step	Action
1	ESO + IRSA + `SecretStore` `aws-secret-store`
2	Create broker password in Secrets Manager
3	Mirror `bhrabbitmq` chart and custom `bh-rabbitmq` image — Release Manifest & Distribution
4	Deploy broker — Option A, B, or validate Option C
5	Deploy `bighammer` — Platform Deployment

AMQP (port 5672) must be reachable before workers start. Install the broker before or alongside bighammer.

Troubleshooting

Symptom	Check
`ExternalSecret` not ready	`SecretStore` `aws-secret-store` exists; ESO IRSA can read Secrets Manager
`ImagePullBackOff`	Custom image mirrored to your ECR; `image.registry` / `tag` match release notes
Auth failed	Secrets Manager password matches broker; same key in `global.remoteSecrets.rabbitmq`
Connection refused	`kubectl get svc bhrabbitmq -n bh-control-plane`; security groups / network policies
Workers idle	RabbitMQ pod running; `RABBITMQ_HOST` correct in worker env
Upgrade password error	Pass current password via `--set auth.password=...` on `helm upgrade`

Helm chart release notes — 16.0.18

Initial BigHammer bhrabbitmq release published to the AWS catalog path.

Published artifacts

Artifact	URI
Helm chart (OCI)	`oci://<bh-aws-account-id>.dkr.ecr.<bh-catalog-aws-region>.amazonaws.com/aws/bh-helm-release/bhrabbitmq:16.0.18`
Custom RabbitMQ image	`<bh-aws-account-id>.dkr.ecr.<bh-catalog-aws-region>.amazonaws.com/bh-docker-public/bh-rabbitmq:4.3.6-2-rc`

Field	Value
Chart version	16.0.18
App version	4.3.6
Chart name	`bhrabbitmq`
Catalog path	`aws/bh-helm-release/bhrabbitmq`
Minimum	EKS 1.28+, Helm 3.10+, ESO 0.18.x, EBS CSI driver

Added

AWS-targeted bhrabbitmq chart in the shared BH ECR catalog
BigHammer custom image bh-docker-public/bh-rabbitmq:4.3.6-2-rc (not Bitnami Docker Hub)
AWS Secrets Manager + ESO integration via SecretStore aws-secret-store
EBS CSI gp3 storage class defaults (values-aws.yaml)
Celery-compatible RabbitMQ 4.x setting: deprecated_features.permit.transient_nonexcl_queues = true

Customer actions

Entitlement must include Helm chart bhrabbitmq and the custom image repository (bh-docker-public/bh-rabbitmq or as listed in your handoff).
Mirror both chart 16.0.18 and image tag 4.3.6-2-rc using Release Manifest & Distribution.
Install with an AWS values overlay (see Option A).
Pin versions in your release manifest — do not float tags.

Pull commands (after BH reader role or from your mirror)

# Chart
helm pull oci://<bh-aws-account-id>.dkr.ecr.<bh-catalog-aws-region>.amazonaws.com/aws/bh-helm-release/bhrabbitmq \
  --version 16.0.18

# Custom image (required)
docker pull <bh-aws-account-id>.dkr.ecr.<bh-catalog-aws-region>.amazonaws.com/bh-docker-public/bh-rabbitmq:4.3.6-2-rc

Topic	Link
Secrets reference	Secrets Manager Secrets
Celery / RabbitMQ overview	RabbitMQ & Celery
Mirror chart + image	Release Manifest & Distribution
Platform deploy	Platform Deployment
GCP equivalent	GCP RabbitMQ Broker
Azure equivalent	Azure RabbitMQ Broker

Choose an installation path​

Helm versions​

Option A — BigHammer bhrabbitmq chart (OCI)​

Custom image​

1. Create the password in Secrets Manager​

2. Authenticate and pull the chart​

3. Mirror the custom Docker image​

4. Prepare EKS values overlay​

5. Install​

6. Verify​

Credential flow​

Management UI — ALB / Ingress​

Option B — Upstream Bitnami RabbitMQ chart​

Option C — Existing RabbitMQ broker​

1. Secrets Manager​

2. bighammer connection settings​

3. Network access​

Install order​

Troubleshooting​

Helm chart release notes — 16.0.18​

Published artifacts​

Added​

Customer actions​

Pull commands (after BH reader role or from your mirror)​

Related​