RabbitMQ and Celery Workers
Overview of message queuing and background task processing in the platform.
Overview
The platform uses RabbitMQ as a message broker and Celery for distributed task execution. API services publish tasks to queues; Celery workers consume and process them asynchronously.
RabbitMQ
What It Does
RabbitMQ is the message broker that:
- Receives task messages from API services (Catalog, Audit, AI Agent)
- Stores messages in queues until workers are ready
- Delivers messages to Celery workers for processing
- Enables decoupled, scalable async processing
Usage
| Service | Role |
|---|---|
| Catalog API | Publishes catalog indexing and sync tasks |
| Audit API | Publishes audit event processing tasks |
| AI Agent | Publishes AI inference and pipeline tasks |
| Workers | Consume tasks from queues and execute them |
Setup
RabbitMQ is deployed as part of the platform. Key configuration:
| Setting | Description | Default |
|---|---|---|
| Host | Kubernetes service name | bhrabbitmq.bh-control-plane.svc.cluster.local |
| Port | AMQP port | 5672 |
| User | Admin username | admin |
| Password | From Key Vault secret bh-rabbitmq-password | — |
Accessing RabbitMQ UI
The RabbitMQ management UI is exposed via ingress. Typical host: rabbitmq.<env>.az.<domain> (e.g., rabbitmq.<env>.<your-domain>). Use the same admin credentials.
Celery Workers
What They Do
Celery workers run in the background and:
- Consume tasks from RabbitMQ queues
- Execute long-running or CPU-intensive work
- Keep API responses fast by offloading heavy operations
Worker Types
| Worker | Purpose |
|---|---|
| Audit Worker | Processes audit events and logs |
| Catalog Worker | Indexes data, syncs metadata, runs catalog jobs |
| AI Agent Worker | Runs AI inference, model execution, pipeline steps |
| Celery Beat | Schedules periodic tasks (cron-like) |
Environment Variables
Workers and APIs that use Celery share these settings:
| Variable | Description |
|---|---|
USE_CELERY | True to enable Celery; False to run synchronously |
RABBITMQ_HOST | RabbitMQ service host |
RABBITMQ_PORT | RabbitMQ port (5672) |
RABBITMQ_USER | RabbitMQ username |
RABBITMQ_PASSWORD | From Key Vault (bh-rabbitmq-password) |
CELERY_SCHEMA | PostgreSQL schema for Celery result backend (e.g., celery_db) |
CELERY_APP_NAME | Celery app name (e.g., bh_celery_worker) |
Celery Beat
Celery Beat is a scheduler that triggers periodic tasks. It runs as a separate deployment and uses the same RabbitMQ and database for coordination.
Key Vault Secret
Ensure bh-rabbitmq-password exists in Azure Key Vault. The install script can generate it. External Secrets Operator syncs it into Kubernetes for the API and worker pods.
Troubleshooting
| Issue | Check |
|---|---|
| Workers not processing | Verify RabbitMQ is running: kubectl get pods -n bh-control-plane -l app=bhrabbitmq |
| Connection refused | Confirm RABBITMQ_HOST and RABBITMQ_PORT match the RabbitMQ service |
| Auth failures | Ensure bh-rabbitmq-password is synced from Key Vault |
| Tasks stuck | Check worker logs: kubectl logs -n bh-control-plane -l app=bhcatalogworker -f |
References
- Key Vault Secrets — Required secrets including RabbitMQ password
- Platform Deployment — Deployment overview