POC Setup — BigHammer on K3s (EC2)

Use this path for a proof-of-concept (POC) or lab: run the bighammer umbrella chart on a single-node K3s cluster on AWS EC2, with in-cluster Postgres and RabbitMQ, Traefik ingress, and optional ALB + ACM in front.

For production EKS installs, use Platform Deployment instead.

Artifact sharing must be in place before Helm install

Container images and the Helm chart are pulled from Amazon ECR. A POC EC2/K3s host cannot install successfully until BigHammer has entitled your account and you have mirrored (or can pull) the chart and images.

Contact the BigHammer support team to obtain entitlement, the current release manifest, and the latest approved image tags / chart version for your POC.
Complete Release Manifest & Distribution (customer profile, reader role / ExternalId, mirror to your ECR — or the pull path BigHammer specifies for your POC).
Only then install K3s dependencies and run helm upgrade with those pinned tags in values-k3s-ec2.yaml.

Do not guess image tags from public registries. Tags that are not in your entitled ECR will fail with ImagePullBackOff (not found or 403).

Talk to BigHammer support for the latest images

Always ask BigHammer support for the latest chart version and container image tags before a POC install or upgrade. Image tags in sample values files can lag behind what is published for your entitlement. Support will confirm what to pin in your release manifest and values overlay.

When to use POC vs EKS

	POC (this page)	EKS platform
Cluster	Single-node K3s on EC2	Managed EKS
Database	In-cluster Postgres (`bh-postgres`)	Amazon RDS
Broker	In-cluster / chart RabbitMQ	Separate `bhrabbitmq` (or existing)
Ingress	Traefik (`className: traefik`)	ALB Ingress Controller
Values overlay	`values-k3s-ec2.yaml`	`values-release.yaml` → `values-<env>.yaml`
Hostnames	Distinct from EKS, e.g. `*-k3s.<your-domain>`	`*.<your-domain>`

Architecture (high level)

Browser → HTTPS :443 → ALB (ACM)  [optional but recommended]
                    → EC2 :80 Traefik
                    → Ingress Host rules
                    → Services / Pods in bh-control-plane

Secrets: AWS Secrets Manager → ESO (instance profile / IRSA) → K8s Secrets
Images:  Amazon ECR (refresh pull secret every ~12h)

Prerequisites

Item	Notes
Artifact sharing (required first)	Entitlement + mirror/pull from ECR — see warning above; Release Manifest & Distribution
Latest images / chart	Confirm with BigHammer support before install
EC2	Ubuntu 22.04/24.04, recommend ≥ 4 vCPU / 16 GiB RAM for the full suite
IAM instance profile	ECR pull (your mirrored registry or entitled path), Secrets Manager read, SSM (optional), S3 if logging
AWS Secrets Manager	Create names mapped in `global.remoteSecrets` — see Secrets Manager Secrets
Helm 3 + kubectl	On the node or via SSM
Chart	Pin the version from your release manifest (baseline docs: v1.0.12)

POC pods pull private ECR images. Establish sharing before chart install:

Step	Action
1	Contact BigHammer support — request POC entitlement, reader-role handoff, and the latest chart + image tag list
2	Complete customer onboarding / intake in Release Manifest & Distribution
3	Mirror entitled Helm OCI (`bighammer`, and `bhrabbitmq` if used) and container images into your ECR (recommended), or use the pull method support provides
4	Point `global.image.repository` (and related) in `values-k3s-ec2.yaml` at your ECR prefix
5	Pin every service `image.tag` to tags from the release manifest support gave you
6	On the K3s node, create/refresh the ECR dockerconfig pull secret (and CronJob) so kubelet can pull — see `agent-k3` guide §7

Until steps 1–5 are done, skip Helm install — you will only hit ImagePullBackOff.

EC2 ports (security group)

Port	Source	Purpose
80	ALB security groups (preferred) or your IP	Traefik HTTP
443	Optional	Direct Traefik TLS
22	Your IP only	SSH (optional if SSM-only)
6443	Admin / VPN only	K3s API — never public `0.0.0.0/0`

Do not expose Postgres 5432 or RabbitMQ 5672 / 15672 to the internet.

Chart and values

Artifact	Location
Umbrella chart	`bh-helm-charts/incubator/bighammer` (or mirrored ECR OCI)
POC values	`values-k3s-ec2.yaml` in the chart
Detailed build guide	`bh-helm-charts/incubator/agent-k3/K3S_EC2_DEPLOYMENT_GUIDE.md`
Day-2 ops	`bh-helm-charts/incubator/agent-k3/K3_RUNBOOK.md`
Install fixes	`bh-helm-charts/incubator/agent-k3/K3S_HELM_CHART_INSTALL_FIXES.md`

Canonical detail lives in agent-k3

This page is the platform index for POC. Step-by-step host prep, ESO, Postgres, RabbitMQ, ALB scripts, and troubleshooting are maintained under bh-helm-charts/incubator/agent-k3. Follow those guides and keep secrets/chart versions aligned with this docs site.

Deploy sequence (summary)

Artifact sharing — contact BigHammer support for latest images/chart; complete Release Manifest & Distribution; mirror to your ECR and pin tags in values (required before install).
Prepare EC2 + K3s — install K3s, Helm, AWS CLI (K3S_EC2_DEPLOYMENT_GUIDE.md §3).
ESO + SecretStore — sync from Secrets Manager (guide §4).
Create Secrets Manager secrets — Secrets Manager Secrets (map names in values-k3s-ec2.yaml global.remoteSecrets).
RabbitMQ + Postgres in-cluster — guide §5–6; grants in K3_RUNBOOK.md §3.
ECR image pull secret (+ CronJob refresher) for your registry — guide §7.
Helm install with POC values (images already mirrored / pullable):

# From chart source (example)
helm upgrade --install bighammer ./bighammer \
  --namespace bh-control-plane \
  --create-namespace \
  -f ./bighammer/values-k3s-ec2.yaml

Or pull the published OCI chart (v1.0.12), then apply your edited copy of values-k3s-ec2.yaml.

Ingress / ALB / DNS — Traefik on the node; optional ALB host rules for *-k3s.<your-domain> only (guide §9). Do not reuse production EKS hostnames.
Keycloak HTTPS — set KC_HOSTNAME to the full https://… URL and KC_PROXY_HEADERS: xforwarded when TLS terminates on the ALB (guide §9.6).

POC values highlights (`values-k3s-ec2.yaml`)

Setting	POC expectation
`ingress.className`	`traefik`
`ingress.annotations`	`{}` (no ALB Ingress Controller annotations)
`ingress.backendPort`	`80`
`ingress.*HostName`	Distinct `*-k3s.<your-domain>` names
In-cluster DNS	`http://<service>.bh-control-plane.svc.cluster.local`
Autoscaling	Typically disabled / single replica on small hosts

Verify

kubectl -n bh-control-plane get pods
kubectl -n bh-control-plane get ingress
kubectl -n bh-control-plane get externalsecret

# After ALB + DNS
curl -sk -o /dev/null -w '%{http_code}\n' https://ui-k3s.<your-domain>/
curl -sk https://kc-k3s.<your-domain>/admin/master/console/config
# auth-server-url should be https://kc-k3s.<your-domain>/

Troubleshooting

Symptom → cause → fix tables: bh-helm-charts/incubator/agent-k3/K3S_HELM_CHART_INSTALL_FIXES.md

Common POC issues:

Symptom	First check
`CreateContainerConfigError`	Missing Secrets Manager name — secrets list
`ImagePullBackOff` `403`	Refresh ECR dockerconfig secret + CronJob; confirm instance can authenticate to your ECR
`ImagePullBackOff` `not found`	Tag not in entitled/mirrored ECR — ask BigHammer support for the latest tags; update values
Keycloak admin `somethingWentWrong`	`KC_HOSTNAME` must be `https://…` behind ALB
ALB `503` / empty Endpoints	Pods not Ready; fix secrets/probes first

Topic	Link
EKS production deploy	Platform Deployment
Secrets checklist	Secrets Manager Secrets
Chart release (OCI)	Helm Releases — v1.0.12
Mirror charts/images	Release Manifest & Distribution
RabbitMQ (EKS-style separate chart)	RabbitMQ Broker

When to use POC vs EKS​

Architecture (high level)​

Prerequisites​

Artifact sharing before POC install​

EC2 ports (security group)​

Chart and values​

Deploy sequence (summary)​

POC values highlights (values-k3s-ec2.yaml)​

Verify​

Troubleshooting​

Related platform docs​