Skip to main content

v1.0.13 Release

Release Tag: v1.0.13-bighammer-aws
Chart Version: 1.0.13
Base Release: v1.0.12-bighammer-aws

BigHammer Helm Chart v1.0.13 introduces native Keyless IAM Role Authentication across all microservices on both Amazon EKS (IRSA) and AWS EC2 (IAM Instance Profiles / K3s / Standalone Docker), enterprise External Secrets Operator (ESO) integration with AWS Secrets Manager, and dynamic in-pod cloud license resolution.

This release completely eliminates static AWS Access Keys (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) from environment variables and Kubernetes Secret manifests, establishing an enterprise-grade least-privilege security posture.

Published Chart & Download Instructions​

Customers entitled to BigHammer on AWS can pull the published chart using any of the following methods.

Prerequisites​

  • Kubernetes: Amazon EKS 1.28+ or Standalone K3s on AWS EC2
  • Helm: v3.10+ (v3.17+ recommended)
  • IAM Authentication:
    • For EKS: AWS IAM OIDC Provider enabled on the EKS cluster.
    • For EC2: AWS IAM Instance Profile attached to the EC2 host with IMDSv2 Hop Limit set to 2+.
  • Secrets Management: External Secrets Operator (ESO) v0.18.x+ with ambient IRSA / Instance Profile → AWS Secrets Manager
  • Ingress Controller: AWS Load Balancer Controller (ALB Ingress Class alb) on EKS, or Traefik on K3s
  • Message Broker: RabbitMQ deployed via dedicated Helm release — RabbitMQ Broker
  • Database: Amazon RDS PostgreSQL with pgvector (or in-cluster PostgreSQL for standalone EC2 POCs)

In production, pull the chart from your mirrored Amazon ECR (see Release Manifest & Distribution):

aws ecr get-login-password --region <your-aws-region> | \
  helm registry login --username AWS --password-stdin \
  <your-aws-account-id>.dkr.ecr.<your-aws-region>.amazonaws.com

helm pull oci://<your-aws-account-id>.dkr.ecr.<your-aws-region>.amazonaws.com/bh-helm-release/bighammer \
  --version 1.0.13

tar -xzf bighammer-1.0.13.tgz
cd bighammer

Option 2: Pull Directly from BigHammer ECR Catalog​

If your CI/CD runner is authorized to assume the BigHammer reader role (bh-artifact-reader-<customer-id>):

aws ecr get-login-password --region <bh-catalog-aws-region> | \
  helm registry login --username AWS --password-stdin \
  <bh-aws-account-id>.dkr.ecr.<bh-catalog-aws-region>.amazonaws.com

helm pull oci://<bh-aws-account-id>.dkr.ecr.<bh-catalog-aws-region>.amazonaws.com/aws/bh-helm-release/bighammer \
  --version 1.0.13

tar -xzf bighammer-1.0.13.tgz
cd bighammer

Option 3: Source Checkout via Git​

git clone --depth 1 https://github.com/bh-ai/bh-helm-charts.git
cd bh-helm-charts/incubator/bighammer
git checkout v1.0.13-bighammer-aws
helm dependency build .

Release Summary & Key Capabilities​

1. Keyless Authentication Across EKS & EC2​

  • Amazon EKS (IRSA): Microservice pods authenticate via short-lived OIDC Web Identity Tokens projected automatically into pods (sts:AssumeRoleWithWebIdentity).
  • AWS EC2 (Instance Profile): Pods running in K3s or Docker containers on EC2 automatically inherit ambient temporary credentials from the EC2 Instance Metadata Service (IMDSv2) via the attached IAM Instance Profile.
  • Zero Static Keys: Neither deployment model requires static AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY in environment variables or Secret objects.

2. External Secrets Operator (ESO) & AWS Secrets Manager Direct Sync​

  • Standardized SecretStore (aws-secret-store) automatically resolves credentials:
    • On EKS: Via ServiceAccount JWT tokens (spec.provider.aws.auth.jwt.serviceAccountRef).
    • On EC2: Via ambient Instance Profile credentials (omitting the auth block in SecretStore).
  • All application secrets (database credentials, RabbitMQ passwords, JWT keys, Keycloak configs) are dynamically synced from AWS Secrets Manager into Kubernetes secrets.

3. Dynamic Cloud License Resolution (bh-catalog-api)​

  • BigHammer platform license and RSA verification public key are dynamically fetched from AWS Secrets Manager at container initialization (USE_CLOUD_SECRETS_FOR_LICENSE: true).
  • Verifies license signatures without mounting local secret files or baking licenses into containers.

4. Resilient CloudFactoryService Ambient Fallback​

  • In-pod cloud SDK operations (such as migration credential persistence in bh-migrate-api and LLM configurations in bh-ai-agent) seamlessly fallback to ambient pod_role credentials if static keys are omitted.
  • Prevents runtime credential exceptions during project creation and dynamic tenant execution.

5. Multi-Region Compatibility​

  • The entire Helm deployment and underlying Python microservice suite is region-agnostic.
  • Setting AWS_REGION in your values file dynamically configures Secrets Manager, STS, and Bedrock routing to the target region.

6. Full Backward Compatibility​

  • Existing clusters that still require static IAM access keys can retain them by setting: 
    global:
      remoteSecrets:
        app:
          awsCredentials:
            enabled: true
    This preserves zero-downtime upgrades for both keyless and legacy infrastructure.

Keyless Authentication on AWS EC2 (Instance Profile & K3s)​

For environments running standalone K3s or Docker containers on an AWS EC2 instance (e.g., single-node POCs or dedicated VM deployments), BigHammer supports keyless authentication via the EC2 Instance Profile.

How to Configure EC2 Keyless Authentication​

Step 1: Create and Attach the IAM Instance Profile to the EC2 Instance​

  1. Create an IAM Role with trust policy allowing EC2: 
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "ec2.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
  2. Attach an IAM Policy granting access to BigHammer Secrets Manager secrets: 
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "secretsmanager:GetSecretValue",
            "secretsmanager:DescribeSecret"
          ],
          "Resource": "arn:aws:secretsmanager:<AWS_REGION>:<AWS_ACCOUNT_ID>:secret:bh/*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "kms:Decrypt"
          ],
          "Resource": "*"
        }
      ]
    }
  3. Attach this IAM Role to your EC2 instance as an Instance Profile.

Step 2: Configure EC2 Instance Metadata Service (IMDSv2 Hop Limit)​

By default, AWS EC2 IMDSv2 sets the HTTP response hop limit to 1. Because container runtimes (such as Docker or K3s Flannel) operate on an internal bridge network, packets originating from inside a container are at hop count 2.

To allow containers to reach the metadata service, increase the hop limit to 2:

aws ec2 modify-instance-metadata-options \
  --instance-id <YOUR_EC2_INSTANCE_ID> \
  --http-put-response-hop-limit 2 \
  --http-endpoint enabled

Step 3: Use the Dedicated EC2 Values File​

Deploy using values-k3s-ec2.yaml. In this overlay:

  • secretStore.accessKeyIDSecretRef.name: "" and secretAccessKeySecretRef.name: "" are left empty, which instructs the External Secrets Operator to inherit the host's EC2 Instance Profile.
  • Service accounts do not specify eks.amazonaws.com/role-arn annotations (as there is no EKS OIDC provider on EC2).
  • Pods natively query http://169.254.169.254/latest/meta-data/ to obtain short-lived AWS credentials.

EC2 Keyless Authentication: Potential Issues & Solutions​

Issue / SymptomRoot CauseSolution
NoCredentialsError / Unable to locate credentials inside pods (Host shell works, but pods fail)IMDS Hop Limit is set to 1. The IP TTL expires when metadata responses transit the container bridge network.Run aws ec2 modify-instance-metadata-options --instance-id <ID> --http-put-response-hop-limit 2. Verify connectivity inside a test pod: curl http://169.254.169.254/latest/meta-data/.
SecretStore reports ValidationFailed or cannot authenticatesecretStore in Helm has lingering static secret references or invalid JWT config.In values-k3s-ec2.yaml, leave accessKeyIDSecretRef and secretAccessKeySecretRef empty. ESO will automatically fall back to the EC2 Instance Profile without an explicit auth block.
Connection Timeout to AWS Secrets Manager (EndpointConnectionError)EC2 instance is located in a private subnet without internet access or VPC Endpoints.Ensure the subnet routes through a NAT Gateway, or configure an AWS VPC Interface Endpoint for com.amazonaws.<region>.secretsmanager.
AccessDenied on kms:DecryptSecrets in Secrets Manager are encrypted with a Customer Managed Key (CMK), but the EC2 IAM Role lacks KMS permissions.Add kms:Decrypt for the CMK ARN to the EC2 Instance Profile IAM Policy.
ClientError: 401 Unauthorized on IMDS token retrievalIMDSv2 enforcement enabled without session token support.BigHammer microservices use boto3 which natively handles IMDSv2 token acquisition (PUT /latest/api/token). Ensure hop limit is set to 2 so token requests succeed.

Comparison: Amazon EKS vs AWS EC2​

DimensionAmazon EKS DeploymentAWS EC2 (K3s / Standalone)
Authentication MechanismIAM Roles for Service Accounts (IRSA)IAM Instance Profile (IMDSv2)
Credential ScopePod-level granularity (per ServiceAccount)Host-level granularity (shared across pods)
OIDC Provider Required?Yes (Cluster OIDC issuer)No (Direct EC2 metadata)
Required IMDS Hop LimitN/A (EKS injects web identity token)http-put-response-hop-limit: 2
Helm Values Overlayvalues-release.yaml / values-dev.yamlvalues-k3s-ec2.yaml
ServiceAccount Role AnnotationRequired (eks.amazonaws.com/role-arn)Omitted (Ambient host credentials)
Ingress ControllerAWS Load Balancer Controller (alb)Traefik / NodePort

Upgrading from v1.0.12 to v1.0.13​

Scenario A: Upgrading on Amazon EKS (IRSA)​

  1. Update IAM Role Trust Relationship: Ensure the IAM role trusts your EKS cluster's OIDC provider using StringLike evaluation: 
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>"
          },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": {
              "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>:aud": "sts.amazonaws.com"
            },
            "StringLike": {
              "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>:sub": [
                "system:serviceaccount:bh-control-plane:*",
                "system:serviceaccount:default:external-secrets"
              ]
            }
          }
        }
      ]
    }
  2. Deploy Helm Upgrade: 
    helm upgrade bighammer ./bighammer \
      --namespace bh-control-plane \
      -f values-<env>.yaml

Scenario B: Upgrading on AWS EC2 (Standalone K3s)​

  1. Verify IMDS Hop Limit: 
    aws ec2 modify-instance-metadata-options \
      --instance-id <YOUR_EC2_INSTANCE_ID> \
      --http-put-response-hop-limit 2
  2. Verify EC2 Instance Profile: Ensure the instance profile role has permissions to read bh/* secrets from AWS Secrets Manager.
  3. Deploy Helm Upgrade: 
    helm upgrade bighammer ./bighammer \
      --namespace bh-control-plane \
      -f values-k3s-ec2.yaml

Post-Upgrade Verification (Both EKS & EC2)​

  1. Check External Secrets Sync:

    kubectl get secretstore,es -n bh-control-plane

    Expected Status: SecretSynced: True for all 12 ExternalSecret resources.

  2. Verify Microservice Health:

    kubectl get pods -n bh-control-plane

    All microservice pods should be Running and passing readiness probes.

GuideDescription
IAM Role Authentication & ESO Setup GuideIn-depth architectural guide on IRSA, EC2 IMDS, and trust configurations
K3s Standalone POC Setup on EC2Step-by-step guide to installing BigHammer on an EC2 instance
Secrets Manager Secrets ChecklistList of all required secret keys and formats
AWS Platform DeploymentComplete end-to-end installation walkthrough on Amazon EKS
v1.0.12 Release NotesPrevious AWS baseline release documentation