Database Bootstrap Troubleshooting

Guidance for PostgreSQL application database setup after infrastructure provisioning and before or after Helm deployment.

Bootstrap script: create-dbs-azure.sh (deployment repository) or equivalent SQL from sql/grant_application_schemas.sql.

Related: Infrastructure — PostgreSQL, Key Vault Secrets, Scripts Reference — Database creation

Overview

Infrastructure provisioning creates:

• PostgreSQL Flexible Server and application database (e.g. bighammer_db_<env>)
• Database users: admin, bh_dev_user, bh_app_user, keycloak_db_user
• Key Vault passwords: azure-postgres-master-password, bh-app-user-password, bh-kcdbpwd

Provisioning does not always create application schemas or grants. Run create-dbs-azure.sh as the admin user on the application database.

Recommended order of operations

Infrastructure (instance, database, users, Key Vault passwords)
    → create-dbs-azure.sh (schemas and grants on application database)
    → Helm deploy / pod startup

Running the bootstrap script before app deployment avoids schema ownership and permission errors.

If Helm deploys before bootstrap, pods may create schemas as the wrong user. Recovery steps are in Recovery — wrong schema ownership below.

Prerequisites

Verify before running bootstrap or debugging pod errors:

Check	Expected
Application database	Exists; name matches POSTGRES_DB in Helm values
Users	bh_dev_user, bh_app_user exist
Key Vault passwords	azure-postgres-master-password, bh-app-user-password match PostgreSQL users
Helm migration user	POSTGRES_USER: bh_dev_user
Helm runtime user	BH_APP_USER: bh_app_user
remoteSecrets.postgres.password	azure-postgres-master-password or bh-dev-user-password per your overlay
Bootstrap executed	Full schema grants on application database
Bootstrap identity	Script run as PostgreSQL admin with DDL privileges

Credential model (dual-role)

K8s env vars	DB user	Key Vault secret	Purpose
POSTGRES_USER / POSTGRES_PASSWORD	bh_dev_user	azure-postgres-master-password or bh-dev-user-password	Alembic, migrations, DDL
BH_APP_USER / BH_APP_USER_PASSWORD	bh_app_user	bh-app-user-password	Runtime SQLAlchemy queries (DML)

Keycloak server DB (separate): keycloak_db_user / bh-kcdbpwd on keycloak_db_<env>.

Running create-dbs-azure.sh

When the database and users already exist (typical IaC flow):

1. Connect as admin to the application database.
2. Ensure TARGET_DB in config matches POSTGRES_DB in Helm values.
3. Run ./create-dbs-azure.sh — idempotent when AUTO_SKIP_EXISTING_DATABASES is enabled.
4. Confirm CREATE EXTENSION IF NOT EXISTS vector succeeds on catalog_db (requires admin privileges).

Common errors

Error / symptom	What to check	Fix
password authentication failed for user "bh_dev_user"	POSTGRES_PASSWORD ExternalSecret; Key Vault vs PostgreSQL	Align secret with DB user; refresh ExternalSecret; redeploy
password authentication failed for user "bh_app_user"	BH_APP_USER_PASSWORD; Key Vault bh-app-user-password	Align secret with DB user
permission denied for schema <name> (Alembic / bh_dev_user)	Schema owner; POSTGRES_USER in values	Owner should be bh_dev_user. If bh_app_user owns schemas, see Recovery
permission denied for schema catalog_db (runtime / bh_app_user)	has_schema_privilege('bh_app_user', 'catalog_db', 'USAGE')	Re-run grant SQL as bh_dev_user
permission denied for schema celery_db at pod startup	Grants on celery_db	Re-run full bootstrap grants for all six schemas
Cross-schema query fails	App user USAGE on every schema the service queries	Grant USAGE on all required schemas
Bootstrap hangs	Active pod DB connections	Scale affected deployments to 0; retry
CREATE EXTENSION vector fails	Extension allow-list on Flexible Server	Enable vector on server; run as admin

Diagnostic queries

Run on the application database.

Session:

SELECT current_user, current_database();

Schema ownership (expect bh_dev_user for all application schemas):

SELECT n.nspname AS schema, r.rolname AS owner
FROM pg_namespace n
JOIN pg_roles r ON r.oid = n.nspowner
WHERE n.nspname IN (
  'catalog_db', 'agent_db', 'audit_db',
  'celery_db', 'keycloakapi_db', 'llm_observability_db'
);

Recovery — wrong schema ownership

If pods created schemas as bh_app_user before bootstrap:

1. Scale platform deployments to 0.
2. As admin, reassign ownership to bh_dev_user:

REASSIGN OWNED BY bh_app_user TO bh_dev_user;

3. Re-run grant sections from sql/grant_application_schemas.sql as bh_dev_user.
4. Scale deployments back up.

Post-fix verification

• Diagnostic queries: all schemas owned by bh_dev_user; app user has USAGE on required schemas.
• Pod logs: schema setup and alembic upgrade head complete without permission denied.
• Restart affected deployments after grant changes.

Platform-managed Key Vault secrets

Infrastructure provisioning creates prerequisite DB and SSL secrets. Do not recreate them manually — see Platform prerequisites.

Integration and license secrets: see install scripts in Scripts Reference.

Related

Topic	Link
Scripts	Scripts Reference
Secrets	Key Vault Secrets
GCP equivalent	GCP Database Bootstrap Troubleshooting
General troubleshooting	Troubleshooting