Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.emergence.ai/llms.txt

Use this file to discover all available pages before exploring further.

​
Upgrades

This guide covers the em-runtime Helm chart upgrade process, version promotion through environments, migration procedures, and rollback strategies.

​
Upgrade Procedures

​
Standard Helm Upgrade

For direct Helm deployments (not using ArgoCD): 
# Check current version
helm list -n $NAMESPACE

# Review what will change
helm diff upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
  --version $NEW_VERSION \
  --namespace $NAMESPACE \
  -f values.yaml

# Perform the upgrade
helm upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
  --version $NEW_VERSION \
  --namespace $NAMESPACE \
  -f values.yaml

​
Pre-Upgrade Checklist

Always back up PostgreSQL, Redis, and the em-runtime-secrets Secret before upgrading.
Before upgrading:
  1. Review the changelog for breaking changes and required migrations
  2. Back up databases: 
    # PostgreSQL
pg_dumpall -h $PG_HOST -U postgres > backup-pre-upgrade.sql

# Kubernetes Secrets
kubectl get secret em-runtime-secrets -n $NAMESPACE -o yaml > em-runtime-secrets-backup.yaml
kubectl get secret infisical-bootstrap-secret -n $NAMESPACE -o yaml > infisical-bootstrap-backup.yaml
  3. Verify current health: 
    kubectl get pods -n $NAMESPACE
curl https://$HOSTNAME/api/governance/health
  4. Test in non-production first — promote through dev and staging before production

​
Database Migrations

EM-Runtime services run Alembic database migrations automatically on startup. Each service manages its own database migrations independently.

​
Migration Behavior

AspectBehavior
TriggerAutomatic on pod startup
DirectionForward only (upgrade)
ScopePer-service, per-database
SafetyIdempotent — safe to run multiple times

​
Migration Order

Migrations are applied in service startup order: 
PostgreSQL ready -> Keycloak (schema) -> OpenFGA (schema) -> Infisical (schema)
                 -> Governance (alembic) -> Assets (alembic) -> Utils (alembic)
If a migration fails, the pod will restart and retry. Check pod logs for migration errors before investigating further.

​
Rollback

​
Helm Rollback

# List revision history
helm history em-runtime -n $NAMESPACE

# Rollback to previous revision
helm rollback em-runtime $PREVIOUS_REVISION -n $NAMESPACE

​
Database Rollback Considerations

Alembic migrations are forward-only in production. If a database migration must be reversed, you must restore from backup.
  • Schema-only changes (add column, add table): Generally safe to keep during rollback; old code ignores new columns
  • Data migrations: Require database restore from pre-upgrade backup
  • Destructive migrations (drop column, rename): Require database restore

​
Direct Helm

Always specify --version in production: 
helm upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
  --version 4.8.6 \
  --namespace production \
  -f production-values.yaml

​
Upgrade Troubleshooting

Check pod logs for migration errors:
kubectl logs -n $NAMESPACE deploy/em-runtime-governance --previous
Common causes:
  • Database migration conflict (two migrations with same parent)
  • Missing database or extension
  • Incompatible secret format
Init containers wait for dependencies (PostgreSQL, Redis, Keycloak). Check that all infrastructure services are healthy:
kubectl get pods -n $NAMESPACE -l app.kubernetes.io/component=database
kubectl logs -n $NAMESPACE -l app=keycloak
Verify that hostname and scheme values match your Gateway and DNS configuration:
kubectl get secret em-runtime-secrets -n $NAMESPACE \
  -o jsonpath='{.data.base-url}' | base64 -d
The CMP plugin may need to rebuild the chart dependency. Force a sync:
kubectl get application em-runtime -n argocd -o jsonpath='{.status.sync.status}'
# If OutOfSync, trigger a hard refresh
argocd app get em-runtime --hard-refresh
argocd app sync em-runtime

​
Next Steps

Values Reference

Complete reference for all Helm chart values.