Upgrades

This guide covers the em-runtime Helm chart upgrade process, version promotion through environments, migration procedures, and rollback strategies.

Upgrade Procedures

Standard Helm Upgrade

For direct Helm deployments (not using ArgoCD):

# Check current version
helm list -n $NAMESPACE

# Review what will change
helm diff upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
  --version $NEW_VERSION \
  --namespace $NAMESPACE \
  -f values.yaml

# Perform the upgrade
helm upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
  --version $NEW_VERSION \
  --namespace $NAMESPACE \
  -f values.yaml

Pre-Upgrade Checklist

Always back up PostgreSQL, Redis, and the em-runtime-secrets Secret before upgrading.

Before upgrading:

Review the changelog for breaking changes and required migrations

Back up databases:

# PostgreSQL
pg_dumpall -h $PG_HOST -U postgres > backup-pre-upgrade.sql

# Kubernetes Secrets
kubectl get secret em-runtime-secrets -n $NAMESPACE -o yaml > em-runtime-secrets-backup.yaml
kubectl get secret infisical-bootstrap-secret -n $NAMESPACE -o yaml > infisical-bootstrap-backup.yaml

Verify current health:

kubectl get pods -n $NAMESPACE
# The gateway-routed root (/api/<svc>/) returns service metadata with 200
# when the service is healthy. The Kubernetes liveness/readiness probe target
# /health lives at the service root, not under the per-service prefix.
curl https://$HOSTNAME/api/governance/

Test in non-production first — promote through dev and staging before production

Database Migrations

EM-Runtime services run Alembic database migrations automatically on startup. Each service manages its own database migrations independently.

Migration Behavior

Aspect	Behavior
Trigger	Automatic on pod startup
Direction	Forward only (upgrade)
Scope	Per-service, per-database
Safety	Idempotent — safe to run multiple times

Migration Order

Migrations are applied in service startup order:

PostgreSQL ready -> Keycloak (schema) -> OpenFGA (schema) -> Infisical (schema)
                 -> Governance (alembic) -> Assets (alembic) -> Utils (alembic)

If a migration fails, the pod will restart and retry. Check pod logs for migration errors before investigating further.

Rollback

Helm Rollback

# List revision history
helm history em-runtime -n $NAMESPACE

# Rollback to previous revision
helm rollback em-runtime $PREVIOUS_REVISION -n $NAMESPACE

Database Rollback Considerations

Alembic migrations are forward-only in production. If a database migration must be reversed, you must restore from backup.

Schema-only changes (add column, add table): Generally safe to keep during rollback; old code ignores new columns
Data migrations: Require database restore from pre-upgrade backup
Destructive migrations (drop column, rename): Require database restore

Direct Helm

Always specify --version in production:

helm upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
  --version 4.8.6 \
  --namespace production \
  -f production-values.yaml

Upgrade Troubleshooting

Pods stuck in CrashLoopBackOff after upgrade

Check pod logs for migration errors:

kubectl logs -n $NAMESPACE deploy/em-runtime-governance --previous

Common causes:

Database migration conflict (two migrations with same parent)
Missing database or extension
Incompatible secret format

Init containers timing out

Init containers wait for dependencies (PostgreSQL, Redis, Keycloak). Check that all infrastructure services are healthy:

kubectl get pods -n $NAMESPACE -l app.kubernetes.io/component=database
kubectl logs -n $NAMESPACE -l app=keycloak

HTTPS/CORS errors after upgrade

Verify that hostname and scheme values match your Gateway and DNS configuration:

kubectl get secret em-runtime-secrets -n $NAMESPACE \
  -o jsonpath='{.data.base-url}' | base64 -d

ArgoCD shows OutOfSync after promotion

The CMP plugin may need to rebuild the chart dependency. Force a sync:

kubectl get application em-runtime -n argocd -o jsonpath='{.status.sync.status}'
# If OutOfSync, trigger a hard refresh
argocd app get em-runtime --hard-refresh
argocd app sync em-runtime

Next Steps

Values Reference

Complete reference for all Helm chart values.

​Upgrades

​Upgrade Procedures

​Standard Helm Upgrade

​Pre-Upgrade Checklist

​Database Migrations

​Migration Behavior

​Migration Order

​Rollback

​Helm Rollback

​Database Rollback Considerations

​Direct Helm

​Upgrade Troubleshooting

​Next Steps

Values Reference

Upgrades

Upgrade Procedures

Standard Helm Upgrade

Pre-Upgrade Checklist

Database Migrations

Migration Behavior

Migration Order

Rollback

Helm Rollback

Database Rollback Considerations

Direct Helm

Upgrade Troubleshooting

Next Steps