> ## Documentation Index
> Fetch the complete documentation index at: https://docs.emergence.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrades

> Version upgrade procedures, Helm chart promotion workflow, migration steps, and rollback strategies for the em-runtime platform.

# Upgrades

This guide covers the em-runtime Helm chart upgrade process, version promotion through environments, migration procedures, and rollback strategies.

## Upgrade Procedures

### Standard Helm Upgrade

For direct Helm deployments (not using ArgoCD):

```bash theme={null}
# Check current version
helm list -n $NAMESPACE

# Review what will change
helm diff upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
  --version $NEW_VERSION \
  --namespace $NAMESPACE \
  -f values.yaml

# Perform the upgrade
helm upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
  --version $NEW_VERSION \
  --namespace $NAMESPACE \
  -f values.yaml
```

## Pre-Upgrade Checklist

<Warning>
  Always back up PostgreSQL, Redis, and the `em-runtime-secrets` Secret before upgrading.
</Warning>

Before upgrading:

1. **Review the changelog** for breaking changes and required migrations
2. **Back up databases:**
   ```bash theme={null}
   # PostgreSQL
   pg_dumpall -h $PG_HOST -U postgres > backup-pre-upgrade.sql

   # Kubernetes Secrets
   kubectl get secret em-runtime-secrets -n $NAMESPACE -o yaml > em-runtime-secrets-backup.yaml
   kubectl get secret infisical-bootstrap-secret -n $NAMESPACE -o yaml > infisical-bootstrap-backup.yaml
   ```
3. **Verify current health:**
   ```bash theme={null}
   kubectl get pods -n $NAMESPACE
   # The gateway-routed root (/api/<svc>/) returns service metadata with 200
   # when the service is healthy. The Kubernetes liveness/readiness probe target
   # /health lives at the service root, not under the per-service prefix.
   curl https://$HOSTNAME/api/governance/
   ```
4. **Test in non-production first** -- promote through dev and staging before production

## Database Migrations

EM-Runtime services run Alembic database migrations automatically on startup. Each service manages its own database migrations independently.

### Migration Behavior

| Aspect        | Behavior                                 |
| ------------- | ---------------------------------------- |
| **Trigger**   | Automatic on pod startup                 |
| **Direction** | Forward only (upgrade)                   |
| **Scope**     | Per-service, per-database                |
| **Safety**    | Idempotent -- safe to run multiple times |

### Migration Order

Migrations are applied in service startup order:

```text theme={null}
PostgreSQL ready -> Keycloak (schema) -> OpenFGA (schema) -> Infisical (schema)
                 -> Governance (alembic) -> Assets (alembic) -> Utils (alembic)
```

<Note>
  If a migration fails, the pod will restart and retry. Check pod logs for migration errors before investigating further.
</Note>

## Rollback

### Helm Rollback

```bash theme={null}
# List revision history
helm history em-runtime -n $NAMESPACE

# Rollback to previous revision
helm rollback em-runtime $PREVIOUS_REVISION -n $NAMESPACE
```

### Database Rollback Considerations

<Warning>
  Alembic migrations are forward-only in production. If a database migration must be reversed, you must restore from backup.
</Warning>

* **Schema-only changes** (add column, add table): Generally safe to keep during rollback; old code ignores new columns
* **Data migrations**: Require database restore from pre-upgrade backup
* **Destructive migrations** (drop column, rename): Require database restore

### Direct Helm

Always specify `--version` in production:

```bash theme={null}
helm upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
  --version 4.8.6 \
  --namespace production \
  -f production-values.yaml
```

## Upgrade Troubleshooting

<AccordionGroup>
  <Accordion title="Pods stuck in CrashLoopBackOff after upgrade">
    Check pod logs for migration errors:

    ```bash theme={null}
    kubectl logs -n $NAMESPACE deploy/em-runtime-governance --previous
    ```

    Common causes:

    * Database migration conflict (two migrations with same parent)
    * Missing database or extension
    * Incompatible secret format
  </Accordion>

  <Accordion title="Init containers timing out">
    Init containers wait for dependencies (PostgreSQL, Redis, Keycloak). Check that all infrastructure services are healthy:

    ```bash theme={null}
    kubectl get pods -n $NAMESPACE -l app.kubernetes.io/component=database
    kubectl logs -n $NAMESPACE -l app=keycloak
    ```
  </Accordion>

  <Accordion title="HTTPS/CORS errors after upgrade">
    Verify that `hostname` and `scheme` values match your Gateway and DNS configuration:

    ```bash theme={null}
    kubectl get secret em-runtime-secrets -n $NAMESPACE \
      -o jsonpath='{.data.base-url}' | base64 -d
    ```
  </Accordion>

  <Accordion title="ArgoCD shows OutOfSync after promotion">
    The CMP plugin may need to rebuild the chart dependency. Force a sync:

    ```bash theme={null}
    kubectl get application em-runtime -n argocd -o jsonpath='{.status.sync.status}'
    # If OutOfSync, trigger a hard refresh
    argocd app get em-runtime --hard-refresh
    argocd app sync em-runtime
    ```
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Values Reference" icon="file-code" href="/deployment/helm/values-reference">
    Complete reference for all Helm chart values.
  </Card>
</CardGroup>