> ## Documentation Index
> Fetch the complete documentation index at: https://docs.emergence.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Helm Configuration

> Configure the em-runtime Helm chart for development, staging, and production environments with external databases, S3 storage, and HTTPS.

# Helm Configuration

The em-runtime Helm chart packages all platform services and their dependencies into a single deployable unit. This guide covers the chart architecture, deployment modes, secret management, and production configuration.

## Chart Overview

The chart deploys and manages the following components:

| Component                 | Description                                                                                                                        |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| **em-runtime-governance** | Organizations, projects, and permissions                                                                                           |
| **em-runtime-assets**     | Artifacts, data connections, files, and models                                                                                     |
| **em-runtime-utils**      | Data catalog, scheduling, context packs, and memories                                                                              |
| **Keycloak**              | Identity and access management (multi-tenant realms)                                                                               |
| **OpenFGA**               | Fine-grained authorization (Zanzibar model)                                                                                        |
| **Infisical**             | Secrets management (optional; can be replaced by ESO + GCP Secret Manager, see [Secrets Management](/security/secrets-management)) |
| **PostgreSQL**            | Persistent data storage (optional, in-cluster)                                                                                     |
| **Redis**                 | Caching and session storage (optional, in-cluster)                                                                                 |

## Deployment Modes

<Tabs>
  <Tab title="In-Cluster (Default)">
    All dependencies (PostgreSQL, Redis) are deployed as part of the Helm release within the cluster.

    ```bash theme={null}
    helm install em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
      --version $VERSION \
      --namespace=$NAMESPACE
    ```

    Best for development, testing, and quick setup.
  </Tab>

  <Tab title="External Databases (Production)">
    PostgreSQL and Redis run outside the cluster (managed services or self-hosted). The chart connects via configuration.

    ```bash theme={null}
    helm install em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
      --version $VERSION \
      --namespace=$NAMESPACE \
      --set postgres.enabled=false \
      --set redis.enabled=false \
      --set existingSecret=em-runtime-external-credentials
    ```

    Best for production with managed database services.
  </Tab>
</Tabs>

## Secret Management

All platform credentials are consolidated into a single chart-managed Secret named `em-runtime-secrets`. In external mode, you provide base connection details, and the chart auto-computes derived values.

### External Credentials Secret

Create this before installing the chart when using external databases:

```bash theme={null}
kubectl create secret generic em-runtime-external-credentials \
  --namespace=$NAMESPACE \
  --from-literal=postgres-password='<password>' \
  --from-literal=postgres-host='<hostname>' \
  --from-literal=postgres-port='5432' \
  --from-literal=postgres-username='postgres' \
  --from-literal=redis-password='<password>' \
  --from-literal=redis-host='<hostname>' \
  --from-literal=redis-port='6379' \
  --from-literal=redis-scheme='redis' \
  --from-literal=redis-username='default'
```

<Note>
  Set `redis-scheme` to `rediss` (double `s`) for TLS connections (e.g., GCP Memorystore with in-transit encryption).
</Note>

### S3 Storage Credentials

```bash theme={null}
kubectl create secret generic em-runtime-s3 \
  --namespace=$NAMESPACE \
  --from-literal=access-key-id='<YOUR_ACCESS_KEY>' \
  --from-literal=secret-access-key='<YOUR_SECRET_KEY>'
```

Skip this step if using IRSA (AWS) or GCP Workload Identity.

## External Object Storage

For production, use S3-compatible object storage instead of the default PVC:

```yaml theme={null}
em-runtime-assets:
  storage:
    enabled: false                              # Disable PVC
  extraEnvVars:
    - name: STORAGE_TYPE
      value: "s3"
    - name: S3_REGION
      value: "us-east-1"
    - name: S3_BUCKET_NAME
      value: "my-bucket"
    - name: S3_PREFIX_UPLOADED
      value: "uploaded"
    - name: S3_PREFIX_GENERATED
      value: "generated"
    - name: S3_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef: { name: em-runtime-s3, key: access-key-id }
    - name: S3_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef: { name: em-runtime-s3, key: secret-access-key }
```

### Provider-Specific Notes

| Provider                 | Configuration                                                                                    |
| ------------------------ | ------------------------------------------------------------------------------------------------ |
| **AWS S3**               | No `S3_ENDPOINT_URL` needed. Set `S3_REGION` to your bucket's region.                            |
| **Google Cloud Storage** | Use HMAC keys. Set `S3_ENDPOINT_URL` to `https://storage.googleapis.com`, `S3_REGION` to `auto`. |
| **MinIO**                | Set `S3_ENDPOINT_URL` to your MinIO endpoint (e.g., `http://minio:9000`).                        |

## Enabling HTTPS

EM-Runtime uses Gateway API for routing. TLS termination is configured on the Gateway resource using cert-manager.

<Steps>
  <Step title="Install cert-manager">
    Install cert-manager with Gateway API support enabled (`config.enableGatewayAPI=true`).
  </Step>

  <Step title="Create ClusterIssuer">
    Create a ClusterIssuer for your ACME provider (e.g., Let's Encrypt).
  </Step>

  <Step title="Annotate Gateway">
    Add `cert-manager.io/cluster-issuer` annotation and configure an HTTPS listener on your Gateway resource.
  </Step>
</Steps>

## Service URL Environment Variables

Service-to-service communication uses canonical URL environment variables. These names were standardized as part of PE-200; legacy aliases are retained as temporary hotfixes for older application versions.

| Canonical name          | Replaces (legacy)                           | Purpose                 |
| ----------------------- | ------------------------------------------- | ----------------------- |
| `KEYCLOAK_URL`          | `AUTH_URL`, `EM_RUNTIME_KEYCLOAK_URL`       | Keycloak base URL       |
| `EM_RUNTIME_ASSETS_URL` | `RUNTIME_URL` (when used for asset routing) | Assets service base URL |
| `EM_RUNTIME_UTILS_URL`  | `RUNTIME_URL` (when used for utils routing) | Utils service base URL  |

Application versions still on the legacy names continue to work because both are set during the rollout. Once all consumer applications upgrade past the migration point, the legacy aliases will be removed.

### UI compliance and external link variables

The Runtime UI surfaces compliance links (Terms of Service, Privacy Policy, DPA) and an embedded analytics script. Configure them via env vars on the `em-runtime-ui` deployment:

| Variable                   | Purpose                                                             |
| -------------------------- | ------------------------------------------------------------------- |
| `YIELD_INSIGHTS_URL`       | URL of the Yield Insights solution (semiconductor deployments only) |
| `TERMS_OF_SERVICE_URL`     | Terms of Service link surfaced in the UI footer                     |
| `PRIVACY_POLICY_URL`       | Privacy Policy link surfaced in the UI footer                       |
| `EMERGENCE_DPA_URL`        | Data Processing Agreement link                                      |
| `EMERGENCE_DPA_UPDATE_URL` | DPA update notification link                                        |
| `TERMLY_SCRIPT_SRC`        | Termly script source URL for compliance banner                      |

These are configured via the `em-runtime-ui.env` block in `values.yaml`, the same way as service URLs above.

## Environment Variable Override System

Each runtime service has three layers for environment variables:

| Layer          | Format                  | Purpose                                                                   |
| -------------- | ----------------------- | ------------------------------------------------------------------------- |
| `env`          | `map[string]string`     | Simple key-value pairs. **Use this to override defaults.**                |
| `envVars`      | List of K8s env objects | Variables using `valueFrom` (secrets, config maps). Defined by the chart. |
| `extraEnvVars` | List of K8s env objects | Additional variables appended after `envVars`.                            |

### Override Precedence

```text theme={null}
env (wins) --> envVars (skipped if name exists in env) --> extraEnvVars (skipped if name exists in env)
```

**Example -- override defaults across services:**

```yaml theme={null}
em-runtime-governance:
  env:
    DOCS_ENABLED: "false"       # Disable API docs
    LOG_LEVEL: "WARNING"        # Reduce log verbosity
    CORS_ENABLED: "false"       # Disable CORS

em-runtime-assets:
  env:
    DOCS_ENABLED: "false"
    LOG_LEVEL: "WARNING"

em-runtime-utils:
  env:
    DOCS_ENABLED: "false"
    LOG_LEVEL: "WARNING"
```

## Bootstrap Configuration

On first startup, the platform bootstraps a default organization:

```yaml theme={null}
bootstrap:
  organizationId: "acme"              # Also used as Keycloak realm name
  organizationName: "ACME Corp"
  organizationDescription: "ACME Corporation platform"
  adminEmail: "admin@acme.com"
```

The bootstrap is **idempotent** -- it checks for existing resources before creating.

<Note>
  The `organizationId` must contain only alphanumeric characters, hyphens, or underscores.
</Note>

## Production Values File

Below is a sample `production-values.yaml`:

```yaml theme={null}
# External hostname and scheme
hostname: "api.example.com"
scheme: "https"

# Bootstrap organization
bootstrap:
  organizationId: "my-org"
  organizationName: "My Organization"
  adminEmail: "admin@example.com"

# External databases
existingSecret: "em-runtime-external-credentials"
postgres:
  enabled: false
redis:
  enabled: false

# Keycloak
keycloak:
  replicaCount: 2
  resources:
    requests: { memory: "1536Mi", cpu: "1000m" }
    limits:   { memory: "2Gi",    cpu: "2000m" }

# OpenFGA with HPA
openfga:
  replicaCount: 2
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 75

# Governance with HPA
em-runtime-governance:
  replicaCount: 2
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 75
  env:
    LOG_LEVEL: "WARNING"
    DOCS_ENABLED: "false"

# Assets with HPA and S3
em-runtime-assets:
  replicaCount: 2
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
  storage:
    enabled: false
  env:
    LOG_LEVEL: "WARNING"
    DOCS_ENABLED: "false"
  extraEnvVars:
    - name: STORAGE_TYPE
      value: "s3"
    - name: S3_REGION
      value: "us-east-1"
    - name: S3_BUCKET_NAME
      value: "my-org-em-runtime"

# Utils with HPA
em-runtime-utils:
  replicaCount: 2
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
  env:
    LOG_LEVEL: "WARNING"
    DOCS_ENABLED: "false"
```

### Install with Production Values

```bash theme={null}
helm install em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
  --version $VERSION \
  --namespace production \
  -f production-values.yaml
```

## Testing

The chart includes a Helm test hook that runs end-to-end tests:

```bash theme={null}
helm test $RELEASE --namespace $NAMESPACE --timeout 10m
```

The test job creates a temporary organization, validates role-based access across all permission groups, and cleans up.

## Backups

Back up these components together for a consistent restore:

| Component                      | Contains                                    | Backup Method                                           |
| ------------------------------ | ------------------------------------------- | ------------------------------------------------------- |
| **PostgreSQL**                 | All service data                            | `pg_dumpall` or cloud snapshots                         |
| **Redis**                      | Session and cache data                      | `redis-cli BGSAVE` or cloud snapshots                   |
| **em-runtime-secrets**         | All platform credentials                    | `kubectl get secret em-runtime-secrets -o yaml`         |
| **infisical-bootstrap-secret** | Machine identity token (if using Infisical) | `kubectl get secret infisical-bootstrap-secret -o yaml` |
| **Assets storage**             | Uploaded/generated artifacts                | VolumeSnapshots or S3 bucket backup                     |

<Warning>
  If using Infisical, the `em-runtime-secrets` secret contains the `ENCRYPTION_KEY` and `AUTH_SECRET`. Without these, encrypted data stored by Infisical cannot be decrypted. If using ESO + GCP Secret Manager, secrets are sourced from GCP SM directly, back up the GCP Secret Manager secrets instead.
</Warning>

## Next Steps

<CardGroup cols={2}>
  <Card title="Values Reference" icon="file-code" href="/deployment/helm/values-reference">
    Complete reference for all Helm chart values.
  </Card>

  <Card title="Upgrades" icon="arrow-up" href="/deployment/helm/upgrades">
    Version upgrades, migration steps, and rollback procedures.
  </Card>

  <Card title="OpenTelemetry" icon="wave-square" href="/deployment/observability/opentelemetry">
    Configure telemetry for all runtime services.
  </Card>
</CardGroup>