> ## Documentation Index
> Fetch the complete documentation index at: https://docs.emergence.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Backup & Restore

> How to back up and restore CRAFT data including PostgreSQL databases, Keycloak realms, OpenFGA state, and Infisical secrets.

# Backup and Restore Platform Data

This guide covers backup and restore procedures for all stateful components of CRAFT. Each service owns its own database, and infrastructure services (Keycloak, OpenFGA, Infisical) maintain independent state that must be backed up separately.

## Stateful Components

The platform has the following stateful components that require backup:

| Component         | Storage         | Data                                              |
| ----------------- | --------------- | ------------------------------------------------- |
| **Governance DB** | PostgreSQL      | Organizations, projects, users, role assignments  |
| **Assets DB**     | PostgreSQL      | Artifacts, data connections, files, models        |
| **Utils DB**      | PostgreSQL      | Data catalog, scheduling, context packs, memories |
| **Keycloak**      | PostgreSQL      | Realms, users, IdP configurations, sessions       |
| **OpenFGA**       | PostgreSQL      | Authorization model, relationship tuples          |
| **Infisical**     | Internal store  | Application secrets, data connection credentials  |
| **Redis**         | In-memory + AOF | Cache, session state, event streams               |
| **Solution DBs**  | PostgreSQL      | Data Insights sessions, Data Governance profiles  |

<Warning>
  Each service has a separate PostgreSQL database. There are no cross-service foreign keys, so databases can be backed up independently without coordination.
</Warning>

## PostgreSQL Backup

### Automated Backups (Cloud-Managed)

For cloud-managed PostgreSQL (Cloud SQL, RDS, Azure Database):

<Tabs>
  <Tab title="GCP Cloud SQL">
    Cloud SQL provides automated daily backups with configurable retention.

    ```bash theme={null}
    # Enable automated backups
    gcloud sql instances patch <instance-name> \
      --backup-start-time=02:00 \
      --retained-backups-count=30 \
      --retained-transaction-log-days=7

    # Create an on-demand backup
    gcloud sql backups create --instance=<instance-name>

    # List available backups
    gcloud sql backups list --instance=<instance-name>
    ```
  </Tab>

  <Tab title="AWS RDS">
    ```bash theme={null}
    # Enable automated backups with 30-day retention
    aws rds modify-db-instance \
      --db-instance-identifier <instance-id> \
      --backup-retention-period 30

    # Create a manual snapshot
    aws rds create-db-snapshot \
      --db-instance-identifier <instance-id> \
      --db-snapshot-identifier emergence-backup-$(date +%Y%m%d)
    ```
  </Tab>

  <Tab title="Azure">
    ```bash theme={null}
    # Azure Database for PostgreSQL has automated backups enabled by default
    # Configure retention (7-35 days)
    az postgres flexible-server update \
      --resource-group <rg> \
      --name <server-name> \
      --backup-retention 30
    ```
  </Tab>
</Tabs>

### Manual Backups (On-Premises)

For self-managed PostgreSQL deployments:

```bash theme={null}
# Back up all platform databases
DATABASES="keycloak openfga infisical governance assets utils prefect datareadiness talk2data"
BACKUP_DIR="/backups/emergence/$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"

for db in $DATABASES; do
  pg_dump -h localhost -U postgres -Fc -f "$BACKUP_DIR/$db.dump" "$db"
done

# Verify backups
for dump in "$BACKUP_DIR"/*.dump; do
  pg_restore --list "$dump" > /dev/null && echo "OK: $dump" || echo "FAILED: $dump"
done
```

<Tip>
  Use `pg_dump -Fc` (custom format) for backups. It supports parallel restore and selective table restoration, and compresses data automatically.
</Tip>

### Point-in-Time Recovery

For production deployments, enable WAL archiving for point-in-time recovery:

```bash theme={null}
# postgresql.conf
archive_mode = on
archive_command = 'cp %p /archive/wal/%f'
wal_level = replica
```

## Keycloak Backup

Keycloak state is primarily stored in its PostgreSQL database, but realm configuration can also be exported as JSON for version control.

```bash theme={null}
# Export realm configuration (includes users, IdPs, clients)
# Run from within the Keycloak container or pod
/opt/keycloak/bin/kc.sh export \
  --dir /tmp/keycloak-export \
  --realm <realm-name> \
  --users realm_file

# Copy the export out of the container
kubectl cp <keycloak-pod>:/tmp/keycloak-export ./keycloak-backup/
```

<Note>
  Realm exports include user accounts but not user credentials (passwords). Users will need to reset passwords after a realm import. SSO users are unaffected since their credentials are managed by the external IdP.
</Note>

## OpenFGA Backup

OpenFGA stores its authorization model and relationship tuples in PostgreSQL. The database backup covers all OpenFGA state. For additional safety, export the authorization model:

```bash theme={null}
# Export the current authorization model
curl -s "http://<openfga-host>:8080/stores/<store-id>/authorization-models" \
  -H "Authorization: Bearer $TOKEN" | jq . > openfga-model-backup.json
```

## Secrets Backend Backup

The platform supports two secrets backends. Back up whichever you are using.

<Tabs>
  <Tab title="Infisical">
    Infisical manages application secrets including data connection credentials. Back up the Infisical database (PostgreSQL `infisical` database) and preserve the encryption keys.

    <Warning>
      Infisical secrets are encrypted at rest. Backing up the database preserves the encrypted data, but you must also preserve the `ENCRYPTION_KEY` and `AUTH_SECRET` for restoration, without them, encrypted data cannot be decrypted.
    </Warning>

    ```bash theme={null}
    pg_dump -h localhost -U postgres -Fc -f infisical-backup.dump infisical
    # Also export the encryption key from the infisical-bootstrap-secret:
    kubectl get secret infisical-bootstrap-secret -n em-runtime -o yaml > infisical-bootstrap-secret.yaml
    ```
  </Tab>

  <Tab title="ESO + GCP Secret Manager">
    Secrets stored in GCP Secret Manager have built-in version history. No separate backup action is required, GCP retains all secret versions.

    To export a snapshot of all current secret values:

    ```bash theme={null}
    # List all em-runtime secrets
    gcloud secrets list --project=your-project-id --filter="name:em-runtime"

    # Verify each secret has at least one active version
    gcloud secrets versions list em-runtime --project=your-project-id
    ```

    For disaster recovery, ensure IAM bindings for the ESO service account are documented and reproducible via Terraform.
  </Tab>
</Tabs>

## Redis Backup

Redis serves as a cache and event stream. While cache data is ephemeral, you may want to back up Redis for faster recovery:

```bash theme={null}
# Trigger a manual RDB snapshot
redis-cli -a $REDIS_PASSWORD BGSAVE

# Copy the dump file
kubectl cp <redis-pod>:/data/dump.rdb ./redis-backup/dump.rdb
```

## Restore Procedures

### Restore PostgreSQL Databases

```bash theme={null}
# Restore a single database from a custom-format dump
pg_restore -h localhost -U postgres -d governance -Fc --clean --if-exists governance.dump

# Restore all platform databases
for db in keycloak openfga infisical governance assets utils prefect datareadiness talk2data; do
  pg_restore -h localhost -U postgres -d "$db" -Fc --clean --if-exists "$db.dump"
done
```

### Restore Keycloak Realms

```bash theme={null}
# Import realm configuration
/opt/keycloak/bin/kc.sh import \
  --dir /tmp/keycloak-import \
  --override true
```

### Restore from Cloud Managed Backups

<Tabs>
  <Tab title="GCP Cloud SQL">
    ```bash theme={null}
    # Restore from an automated backup
    gcloud sql backups restore <backup-id> \
      --restore-instance=<instance-name>
    ```
  </Tab>

  <Tab title="AWS RDS">
    ```bash theme={null}
    # Restore from a snapshot (creates a new instance)
    aws rds restore-db-instance-from-db-snapshot \
      --db-instance-identifier <new-instance-id> \
      --db-snapshot-identifier <snapshot-id>
    ```
  </Tab>
</Tabs>

## Backup Schedule Recommendations

| Component                   | Frequency              | Retention  | Method                                                       |
| --------------------------- | ---------------------- | ---------- | ------------------------------------------------------------ |
| PostgreSQL (all DBs)        | Daily + continuous WAL | 30 days    | Cloud-managed or pg\_dump                                    |
| Keycloak realm exports      | Weekly                 | 90 days    | JSON export                                                  |
| OpenFGA model               | On change              | Indefinite | JSON export in version control                               |
| Redis                       | Daily                  | 7 days     | RDB snapshot                                                 |
| Secrets backend (Infisical) | Daily                  | 30 days    | Database backup with encryption keys                         |
| Secrets backend (GCP SM)    | N/A, managed           | Indefinite | GCP retains all versions; document IAM bindings in Terraform |

## Disaster Recovery Checklist

<Steps>
  <Step title="Restore PostgreSQL databases">
    Restore all platform databases from the latest backup. Verify row counts and data integrity.
  </Step>

  <Step title="Restore Keycloak">
    Import realm configurations. SSO users will re-authenticate via their IdP. Local users may need password resets.
  </Step>

  <Step title="Verify OpenFGA schema">
    Confirm the authorization model is loaded. The Governance service re-applies the schema on startup.
  </Step>

  <Step title="Restore secrets backend">
    **Infisical**: Restore the Infisical database and encryption keys. Verify data connection credentials are accessible. **ESO + GCP Secret Manager**: Verify GCP Secret Manager secrets are intact and ESO ClusterSecretStore can authenticate via Workload Identity.
  </Step>

  <Step title="Restart platform services">
    Follow the startup order: PostgreSQL -> Redis -> Keycloak -> OpenFGA -> Governance -> Assets/Utils -> Solutions.
  </Step>

  <Step title="Validate end-to-end">
    Run health checks on all services. Test authentication, permission checks, and data connection queries.
  </Step>
</Steps>

## Next Steps

<CardGroup cols={2}>
  <Card title="Deployment Overview" icon="cloud" href="/deployment/overview">
    Review the full deployment architecture and infrastructure requirements.
  </Card>

  <Card title="Helm Configuration" icon="dharmachakra" href="/deployment/helm/configuration">
    Configure Helm values for backup-related settings.
  </Card>

  <Card title="GDPR Compliance" icon="scale-balanced" href="/security/compliance/gdpr">
    Understand data retention requirements for GDPR compliance.
  </Card>

  <Card title="Network Security" icon="shield" href="/security/network-security">
    Secure backup data in transit and at rest.
  </Card>
</CardGroup>