> ## Documentation Index
> Fetch the complete documentation index at: https://docs.emergence.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Secrets Infrastructure

> Deploy and configure ESO + GCP Secret Manager or Infisical for secrets management in CRAFT.

# Secrets Infrastructure

This guide covers deploying and configuring the secrets management backend for CRAFT. Two options are supported, choose based on your deployment environment.

<Info>
  For a comparison of the two backends and guidance on which to choose, see [Secrets Management](/security/secrets-management).
</Info>

***

```mermaid theme={null}
%%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%%
graph TB
    subgraph OptionA["Option A: ESO + GCP Secret Manager"]
        GCP["GCP Secret Manager\n(Terraform provisioned)"]
        CSS["ClusterSecretStore\n(Workload Identity auth)"]
        ES["ExternalSecret\n(K8s resource, refresh interval)"]
        KS["Kubernetes Secret"]
        SR["Stakater Reloader\n(watches annotations)"]
        POD1["Pod restart\n(rolling update)"]
        GCP --> CSS --> ES --> KS --> SR --> POD1
    end

    subgraph OptionB["Option B: Infisical"]
        INF["Infisical Server\n(Helm subchart)"]
        MI["Machine Identity\n(credentials in Helm values)"]
        POD2["Pods fetch secrets\nat startup via env injection"]
        INF --> MI --> POD2
    end

    classDef gcp fill:#0072B2,stroke:#555555,color:#fff
    classDef k8s fill:#56B4E9,stroke:#555555,color:#000
    classDef infra fill:#E69F00,stroke:#555555,color:#000
    classDef pod fill:#009E73,stroke:#555555,color:#000
    class GCP gcp
    class CSS,ES,KS,SR k8s
    class INF,MI infra
    class POD1,POD2 pod
```

## Option A: ESO + GCP Secret Manager

For GKE and GCP-based deployments. Infrastructure is provisioned by Terraform; Kubernetes resources are managed by ArgoCD.

### Prerequisites

* GKE cluster with Workload Identity enabled
* Terraform-provisioned GCP infrastructure
* ArgoCD deployed and managing the cluster

### 1. Terraform Provisions GCP Secrets

The `gke` Terraform module automatically creates the required secrets in GCP Secret Manager when you run `terraform apply`. No manual secret creation is needed for infrastructure secrets, the module provisions them with placeholder values that you update after provisioning.

After running Terraform:

1. Navigate to [GCP Secret Manager](https://console.cloud.google.com/security/secret-manager) in your project
2. Update each provisioned secret with its actual value
3. ESO will sync the updated values into Kubernetes on the next sync cycle

### 2. Install External Secrets Operator

ESO can be installed as an ArgoCD-managed application or deployed manually:

```bash theme={null}
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --set installCRDs=true
```

### 3. ClusterSecretStore

ESO requires a `ClusterSecretStore` that configures how to authenticate to GCP Secret Manager. This is provisioned by Terraform/ArgoCD:

```yaml theme={null}
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gcp-secret-manager
spec:
  provider:
    gcpsm:
      projectID: "your-project-id"   # Replace with your GCP project
      auth:
        workloadIdentity:
          clusterLocation: us-central1
          clusterName: "your-cluster-name"
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
```

### 4. ExternalSecret Resources

Each service has `ExternalSecret` resources that map GCP SM secrets to K8s Secrets. These are managed by ArgoCD in the `kubernetes/em-runtime/` directory:

```yaml theme={null}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: em-runtime-credentials
  namespace: em-runtime
  annotations:
    reloader.stakater.com/auto: "true"   # Enable auto-restart on change
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-secret-manager
    kind: ClusterSecretStore
  target:
    name: em-runtime-credentials
    creationPolicy: Owner
  data:
    - secretKey: postgres-password
      remoteRef:
        key: em-runtime   # GCP SM secret name
        property: postgres-password
    - secretKey: redis-password
      remoteRef:
        key: em-runtime
        property: redis-password
```

### 5. Install Stakater Reloader

Reloader watches K8s Secrets for changes and triggers rolling restarts of annotated pods:

```bash theme={null}
helm install reloader stakater/reloader \
  --namespace em-runtime \
  --set reloader.watchGlobally=false
```

Pods are automatically enrolled when their Deployment or the Secret has the annotation:

```yaml theme={null}
metadata:
  annotations:
    reloader.stakater.com/auto: "true"
```

### 6. Verify Sync

```bash theme={null}
# Check ExternalSecret status
kubectl get externalsecrets -n em-runtime

# Check the synced K8s Secret exists
kubectl get secret em-runtime-credentials -n em-runtime

# Check Reloader is watching
kubectl logs -n em-runtime -l app.kubernetes.io/name=reloader | head -20
```

### Manual Sync (Immediate Rotation)

To force an immediate sync without waiting for the refresh interval:

```bash theme={null}
kubectl annotate externalsecret em-runtime-credentials \
  --namespace em-runtime \
  force-sync="$(date +%s)" \
  --overwrite
```

***

## Option B: Infisical

For on-premises and cloud-agnostic deployments. Infisical runs as a Helm subchart within em-runtime.

### Prerequisites

* Kubernetes cluster (any CNCF-conformant distribution)
* Helm chart deployed with Infisical subchart enabled

### 1. Enable Infisical in Helm Values

Infisical is enabled by default. Verify in your values file:

```yaml theme={null}
infisical:
  enabled: true
  resources:
    requests:
      cpu: "250m"
      memory: "512Mi"
    limits:
      cpu: "1000m"
      memory: "1Gi"
```

### 2. Bootstrap Infisical

On first deployment, Infisical initializes with the admin email configured in Helm:

```yaml theme={null}
bootstrap:
  adminEmail: "admin@your-domain.com"
```

Access the Infisical UI and complete initial setup:

```bash theme={null}
kubectl port-forward svc/em-runtime-infisical -n em-runtime 8080:80
# Open http://localhost:8080
```

### 3. Create Machine Identity

Create a Kubernetes machine identity in Infisical for each service that needs secret access:

1. In the Infisical UI, navigate to **Project → Access Control → Machine Identities**
2. Create a machine identity with the Kubernetes authentication method
3. Configure the service account name and namespace

### 4. Inject Credentials

Infisical credentials are passed to services via environment variables in Helm values:

```yaml theme={null}
governance:
  extraEnvVars:
    - name: INFISICAL_URL
      value: "http://em-runtime-infisical.em-runtime.svc.cluster.local"
    - name: INFISICAL_CLIENT_ID
      valueFrom:
        secretKeyRef:
          name: infisical-machine-identity
          key: clientId
    - name: INFISICAL_CLIENT_SECRET
      valueFrom:
        secretKeyRef:
          name: infisical-machine-identity
          key: clientSecret
```

### 5. Verify

```bash theme={null}
# Check Infisical pod is running
kubectl get pods -n em-runtime -l app.kubernetes.io/name=infisical

# Check governance service logs for Infisical connection
kubectl logs -n em-runtime -l app=em-runtime-governance | grep -i infisical
```

***

## Backup and Recovery

### ESO + GCP Secret Manager

GCP Secret Manager has built-in version history. To recover a previous secret value:

```bash theme={null}
# List secret versions
gcloud secrets versions list your-secret-name --project=your-project-id

# Access a specific version
gcloud secrets versions access VERSION_ID --secret=your-secret-name --project=your-project-id
```

### Infisical

Back up the Infisical PostgreSQL database (`infisical` database) using standard PostgreSQL backup tools:

```bash theme={null}
kubectl exec -n em-runtime deployment/em-runtime-infisical -- \
  pg_dump -U infisical infisical > infisical-backup-$(date +%Y%m%d).sql
```

Include this backup in your regular database backup rotation. See [Backup and Restore](/guides/backup-restore) for the full backup procedure.

## Next Steps

<CardGroup cols={2}>
  <Card title="Secrets Management Overview" icon="key" href="/security/secrets-management">
    Concepts, architecture, and backend comparison.
  </Card>

  <Card title="Backup and Restore" icon="database" href="/guides/backup-restore">
    Full backup procedures including secrets backends.
  </Card>
</CardGroup>