> ## Documentation Index
> Fetch the complete documentation index at: https://docs.emergence.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Manage Secrets

> Retrieve secrets at runtime — environment injection via em-service, file mounts at /mnt/secrets/, and adding new secrets to your chart.

# Manage Secrets

This page is about how a *solution* gets secrets at runtime. The platform handles sourcing, rotation, and delivery — your job is to declare what you need and read it correctly.

For the platform-side architecture (Infisical vs ESO + GCP Secret Manager, rotation mechanics, Workload Identity), see [Security › Secrets Management](/security/secrets-management) and [Deployment › Infrastructure › Secrets](/deployment/infrastructure/secrets).

## Secret pipeline

```mermaid theme={null}
%%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%%
flowchart LR
    subgraph Upstream["Secret backend (platform-managed)"]
        BACKEND[("Infisical<br/>or<br/>ESO + GCP SM")]
    end
    subgraph Cluster["Your namespace (em-&lt;solution&gt;)"]
        K8S[("K8s Secret<br/>&lt;solution&gt;-secrets")]
        POD["Your pod"]
        ENV["env vars<br/>(via envFrom)"]
        FILE["files at<br/>/mnt/secrets/&lt;key&gt;"]
    end
    RELOAD["Stakater Reloader"]

    BACKEND -->|"sync (~1h)"| K8S
    K8S --> ENV
    K8S --> FILE
    ENV --> POD
    FILE --> POD
    K8S -.->|"on change"| RELOAD
    RELOAD -.->|"rolling restart"| POD

    classDef store fill:#009E73,stroke:#555555,color:#000
    classDef infra fill:#0072B2,stroke:#555555,color:#fff
    classDef act fill:#E69F00,stroke:#555555,color:#000
    classDef sol fill:#56B4E9,stroke:#555555,color:#000
    class BACKEND,K8S store
    class RELOAD infra
    class ENV,FILE act
    class POD sol
```

**Read the diagram left-to-right**: the upstream backend is the source of truth; ESO or Infisical syncs values into a K8s Secret in your namespace; `em-service` injects them into your pod as env vars (via `envFrom`) and/or files (via volume mount). On rotation, Reloader detects the K8s Secret change and rolls your Deployment.

## Two delivery shapes

The `em-service` chart delivers secrets to your pod in two ways. Pick whichever fits the consumer:

| Shape                    | Format                                                                        | Best for                                                                                |
| ------------------------ | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| **Environment variable** | Loaded via `envFrom` from a synthesized `<service>-secrets` Kubernetes Secret | Most use cases (DB URLs, API keys, OIDC client secrets)                                 |
| **File mount**           | Files at `/mnt/secrets/<key>`                                                 | Multi-line secrets (PEM keys, certs); SDKs that expect a file path (gcloud, kubeconfig) |

Both are populated from the *same* upstream secret backend; the difference is purely how your code reads the value.

## Steps

<Steps>
  <Step title="Declare the secret in your chart">
    Reference the secret by `secretKeyRef` in your `values.yaml`. The Kubernetes Secret itself (`<service>-secrets`) is created by the platform's secret pipeline — you do not check secret values into the chart.

    ```yaml charts/<solution>/values.yaml theme={null}
    api:
      env:
        # Plain values: use the simple form
        LOG_LEVEL: "INFO"
        KEYCLOAK_ISSUER_URL: "https://keycloak.example.com/realms/acme"
        KEYCLOAK_AUDIENCE: "<solution>-api"
      envVars:
        # Secret values: use envVars with valueFrom.secretKeyRef
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: <solution>-secrets
              key: database-url
        - name: LLM_GATEWAY_API_KEY
          valueFrom:
            secretKeyRef:
              name: <solution>-secrets
              key: llm-gateway-api-key
    ```

    For file-mount secrets, declare a volume and mount it (the platform's `em-service` chart supports custom mounts via `volumeMounts` and `volumes`):

    ```yaml theme={null}
    api:
      volumeMounts:
        - name: tls
          mountPath: /mnt/secrets/tls
          readOnly: true
      volumes:
        - name: tls
          secret:
            secretName: <solution>-tls
    ```
  </Step>

  <Step title="Read at runtime">
    Environment variables: standard `os.environ`. File mounts: read from disk lazily.

    ```python theme={null}
    import os
    from pathlib import Path

    # Env-injected secret
    db_url = os.environ["DATABASE_URL"]

    # File-mounted secret
    tls_cert = Path("/mnt/secrets/tls/tls.crt").read_text()
    tls_key  = Path("/mnt/secrets/tls/tls.key").read_text()
    ```

    Two rules:

    1. **Never log a secret value.** Wrap reads in repr-safe types or use `pydantic.SecretStr`.
    2. **Don't re-read on every request.** Read once at startup; re-read only when a rotation signal fires (see Stakater Reloader below).
  </Step>

  <Step title="Add a new secret">
    Adding a secret requires two changes:

    1. **Add it to your chart** (the `envVars` snippet above) referencing the key name.
    2. **Add the value to the upstream secret backend** in each environment.

    The upstream backend depends on where you're deploying:

    | Deployment                         | Where to add the secret                                                                            |
    | ---------------------------------- | -------------------------------------------------------------------------------------------------- |
    | Local dev (docker-compose / Kind)  | `.env` (gitignored) or `kubectl create secret generic <solution>-secrets --from-literal=key=value` |
    | Self-hosted (Infisical)            | Infisical UI for the project; ESO syncs to the cluster                                             |
    | Cloud (GCP Secret Manager via ESO) | `gcloud secrets create` + grant Workload Identity SA `secretmanager.secretAccessor`                |

    See [Security › Secrets Management](/security/secrets-management) and [Deployment › Infrastructure › Secrets](/deployment/infrastructure/secrets) for backend setup.
  </Step>

  <Step title="Local development">
    For local-dev secrets:

    * **Use `.env`** (in `.gitignore`) for env vars; `set -a; source .env; set +a` before `uv run`.
    * **Mount fixture files** for file-style secrets via docker-compose `volumes`.
    * **Never commit secrets** — even fixture/dummy ones if they look real. Use values like `dev-only-not-a-real-key`.
    * **Generate fresh keys** for tests (e.g., the `dev-private.pem` from [Authenticate Users](/guides/solution-dev/authenticate-users)).
  </Step>
</Steps>

## Where the values come from

Solutions don't talk to the secret backend directly — the platform's secret pipeline ([ESO](/deployment/infrastructure/secrets) for cloud, [Infisical](/security/secrets-management) for self-hosted) materializes upstream secrets into Kubernetes Secrets in your namespace. Your chart's `secretKeyRef` references those K8s Secrets. The platform handles:

* Authentication to the upstream backend (Workload Identity in GCP, machine identities in Infisical)
* Sync intervals (ESO defaults to 1h; configurable per ExternalSecret)
* Resync on upstream change

You do not see the upstream backend from your service code. Just declare the K8s Secret and key.

## Rotation

The platform pairs secrets with [Stakater Reloader](https://github.com/stakater/Reloader). When the underlying K8s Secret changes (because ESO synced an update from upstream), Reloader rolls your Deployment automatically — *if* you opt in by annotating the Deployment.

`em-service` adds the annotation by default for the secrets it injects via `envFrom`. For custom volume mounts (like `<solution>-tls`), add the annotation manually:

```yaml charts/<solution>/values.yaml theme={null}
api:
  podAnnotations:
    secret.reloader.stakater.com/reload: "<solution>-tls"
```

Verify the rollout fires:

```bash theme={null}
# Update the upstream secret, then watch
kubectl -n em-<solution> rollout status deployment/<solution>-api
```

## Verification

```bash theme={null}
# Confirm the K8s Secret exists with the expected keys
kubectl -n em-<solution> get secret <solution>-secrets -o jsonpath='{.data}' | jq 'keys'

# Confirm the env var reaches the pod
kubectl -n em-<solution> exec deployment/<solution>-api -- printenv DATABASE_URL | head -c 60; echo

# Confirm the file mount reaches the pod (if applicable)
kubectl -n em-<solution> exec deployment/<solution>-api -- ls -la /mnt/secrets/tls
```

If the secret is declared but the pod fails with `secret <name> not found`: ESO/Infisical hasn't synced yet, or the upstream secret doesn't exist. See [Troubleshooting › secrets](/guides/solution-dev/troubleshooting#secrets).

## Next steps

<CardGroup cols={2}>
  <Card title="em-service env model" icon="dharmachakra" href="/deployment/helm/em-service#environment-variable-model">
    Reference for `env` vs `envVars` precedence and dedup rules.
  </Card>

  <Card title="Use shared storage" icon="database" href="/guides/solution-dev/use-shared-storage">
    Storage credentials follow the same pattern as DB creds.
  </Card>

  <Card title="Secrets management" icon="lock" href="/security/secrets-management">
    Platform-side architecture (ESO vs Infisical).
  </Card>

  <Card title="Troubleshooting › secrets" icon="circle-exclamation" href="/guides/solution-dev/troubleshooting#secrets">
    Pod CrashLoopBackOff with "secret not found" and similar.
  </Card>
</CardGroup>