# Delete Agent Source: https://docs.emergence.ai/api-reference/agents/delete-agent /openapi/em-runtime-assets-v2.yaml delete /assets/agents/{resource_uri} Delete an agent permanently. # Get Agent Source: https://docs.emergence.ai/api-reference/agents/get-agent /openapi/em-runtime-assets-v2.yaml get /assets/agents/{resource_uri} Get a registered agent by resource URI. # List Agents Source: https://docs.emergence.ai/api-reference/agents/list-agents /openapi/em-runtime-assets-v2.yaml get /assets/agents List registered agents in project. # Register Agent Source: https://docs.emergence.ai/api-reference/agents/register-agent /openapi/em-runtime-assets-v2.yaml post /assets/agents Register a new agent from an A2A agent card. # Update Agent Source: https://docs.emergence.ai/api-reference/agents/update-agent /openapi/em-runtime-assets-v2.yaml put /assets/agents/{resource_uri} Update an agent's card and metadata (optimistic concurrency via ETag). # Validate Agent Card Source: https://docs.emergence.ai/api-reference/agents/validate-agent-card /openapi/em-runtime-assets-v2.yaml post /assets/agents:validate Dry-run validation of an agent card (no persistence). # Delete Artifact Source: https://docs.emergence.ai/api-reference/artifacts/delete-artifact /openapi/em-runtime-assets-v2.yaml delete /assets/artifacts/{resource_uri} Delete artifact # Download artifact Source: https://docs.emergence.ai/api-reference/artifacts/download-artifact /openapi/em-runtime-assets-v2.yaml get /assets/artifacts/{resource_uri}/download Download an artifact file from object storage # Get Artifact Source: https://docs.emergence.ai/api-reference/artifacts/get-artifact /openapi/em-runtime-assets-v2.yaml get /assets/artifacts/{resource_uri} Get artifact details # List Artifacts Source: https://docs.emergence.ai/api-reference/artifacts/list-artifacts /openapi/em-runtime-assets-v2.yaml get /assets/artifacts List artifacts in project # Preview artifact content Source: https://docs.emergence.ai/api-reference/artifacts/preview-artifact-content /openapi/em-runtime-assets-v2.yaml get /assets/artifacts/{resource_uri}/preview Return a structured preview of an artifact's content. Supports CSV, TSV, plain text, SQL, markdown, HTML, XML, YAML, JSON, and JSONL. # Update Artifact Source: https://docs.emergence.ai/api-reference/artifacts/update-artifact /openapi/em-runtime-assets-v2.yaml put /assets/artifacts/{resource_uri} Update artifact metadata # Upload artifact Source: https://docs.emergence.ai/api-reference/artifacts/upload-artifact /openapi/em-runtime-assets-v2.yaml post /assets/artifacts Upload a file to object storage and create artifact record # Export audit events to CSV Source: https://docs.emergence.ai/api-reference/audit/export-audit-events-to-csv /openapi/em-runtime-governance-v2.yaml get /governance/audit/export Export audit events as a CSV file with pagination. Results are always scoped to the user's organization. Use page/page_size to paginate large exports (max 50k rows per page). # Get audit event by ID Source: https://docs.emergence.ai/api-reference/audit/get-audit-event-by-id /openapi/em-runtime-governance-v2.yaml get /governance/audit/{event_id} Get a specific audit event by its ID. Returns 404 if the event doesn't exist or belongs to a different organization. # Get audit settings Source: https://docs.emergence.ai/api-reference/audit/get-audit-settings /openapi/em-runtime-governance-v2.yaml get /governance/audit/settings Get audit settings (retention period) for the current organization. Returns default settings if none have been configured. # List audit events Source: https://docs.emergence.ai/api-reference/audit/list-audit-events /openapi/em-runtime-governance-v2.yaml get /governance/audit List audit events for the current organization with optional filters. Results are always scoped to the user's organization (no cross-org access). Requires organization admin permissions. # Update audit settings Source: https://docs.emergence.ai/api-reference/audit/update-audit-settings /openapi/em-runtime-governance-v2.yaml patch /governance/audit/settings Update audit settings for the current organization. If retention is reduced, events outside the new retention window are immediately deleted. Requires organization admin permissions. # API Authentication Source: https://docs.emergence.ai/api-reference/authentication How to authenticate with CRAFT APIs using JWT tokens, client credentials, and the auto-generated SDKs. # API Authentication All CRAFT API endpoints require authentication via JWT tokens issued by Keycloak. This page covers how to obtain tokens, authenticate requests, and handle token refresh. ## Authentication Methods For interactive applications, use the **OIDC Authorization Code flow with PKCE**: 1. Redirect the user to the Keycloak authorization endpoint 2. Exchange the authorization code for tokens 3. Include the access token in API requests ``` GET https:///realms//protocol/openid-connect/auth ?response_type=code &client_id= &redirect_uri= &scope=openid email profile &code_challenge= &code_challenge_method=S256 ``` After the user authenticates, exchange the code: ```bash theme={null} curl -X POST \ "https:///realms//protocol/openid-connect/token" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=authorization_code" \ -d "client_id=" \ -d "code=" \ -d "redirect_uri=" \ -d "code_verifier=" ``` For service accounts, CI/CD pipelines, and automation, use the **client credentials grant**: ```bash theme={null} curl -X POST \ "https:///realms//protocol/openid-connect/token" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=" \ -d "client_secret=" ``` Response: ```json theme={null} { "access_token": "", "expires_in": 300, "token_type": "Bearer", "scope": "openid email profile" } ``` Client credentials tokens do not include a refresh token. Obtain a new access token before expiration. For local development, use the `make get-token` shortcut in the em-runtime repository: ```bash theme={null} # Get a token for the org admin user make get-token # Get a platform admin token (Keycloak master realm) make get-platform-token ``` Or use `curl` directly: ```bash theme={null} TOKEN=$(curl -s -X POST \ "http://localhost:8080/realms//protocol/openid-connect/token" \ -d "grant_type=password" \ -d "client_id=admin-cli" \ -d "username=admin" \ -d "password=admin" \ | jq -r '.access_token') ``` The password grant is intended for development only. Never use it in production -- use OIDC with PKCE or client credentials instead. ## Using the Token Include the access token in the `Authorization` header of every API request: ```bash theme={null} curl -X GET "https:///api/governance/organizations" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " ``` ### Required Headers | Header | Required | Description | | --------------- | ---------------- | -------------------------- | | `Authorization` | Always | `Bearer ` | | `X-Project-ID` | Most endpoints | UUID of the target project | | `Content-Type` | Write operations | `application/json` | The `X-Project-ID` header determines which project the request is scoped to. A single JWT token can access multiple projects within the same organization. The project ID is never embedded in the JWT. ## Token Refresh Access tokens have a short lifetime (typically 5 minutes). Use the refresh token to obtain new access tokens without re-authentication: ```bash theme={null} curl -X POST \ "https:///realms//protocol/openid-connect/token" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=refresh_token" \ -d "client_id=" \ -d "refresh_token=" ``` ### Refresh Strategy Track the `exp` claim in the JWT payload. Plan to refresh when 75% of the token lifetime has elapsed. Refresh the token before it expires to avoid 401 errors during in-flight requests. If the refresh token is expired or revoked, redirect the user to the login flow. For service accounts, obtain a new token via client credentials. ## SDK Authentication The auto-generated SDKs follow the [openapi-python-client](https://github.com/openapi-generators/openapi-python-client) and `openapi-generator` patterns — an `AuthenticatedClient` with a `token` constructor arg + per-endpoint module functions, **not** an object/fluent client. See the [Python SDK](/sdks/python) and [TypeScript SDK](/sdks/typescript) overviews for the full client model. ```python theme={null} from em_runtime_governance_sdk import AuthenticatedClient from em_runtime_governance_sdk.api.organizations import list_organizations client = AuthenticatedClient( base_url="https:///api/governance", token=access_token, ) orgs = await list_organizations.asyncio(client=client) # Token refresh: pop the expired client and rebuild with a fresh token. # The SDK has no built-in onTokenExpired callback; wrap your calls in a # helper that catches 401, refreshes, and retries with a new client. ``` ```typescript theme={null} import { Client } from '@emergence-ai/em-runtime-governance-sdk'; import { listOrganizations } from '@emergence-ai/em-runtime-governance-sdk/api'; const client = new Client({ BASE: 'https:///api/governance', TOKEN: accessToken, }); const orgs = await listOrganizations({ client }); ``` ## Token Claims The JWT access token contains these claims used by the platform: ```json theme={null} { "sub": "f47ac10b-58cc-4372-a567-0e02b2c3d479", "iss": "https://keycloak.example.com/realms/org-123", "aud": "emergence-platform", "exp": 1712150400, "iat": 1712150100, "groups": ["admins", "developers"], "email": "user@example.com", "preferred_username": "jdoe" } ``` | Claim | Platform Usage | | -------- | ---------------------------------------------------- | | `sub` | Unique user identifier for audit logs | | `iss` | Organization ID derived from realm in the issuer URL | | `groups` | Mapped to OpenFGA roles for permission checks | | `exp` | Token expiration validation | ## Error Responses Authentication failures return structured error responses: | Status | Code | Description | | ------ | --------------------- | ----------------------------------------------------------- | | `401` | `UNAUTHENTICATED` | No token provided or token signature invalid | | `401` | `TOKEN_EXPIRED` | Access token has expired; refresh and retry | | `403` | `FORBIDDEN` | Token is valid but user lacks permission for this operation | | `429` | `RATE_LIMIT_EXCEEDED` | Too many requests; check `Retry-After` header | ## Next Steps Review the full API structure, endpoints, and rate limits. Understand all error response formats and troubleshooting steps. Configure SSO for enterprise identity providers. Set up roles and permissions for API access control. # Get bootstrap status Source: https://docs.emergence.ai/api-reference/bootstrap/get-bootstrap-status /openapi/em-runtime-governance-v2.yaml get /governance/bootstrap/status Returns information about the bootstrap configuration and default organization. Bootstrap happens automatically during application startup. The default organization id is the Keycloak-assigned UUID (looked up by configured display name). Requires platform developer role. # Add context items Source: https://docs.emergence.ai/api-reference/chat-sessions/add-context-items /openapi/em-talk2data-v2.yaml post /talk2data/chat/sessions/{session_id}/context Add or update datasources, artifacts, and/or attachments on a session. Upserts by (type, resource_uri). Returns the full updated context. 403 if not owned, 404 if not found. # Delete feedback Source: https://docs.emergence.ai/api-reference/chat-sessions/delete-feedback /openapi/em-talk2data-v2.yaml delete /talk2data/chat/sessions/{session_id}/turns/{turn_id}/feedback Delete the current user's feedback for the turn. 403 if not owned, 404 if session/turn/feedback not found. # Delete session Source: https://docs.emergence.ai/api-reference/chat-sessions/delete-session /openapi/em-talk2data-v2.yaml delete /talk2data/chat/sessions/{session_id} Delete a session and cascade-delete messages, artifacts, and feedback. 403 if not owned, 404 if not found. # Get session Source: https://docs.emergence.ai/api-reference/chat-sessions/get-session /openapi/em-talk2data-v2.yaml get /talk2data/chat/sessions/{session_id} Get session metadata with turn_count, message_count, artifact_count. Returns 403 if session does not belong to user/project, 404 if not found. # Get session context Source: https://docs.emergence.ai/api-reference/chat-sessions/get-session-context /openapi/em-talk2data-v2.yaml get /talk2data/chat/sessions/{session_id}/context Get all datasources, artifacts, and attachments pinned to this session. 403 if not owned, 404 if not found. # Get turn by ID Source: https://docs.emergence.ai/api-reference/chat-sessions/get-turn-by-id /openapi/em-talk2data-v2.yaml get /talk2data/chat/sessions/{session_id}/turns/{turn_id} Get a single turn with all its messages and ``task_updates`` (same shape as list turns). 403 if session not owned, 404 if session or turn not found. # List session artifacts Source: https://docs.emergence.ai/api-reference/chat-sessions/list-session-artifacts /openapi/em-talk2data-v2.yaml get /talk2data/chat/sessions/{session_id}/artifacts Paginated artifacts for a session. Optional artifact_type filter (parquet, chart). 403 if session not owned, 404 if not found. # List session turns Source: https://docs.emergence.ai/api-reference/chat-sessions/list-session-turns /openapi/em-talk2data-v2.yaml get /talk2data/chat/sessions/{session_id}/turns Paginated turns (user + assistant messages grouped by turn_id). Each turn includes ``task_updates`` from ``talk2data.task_updates`` (A2A status stream), peer to ``messages``. Turn order by first message time: `order=asc` (oldest first, default) or `order=desc` (newest first). 403 if session not owned, 404 if not found. # List sessions Source: https://docs.emergence.ai/api-reference/chat-sessions/list-sessions /openapi/em-talk2data-v2.yaml get /talk2data/chat/sessions List sessions for the authenticated user and project. Paginated via page and limit. # List turn artifacts Source: https://docs.emergence.ai/api-reference/chat-sessions/list-turn-artifacts /openapi/em-talk2data-v2.yaml get /talk2data/chat/sessions/{session_id}/turns/{turn_id}/artifacts Paginated artifacts for a turn. Optional artifact_type filter (parquet, chart). 403 if session not owned, 404 if session or turn not found. # Remove context items Source: https://docs.emergence.ai/api-reference/chat-sessions/remove-context-items /openapi/em-talk2data-v2.yaml delete /talk2data/chat/sessions/{session_id}/context Remove one or more context items by (item_type, resource_uri). Returns the full updated context. 403 if not owned, 404 if not found. # Rename session Source: https://docs.emergence.ai/api-reference/chat-sessions/rename-session /openapi/em-talk2data-v2.yaml patch /talk2data/chat/sessions/{session_id} Update session title. 403 if session not owned, 404 if not found, 422 if title empty/blank or exceeds 255 chars. # Submit or update feedback Source: https://docs.emergence.ai/api-reference/chat-sessions/submit-or-update-feedback /openapi/em-talk2data-v2.yaml post /talk2data/chat/sessions/{session_id}/turns/{turn_id}/feedback Upsert feedback for a turn. 201 created, 200 updated. 403 if session not owned, 404 if session/turn not found, 422 if rating invalid. # Post Chat Cancel Source: https://docs.emergence.ai/api-reference/chat/post-chat-cancel /openapi/em-talk2data-v2.yaml post /talk2data/chat/messages/{task_id}/cancel Cancel an in-progress task. Sends A2A tasks/cancel to the orchestrator. Only allowed if the task's session belongs to the same user and project. # Post Chat Messages Source: https://docs.emergence.ai/api-reference/chat/post-chat-messages /openapi/em-talk2data-v2.yaml post /talk2data/chat/messages Accept a message and kick off background A2A processing. # Stream Chat Messages Source: https://docs.emergence.ai/api-reference/chat/stream-chat-messages /openapi/em-talk2data-v2.yaml get /talk2data/chat/messages/sessions/{session_id}/turns/{turn_id}/stream Stream persisted SSE events for a turn from durable storage. # Add Memory To Pack Source: https://docs.emergence.ai/api-reference/context-pack/add-memory-to-pack /openapi/em-memory-service-v2.yaml post /context-packs/{context_pack_name}/memories # Archive Pack Source: https://docs.emergence.ai/api-reference/context-pack/archive-pack /openapi/em-memory-service-v2.yaml post /context-packs/{context_pack_name}/archive # Create Pack Source: https://docs.emergence.ai/api-reference/context-pack/create-pack /openapi/em-memory-service-v2.yaml post /context-packs # Deprecate Pack Source: https://docs.emergence.ai/api-reference/context-pack/deprecate-pack /openapi/em-memory-service-v2.yaml post /context-packs/{context_pack_name}/deprecate # Get Pack By Name Source: https://docs.emergence.ai/api-reference/context-pack/get-pack-by-name /openapi/em-memory-service-v2.yaml get /context-packs/{context_pack_name} # List Pack Memories Source: https://docs.emergence.ai/api-reference/context-pack/list-pack-memories /openapi/em-memory-service-v2.yaml get /context-packs/{context_pack_name}/memories # List Packs Source: https://docs.emergence.ai/api-reference/context-pack/list-packs /openapi/em-memory-service-v2.yaml get /context-packs # Permanent Delete Pack Source: https://docs.emergence.ai/api-reference/context-pack/permanent-delete-pack /openapi/em-memory-service-v2.yaml post /context-packs/{context_pack_name}/permanent-delete # Reactivate Pack Source: https://docs.emergence.ai/api-reference/context-pack/reactivate-pack /openapi/em-memory-service-v2.yaml post /context-packs/{context_pack_name}/reactivate # Soft Delete Pack Source: https://docs.emergence.ai/api-reference/context-pack/soft-delete-pack /openapi/em-memory-service-v2.yaml delete /context-packs/{context_pack_name} # Update Pack Source: https://docs.emergence.ai/api-reference/context-pack/update-pack /openapi/em-memory-service-v2.yaml patch /context-packs/{context_pack_name} # Add a memory to a context pack Source: https://docs.emergence.ai/api-reference/context-packs/add-a-memory-to-a-context-pack /openapi/em-runtime-utils-v2.yaml post /utils/context-packs/{name}/memories Add a new memory to the context pack. # Archive a context pack Source: https://docs.emergence.ai/api-reference/context-packs/archive-a-context-pack /openapi/em-runtime-utils-v2.yaml post /utils/context-packs/{name}/archive Archive a context pack. # Create a new context pack Source: https://docs.emergence.ai/api-reference/context-packs/create-a-new-context-pack /openapi/em-runtime-utils-v2.yaml post /utils/context-packs Create a new context pack in the project. # Deprecate a context pack Source: https://docs.emergence.ai/api-reference/context-packs/deprecate-a-context-pack /openapi/em-runtime-utils-v2.yaml post /utils/context-packs/{name}/deprecate Deprecate a context pack. # Get a context pack by name Source: https://docs.emergence.ai/api-reference/context-packs/get-a-context-pack-by-name /openapi/em-runtime-utils-v2.yaml get /utils/context-packs/{name} Get a context pack by name within the project. # Get a memory by name within a context pack Source: https://docs.emergence.ai/api-reference/context-packs/get-a-memory-by-name-within-a-context-pack /openapi/em-runtime-utils-v2.yaml get /utils/context-packs/{name}/memories/{memory_name} Get a memory by name within a context pack. # List context packs Source: https://docs.emergence.ai/api-reference/context-packs/list-context-packs /openapi/em-runtime-utils-v2.yaml get /utils/context-packs List context packs in the project with optional filters. # List memories in a context pack Source: https://docs.emergence.ai/api-reference/context-packs/list-memories-in-a-context-pack /openapi/em-runtime-utils-v2.yaml get /utils/context-packs/{name}/memories List memories in the context pack. # Permanently delete a context pack Source: https://docs.emergence.ai/api-reference/context-packs/permanently-delete-a-context-pack /openapi/em-runtime-utils-v2.yaml post /utils/context-packs/{name}/permanent-delete Permanently delete a context pack and all its memories. # Reactivate a context pack Source: https://docs.emergence.ai/api-reference/context-packs/reactivate-a-context-pack /openapi/em-runtime-utils-v2.yaml post /utils/context-packs/{name}/reactivate Reactivate an archived or deprecated context pack. # Soft delete a context pack Source: https://docs.emergence.ai/api-reference/context-packs/soft-delete-a-context-pack /openapi/em-runtime-utils-v2.yaml delete /utils/context-packs/{name} Soft delete a context pack (can be recovered). # Update a context pack Source: https://docs.emergence.ai/api-reference/context-packs/update-a-context-pack /openapi/em-runtime-utils-v2.yaml patch /utils/context-packs/{name} Update a context pack's description, metadata, or config. # Batch-fetch latest column profiles by table and/or column FQN Source: https://docs.emergence.ai/api-reference/data-assets/batch-fetch-latest-column-profiles-by-table-andor-column-fqn /openapi/em-data-readiness-v2.yaml post /api/data-readiness/data-assets/profiles/columns/batch Return the latest column profile for every column matched by table_fqns and/or column_fqns. At least one filter is required (validated in the request model). Returns an empty list when nothing is profiled — does NOT 404. # Batch-fetch latest table profiles by table FQN Source: https://docs.emergence.ai/api-reference/data-assets/batch-fetch-latest-table-profiles-by-table-fqn /openapi/em-data-readiness-v2.yaml post /api/data-readiness/data-assets/profiles/tables/batch Return the latest table profile for each given table FQN. Returns an empty list when nothing is profiled — does NOT 404. # Clear DQ and/or metadata scores for a table Source: https://docs.emergence.ai/api-reference/data-assets/clear-dq-andor-metadata-scores-for-a-table /openapi/em-data-readiness-v2.yaml delete /api/data-readiness/data-assets/tables/scores Clear one or both score types for a table in a single call. Use scope=data_quality to clear DQ tags and rules, scope=metadata to clear the metadata score, or scope=all to clear everything (the Clear All action). Previous data is preserved for history. Running a new assessment restores the score. # Get a column by FQN Source: https://docs.emergence.ai/api-reference/data-assets/get-a-column-by-fqn /openapi/em-data-readiness-v2.yaml get /api/data-readiness/data-assets/columns/by-fqn Get a specific column by its fully qualified name (FQN). The FQN format is: service.database.schema.table.column Example: PostgreSQL.dvdrental.public.customer.email Args: fqn: Fully qualified name of the column. ctx: Request context for authentication. service: Data assets service instance. Returns: Column domain model. Raises: HTTPException: 404 if column not found. # Get a database by FQN Source: https://docs.emergence.ai/api-reference/data-assets/get-a-database-by-fqn /openapi/em-data-readiness-v2.yaml get /api/data-readiness/data-assets/databases/by-fqn Get a specific database by its fully qualified name (FQN). Args: fqn: Fully qualified name of the database. ctx: Request context for authentication. service: Data assets service instance. Returns: Database domain model. Raises: HTTPException: 404 if database not found. # Get a schema by FQN Source: https://docs.emergence.ai/api-reference/data-assets/get-a-schema-by-fqn /openapi/em-data-readiness-v2.yaml get /api/data-readiness/data-assets/schemas/by-fqn Get a specific schema by its fully qualified name (FQN). The FQN format is: service.database.schema Example: PostgreSQL.dvdrental.public Args: fqn: Fully qualified name of the schema. ctx: Request context for authentication. service: Data assets service instance. workflow_asset_service: Workflow asset integration service instance. profile_service: Profile service for fetching profile data. include_tables: Whether to include tables in the schema (default: true). include_pending_workflow: Whether to include pending workflow info (default: false). include_profile: Whether to include profile summary per table (default: false). search: Filter tables by name or description (case-insensitive). Only applied when include_tables=true. Returns: Schema domain model with tables if requested, SchemaWithProfileResponse if include_profile is true, or SchemaWithWorkflowResponse if include_pending_workflow is true. Raises: HTTPException: 404 if schema not found. # Get a table by FQN Source: https://docs.emergence.ai/api-reference/data-assets/get-a-table-by-fqn /openapi/em-data-readiness-v2.yaml get /api/data-readiness/data-assets/tables/by-fqn Get a specific table by its fully qualified name (FQN). The FQN format is: service.database.schema.table Example: PostgreSQL.dvdrental.public.customer Args: fqn: Fully qualified name of the table. ctx: Request context for authentication. service: Data assets service instance. workflow_asset_service: Workflow asset integration service instance. include_pending_workflow: Whether to include pending workflow info (default: false). include_dq: Whether to include DQ tags, rules, and executions (default: false). Returns: Table domain model with columns if requested, or TableWithWorkflowResponse if include_pending_workflow is true. Raises: HTTPException: 404 if table not found. # Get latest column profile by FQN Source: https://docs.emergence.ai/api-reference/data-assets/get-latest-column-profile-by-fqn /openapi/em-data-readiness-v2.yaml get /api/data-readiness/data-assets/profiles/columns/by-fqn Get the most recent profiling data for a column by its fully qualified name. Args: fqn: Fully qualified name of the column. ctx: Request context for authentication. profile_service: Profile service instance. Returns: Latest column profile data. Raises: HTTPException: 404 if no profile found for the column. # Get latest table profile by FQN Source: https://docs.emergence.ai/api-reference/data-assets/get-latest-table-profile-by-fqn /openapi/em-data-readiness-v2.yaml get /api/data-readiness/data-assets/profiles/tables/by-fqn Get the most recent profiling data for a table by its fully qualified name. Args: fqn: Fully qualified name of the table. ctx: Request context for authentication. profile_service: Profile service instance. Returns: Latest table profile data. Raises: HTTPException: 404 if no profile found for the table. # Update DQ tags for a table Source: https://docs.emergence.ai/api-reference/data-assets/update-dq-tags-for-a-table /openapi/em-data-readiness-v2.yaml put /api/data-readiness/data-assets/tables/dq-tags Update (delete) DQ tags for columns in a table. This is a delete-only operation: for each column in the request, tags in the DB that are NOT in the request will be deleted. Columns not in the request are unchanged. Request tags not in DB are ignored. # Create Data Connection Source: https://docs.emergence.ai/api-reference/data_connections/create-data-connection /openapi/em-runtime-assets-v2.yaml post /assets/data Create data connection # Delete Data Connection Source: https://docs.emergence.ai/api-reference/data_connections/delete-data-connection /openapi/em-runtime-assets-v2.yaml delete /assets/data/{resource_uri} Delete data connection # Get Data Connection Source: https://docs.emergence.ai/api-reference/data_connections/get-data-connection /openapi/em-runtime-assets-v2.yaml get /assets/data/{resource_uri} Get data connection # Get Data Connection Secret Source: https://docs.emergence.ai/api-reference/data_connections/get-data-connection-secret /openapi/em-runtime-assets-v2.yaml get /assets/data/{resource_uri}/secret Get data connection secrets # List Connection Types Source: https://docs.emergence.ai/api-reference/data_connections/list-connection-types /openapi/em-runtime-assets-v2.yaml get /assets/data/types List all data connection types with their credential schemas # List Data Connections Source: https://docs.emergence.ai/api-reference/data_connections/list-data-connections /openapi/em-runtime-assets-v2.yaml get /assets/data Get data connections # Update Data Connection Source: https://docs.emergence.ai/api-reference/data_connections/update-data-connection /openapi/em-runtime-assets-v2.yaml put /assets/data/{resource_uri} Update data connection # Verify Connection Before Save Source: https://docs.emergence.ai/api-reference/data_connections/verify-connection-before-save /openapi/em-runtime-assets-v2.yaml post /assets/data/verify Verify data connection credentials before creating the connection # Verify Data Connection Source: https://docs.emergence.ai/api-reference/data_connections/verify-data-connection /openapi/em-runtime-assets-v2.yaml post /assets/data/{resource_uri}/verify Verify data connection credentials and accessibility # Send an HTML email to one or more recipients Source: https://docs.emergence.ai/api-reference/email/send-an-html-email-to-one-or-more-recipients /openapi/em-runtime-utils-v2.yaml post /utils/email Send an HTML email to one or more recipients. The `body` field is treated as HTML. # Generate an embedding vector Source: https://docs.emergence.ai/api-reference/embeddings/generate-an-embedding-vector /openapi/em-runtime-search-v2.yaml post /search/embeddings Generates an embedding vector for the provided text using the configured embedding provider (default: Google Gemini). Used by publishing services to embed entity text before indexing, and by the global search endpoint to vectorise natural language queries. # Error Codes Source: https://docs.emergence.ai/api-reference/errors Error response format, error codes, and troubleshooting guide for CRAFT APIs. # Error Codes All CRAFT APIs return consistent error responses. This page documents the error format, all error codes, and troubleshooting guidance for each. ## Error Response Format Error responses follow a standard JSON structure: ```json theme={null} { "error": { "code": "RESOURCE_NOT_FOUND", "message": "Agent with ID '550e8400-e29b-41d4-a716-446655440000' not found.", "status": 404, "details": { "resource_type": "artifact", "resource_id": "550e8400-e29b-41d4-a716-446655440000" }, "request_id": "req-a1b2c3d4-e5f6-7890-abcd-ef1234567890", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736" } } ``` | Field | Type | Description | | ------------ | ------- | ----------------------------------------------- | | `code` | string | Machine-readable error code | | `message` | string | Human-readable error description | | `status` | integer | HTTP status code | | `details` | object | Additional context (varies by error type) | | `request_id` | string | Unique request identifier for support | | `trace_id` | string | OpenTelemetry trace ID for end-to-end debugging | ## HTTP Status Codes | Status | Meaning | When Returned | | ------ | --------------------- | --------------------------------------------------------------------- | | `400` | Bad Request | Invalid request body, missing required fields, malformed parameters | | `401` | Unauthorized | Missing, expired, or invalid JWT token | | `403` | Forbidden | Valid token but insufficient permissions | | `404` | Not Found | Resource does not exist or is not accessible | | `409` | Conflict | Resource already exists or version conflict | | `412` | Precondition Failed | `If-Match` provided but does not match current resource version (PUT) | | `413` | Payload Too Large | Uploaded body exceeds the server's size limit (artifact upload) | | `422` | Unprocessable Entity | Request is well-formed but semantically invalid | | `428` | Precondition Required | PUT without `If-Match` header (concurrency control required) | | `429` | Too Many Requests | Rate limit exceeded | | `500` | Internal Server Error | Unexpected server error | | `502` | Bad Gateway | Upstream service (Keycloak, OpenFGA) unavailable | | `503` | Service Unavailable | Service is starting up or shutting down | ## Error Codes Reference The OpenAPI specifications define a base `ErrorCode` enum with 9 values: `INVALID_REQUEST`, `INVALID_TOKEN`, `INSUFFICIENT_PERMISSIONS`, `RESOURCE_NOT_FOUND`, `RESOURCE_ALREADY_EXISTS`, `VALIDATION_ERROR`, `RATE_LIMIT_EXCEEDED`, `SERVICE_UNAVAILABLE`, `INTERNAL_ERROR`. The implementation may return more specific codes within each category as documented below. ### Authentication Errors (401) | Code | Message | Troubleshooting | | ----------------- | ----------------------------------- | ------------------------------------------------------------------------------------- | | `UNAUTHENTICATED` | No authentication token provided | Include `Authorization: Bearer ` header | | `TOKEN_EXPIRED` | Access token has expired | Refresh the token or obtain a new one | | `TOKEN_INVALID` | Token signature verification failed | Ensure the token was issued by the correct Keycloak realm | | `TOKEN_MALFORMED` | Token format is invalid | Check that the token is a valid JWT (three base64-encoded segments separated by dots) | ### Authorization Errors (403) | Code | Message | Troubleshooting | | ----------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | `FORBIDDEN` | Insufficient permissions for this operation | Check your role assignment on the target project; see [RBAC Configuration](/guides/rbac-configuration) | | `PROJECT_ACCESS_DENIED` | No access to the specified project | Verify the project ID in the request matches a project you have been granted access to | | `ORG_ACCESS_DENIED` | Token does not belong to this organization | Authenticate against the correct Keycloak realm for the target organization | ### Resource Errors (404, 409) | Code | Message | Troubleshooting | | ------------------------- | --------------------------------------------- | --------------------------------------------------------------------- | | `RESOURCE_NOT_FOUND` | Resource not found | Verify the resource ID and ensure you have `can_read` permission | | `RESOURCE_ALREADY_EXISTS` | Resource with this identifier already exists | Use a unique name or identifier; use PUT to update existing resources | | `VERSION_CONFLICT` | Resource has been modified by another request | Re-fetch the resource and retry with the current version | ### Validation Errors (400, 422) | Code | Message | Troubleshooting | | ------------------------ | ------------------------------ | ----------------------------------------------------------------- | | `VALIDATION_ERROR` | Request body validation failed | Check the `details` field for specific field-level errors | | `INVALID_PARAMETER` | Invalid query parameter | Verify parameter name, type, and allowed values | | `MISSING_REQUIRED_FIELD` | Required field is missing | Include all required fields in the request body | | `INVALID_FORMAT` | Field value format is invalid | Check the expected format (UUID, URL, email, etc.) | | `SEARCH_QUERY_INVALID` | Malformed search query | Use plain text for search queries; special operators are reserved | ### Rate Limiting Errors (429) | Code | Message | Troubleshooting | | --------------------- | --------------------- | --------------------------------------------------------------------------- | | `RATE_LIMIT_EXCEEDED` | Rate limit exceeded | Wait for the `Retry-After` header duration; implement exponential backoff | | `QUOTA_EXCEEDED` | Tenant quota exceeded | Contact your administrator to increase the quota for the exceeded dimension | ### Server Errors (500, 502, 503) | Code | Message | Troubleshooting | | ---------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------- | | `INTERNAL_ERROR` | An unexpected error occurred | Retry the request; if persistent, report with `request_id` and `trace_id` | | `UPSTREAM_UNAVAILABLE` | Upstream service is unavailable | Keycloak or OpenFGA may be down; check service health. Governance down causes Assets/Utils to return 403 | | `SERVICE_STARTING` | Service is initializing | Wait for the service to complete startup; check readiness probe | | `DATABASE_ERROR` | Database operation failed | Retry the request; if persistent, check database connectivity | ## Handling Errors in Code ```python theme={null} from em_runtime_assets_sdk import AuthenticatedClient from em_runtime_assets_sdk.api.artifacts import get_artifact from em_runtime_assets_sdk.errors import UnexpectedStatus client = AuthenticatedClient( base_url="https:///api/assets", token=token, ) try: agent = await get_artifact.asyncio(client=client, artifact_uri=artifact_uri) except UnexpectedStatus as e: if e.status_code == 401: # Token expired -- refresh and rebuild the client (token is immutable). token = await refresh_token() client = AuthenticatedClient( base_url="https:///api/assets", token=token ) agent = await get_artifact.asyncio(client=client, artifact_uri=artifact_uri) elif e.status_code == 404: print(f"Not found: {e.content!r}") elif e.status_code == 429: # Rate limited -- read Retry-After from response headers retry_after = int(e.response.headers.get("Retry-After", "1")) await asyncio.sleep(retry_after) agent = await get_artifact.asyncio(client=client, artifact_uri=artifact_uri) else: print(f"HTTP {e.status_code}: {e.content!r}") raise ``` `UnexpectedStatus.content` is raw `bytes`; decode and `json.loads()` to read the envelope (`{"error": {"code", "message", "request_id"}}`). ```typescript theme={null} import { Client, ApiError } from '@emergence-ai/em-runtime-assets-sdk'; import { getArtifact } from '@emergence-ai/em-runtime-assets-sdk/api'; const client = new Client({ BASE: 'https:///api/assets', TOKEN: accessToken, }); try { const agent = await getArtifact({ client, artifactUri }); } catch (error) { if (error instanceof ApiError) { const body = error.body as | { error?: { code?: string; message?: string; request_id?: string } } | undefined; switch (error.status) { case 401: // Token expired -- refresh and rebuild the client break; case 404: console.log(`Not found: ${body?.error?.message}`); break; case 429: { const retryAfter = Number(error.headers?.['retry-after'] ?? '1'); await sleep(retryAfter * 1000); break; } default: console.error(`HTTP ${error.status} (${body?.error?.code}): ${body?.error?.message}`); console.error(`Request ID: ${body?.error?.request_id}`); } } } ``` ```bash theme={null} # Check the HTTP status code HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \ -H "Authorization: Bearer $TOKEN" \ "https:///api/assets/artifacts/$ARTIFACT_URI") if [ "$HTTP_STATUS" -eq 401 ]; then echo "Token expired, refreshing..." TOKEN=$(refresh_token) elif [ "$HTTP_STATUS" -eq 404 ]; then echo "Agent not found" elif [ "$HTTP_STATUS" -eq 429 ]; then echo "Rate limited, waiting..." sleep 60 fi ``` ## Retry Strategy For transient errors (429, 500, 502, 503), implement exponential backoff with jitter: | Attempt | Wait Time | Max Wait | | ----------- | ------------------ | ---------- | | 1st retry | 1 second + jitter | 2 seconds | | 2nd retry | 2 seconds + jitter | 4 seconds | | 3rd retry | 4 seconds + jitter | 8 seconds | | 4th retry | 8 seconds + jitter | 16 seconds | | Max retries | 5 | 30 seconds | For 429 errors, always respect the `Retry-After` header rather than using exponential backoff. The header provides the exact wait time until the rate limit window resets. ## Getting Help When reporting an issue, include: 1. The `request_id` from the error response 2. The `trace_id` for end-to-end correlation 3. The timestamp of the request 4. The full error response body ## Next Steps Review the full API structure and endpoints. Learn how to obtain and manage JWT tokens. Use the Python SDK with built-in error handling. Use the TypeScript SDK with built-in error handling. # Delete File Source: https://docs.emergence.ai/api-reference/files/delete-file /openapi/em-runtime-assets-v2.yaml delete /assets/files/{resource_uri} Delete file # Download file Source: https://docs.emergence.ai/api-reference/files/download-file /openapi/em-runtime-assets-v2.yaml get /assets/files/{resource_uri}/download Download a file from object storage # Get File Source: https://docs.emergence.ai/api-reference/files/get-file /openapi/em-runtime-assets-v2.yaml get /assets/files/{resource_uri} Get file details # List Files Source: https://docs.emergence.ai/api-reference/files/list-files /openapi/em-runtime-assets-v2.yaml get /assets/files List files in project # Update File Source: https://docs.emergence.ai/api-reference/files/update-file /openapi/em-runtime-assets-v2.yaml put /assets/files/{resource_uri} Update file metadata # Upload file Source: https://docs.emergence.ai/api-reference/files/upload-file /openapi/em-runtime-assets-v2.yaml post /assets/files Upload a file to object storage and create file record # Bulk upsert FK edges Source: https://docs.emergence.ai/api-reference/fk-graph/bulk-upsert-fk-edges /openapi/em-runtime-utils-v2.yaml put /utils/{resource_type}/{resource_uri}/fk-edges Create or update FK edges for a resource. Existing edges are updated, new edges are created. # Delete FK edges Source: https://docs.emergence.ai/api-reference/fk-graph/delete-fk-edges /openapi/em-runtime-utils-v2.yaml delete /utils/{resource_type}/{resource_uri}/fk-edges Delete all FK edges for a resource. # Expand table selection via FK relationships Source: https://docs.emergence.ai/api-reference/fk-graph/expand-table-selection-via-fk-relationships /openapi/em-runtime-utils-v2.yaml post /utils/{resource_type}/{resource_uri}/fk-edges/expand Given a list of table FQNs, discover related tables via FK traversal (BFS). # Find shortest paths between tables Source: https://docs.emergence.ai/api-reference/fk-graph/find-shortest-paths-between-tables /openapi/em-runtime-utils-v2.yaml post /utils/{resource_type}/{resource_uri}/fk-edges/find-paths Find shortest FK paths between all pairs of input tables. # List FK edges Source: https://docs.emergence.ai/api-reference/fk-graph/list-fk-edges /openapi/em-runtime-utils-v2.yaml get /utils/{resource_type}/{resource_uri}/fk-edges List FK edges for a resource, optionally filtered by table FQN. # Add a tag to a glossary term Source: https://docs.emergence.ai/api-reference/glossary/add-a-tag-to-a-glossary-term /openapi/em-data-readiness-v2.yaml post /api/data-readiness/glossary/terms/{term_id}/tags Add a tag to a glossary term. # Create a glossary term Source: https://docs.emergence.ai/api-reference/glossary/create-a-glossary-term /openapi/em-data-readiness-v2.yaml post /api/data-readiness/glossary/terms Create a new glossary term. # Delete a glossary term Source: https://docs.emergence.ai/api-reference/glossary/delete-a-glossary-term /openapi/em-data-readiness-v2.yaml delete /api/data-readiness/glossary/terms/{term_id} Soft-delete a glossary term. # Get a glossary term by ID Source: https://docs.emergence.ai/api-reference/glossary/get-a-glossary-term-by-id /openapi/em-data-readiness-v2.yaml get /api/data-readiness/glossary/terms/{term_id} Get a single glossary term by its ID. # Get alphabet index with term counts Source: https://docs.emergence.ai/api-reference/glossary/get-alphabet-index-with-term-counts /openapi/em-data-readiness-v2.yaml get /api/data-readiness/glossary/terms/index Return term counts grouped by first letter for the A-Z index navigation. # List glossary terms Source: https://docs.emergence.ai/api-reference/glossary/list-glossary-terms /openapi/em-data-readiness-v2.yaml get /api/data-readiness/glossary/terms List glossary terms across all DB connections with optional filters. # Update a glossary term Source: https://docs.emergence.ai/api-reference/glossary/update-a-glossary-term /openapi/em-data-readiness-v2.yaml patch /api/data-readiness/glossary/terms/{term_id} Update an existing glossary term (partial update). # Verify a glossary term Source: https://docs.emergence.ai/api-reference/glossary/verify-a-glossary-term /openapi/em-data-readiness-v2.yaml post /api/data-readiness/glossary/terms/{term_id}/verify Verify a glossary term (sets status to Verified). # Verify multiple glossary terms Source: https://docs.emergence.ai/api-reference/glossary/verify-multiple-glossary-terms /openapi/em-data-readiness-v2.yaml post /api/data-readiness/glossary/terms/bulk-verify Verify multiple glossary terms at once. # Health Check Source: https://docs.emergence.ai/api-reference/health-check /openapi/em-talk2data-v2.yaml get /talk2data/health # Health Source: https://docs.emergence.ai/api-reference/health/health /openapi/em-runtime-search-v2.yaml get /health Health check endpoint. # Health Check Source: https://docs.emergence.ai/api-reference/health/health-check /openapi/em-data-readiness-v2.yaml get /api/data-readiness/health # Service Info Source: https://docs.emergence.ai/api-reference/info/service-info /openapi/em-runtime-search-v2.yaml get /search/ Service information endpoint. # API Overview Source: https://docs.emergence.ai/api-reference/introduction Overview of CRAFT REST APIs including base URLs, versioning, rate limits, and service endpoints for Governance, Assets, and Utils. # API Overview CRAFT exposes RESTful APIs through five services: **Governance** (identity and permissions), **Assets** (data connections and files), **Utils** (scheduling, context packs, and memories), **Talk2Data** (conversational data analytics), and **Data Readiness** (data profiling and governance). All APIs follow consistent conventions for authentication, versioning, and error handling. ## Base URLs Each platform service runs on a dedicated internal port. In production, services are accessed via path-based routing through a Gateway: | Service | Internal Port | Path Prefix | Responsibility | | ------------------ | ------------- | ----------------- | --------------------------------------------- | | **Governance** | 8000 | `/governance` | Organizations, projects, permissions, secrets | | **Assets** | 8000 | `/assets` | Artifacts, data connections, files | | **Utils** | 8000 | `/utils` | Scheduling, context packs, memories, metadata | | **Talk2Data** | 8080 | `/talk2data` | Chat sessions, NL-to-SQL, data analysis | | **Data Readiness** | 8000 | `/data-readiness` | Profiling workflows, data quality, glossary | The URLs above show direct service access. In production deployments behind an ingress controller, services are exposed on a single domain with path-based routing (e.g., `https://api.example.com/governance`, `https://api.example.com/assets`). The ingress rewrites paths to the service-internal prefix. ## API Versioning CRAFT APIs use service-specific prefixes (/governance, /assets, /utils). The platform follows these versioning principles: * **Path-based versioning**: Each service uses its own prefix (e.g., `/governance/...`, `/assets/...`, `/utils/...`) * **Backward compatibility**: Minor additions (new fields, new endpoints) are added within the same version * **Breaking changes**: Changes that remove fields, rename endpoints, or alter behavior require a new version (`/v2`) * **Deprecation**: Deprecated versions are maintained for at least 6 months with `Sunset` headers ## Common Headers ### Required Headers | Header | Description | Example | | --------------- | ------------------------------------------------------------------- | ------------------ | | `Authorization` | Bearer token for authentication (JWT contains `org_id` for scoping) | `Bearer eyJhbG...` | | `Content-Type` | Request body format | `application/json` | ### Response Headers | Header | Description | Example | | ----------------------- | ------------------------------------- | -------------- | | `X-Request-ID` | Unique request identifier for tracing | `req-a1b2c3d4` | | `X-RateLimit-Limit` | Maximum requests allowed in window | `300` | | `X-RateLimit-Remaining` | Requests remaining in current window | `297` | | `X-RateLimit-Reset` | Unix timestamp when window resets | `1712150400` | ## Rate Limits Rate limits are enforced per authenticated principal using a sliding window algorithm backed by Redis: | Operation Type | Limit | Applies To | | ----------------------------------------------- | ------- | --------------------------------------- | | **Read operations** (GET, POST :search) | 300/min | Authenticated users | | **Write operations** (POST, PUT, PATCH, DELETE) | 60/min | Authenticated users | | **Bulk operations** | 10/min | Authenticated users | | **Anonymous reads** | 30/min | Unauthenticated requests (when enabled) | When a rate limit is exceeded, the API returns: ```json theme={null} { "error": { "code": "RATE_LIMIT_EXCEEDED", "message": "Rate limit exceeded. Retry after 42 seconds.", "status": 429 } } ``` The `Retry-After` response header indicates when the client can retry. Rate limits are configurable per tenant by administrators. Contact your platform admin if the default limits are insufficient for your workload. ## Service Endpoints Summary ### Governance API | Method | Endpoint | Description | | ------ | ------------------------------------ | -------------------------------- | | GET | `/governance/organizations` | List organizations | | GET | `/governance/organizations/{org_id}` | Get organization details | | POST | `/governance/projects` | Create a project | | GET | `/governance/projects` | List projects | | GET | `/governance/projects/{project_id}` | Get project details | | GET | `/governance/permissions/check` | Check if a user has a permission | | POST | `/governance/permissions/grant` | Grant a permission | | POST | `/governance/permissions/revoke` | Revoke a permission | | \* | `/governance/resources/.../secrets` | Secret management (CRUD) | ### Assets API | Method | Endpoint | Description | | ------ | ------------------------------------------- | --------------------------- | | POST | `/assets/artifacts` | Create an artifact | | GET | `/assets/artifacts/{resource_uri}` | Get artifact details | | GET | `/assets/artifacts/{resource_uri}/preview` | Preview artifact content | | GET | `/assets/artifacts/{resource_uri}/download` | Download artifact | | POST | `/assets/data` | Register a data connection | | GET | `/assets/data/{resource_uri}` | Get data connection details | | POST | `/assets/data/{resource_uri}/verify` | Verify a data connection | | POST | `/assets/files` | Upload a file | | GET | `/assets/files/{resource_uri}` | Get file metadata | | GET | `/assets/files/{resource_uri}/download` | Download file | ### Utils API | Method | Endpoint | Description | | ------ | ------------------------------------------------ | -------------------------- | | POST | `/utils/schedules` | Create a schedule | | GET | `/utils/schedules/{schedule_id}` | Get schedule details | | PUT | `/utils/schedules/{schedule_id}` | Update a schedule | | DELETE | `/utils/schedules/{schedule_id}` | Delete a schedule | | POST | `/utils/context-packs` | Create a context pack | | GET | `/utils/context-packs` | List context packs | | POST | `/utils/memories` | Create a memory | | GET | `/utils/memories` | List memories | | POST | `/utils/search/metadata` | Search metadata catalog | | \* | `/utils/{resource_type}/{resource_uri}/metadata` | Metadata management (CRUD) | ### Talk2Data API (Data Insights) | Method | Endpoint | Description | | ------ | ----------------------------------------------------------------------- | ----------------------- | | POST | `/talk2data/chat/sessions` | Create a chat session | | GET | `/talk2data/chat/sessions` | List chat sessions | | GET | `/talk2data/chat/sessions/{session_id}` | Get session details | | PATCH | `/talk2data/chat/sessions/{session_id}` | Rename a session | | DELETE | `/talk2data/chat/sessions/{session_id}` | Delete a session | | POST | `/talk2data/chat/sessions/{session_id}/turns` | Create a new turn | | GET | `/talk2data/chat/sessions/{session_id}/turns` | List turns in a session | | GET | `/talk2data/chat/sessions/{session_id}/turns/{turn_id}` | Get turn details | | GET | `/talk2data/chat/sessions/{session_id}/turns/{turn_id}/artifacts` | Get turn artifacts | | POST | `/talk2data/chat/sessions/{session_id}/turns/{turn_id}/feedback` | Submit feedback | | POST | `/talk2data/chat/messages/sessions/{session_id}/turns/{turn_id}/stream` | Stream message (SSE) | | POST | `/talk2data/chat/messages/{task_id}/cancel` | Cancel a running task | ### Data Readiness API (Data Governance) | Method | Endpoint | Description | | ------ | --------------------------------------------------- | ------------------------------- | | GET | `/data-readiness/workflows` | List workflow configurations | | POST | `/data-readiness/workflows` | Create a workflow configuration | | GET | `/data-readiness/workflows/{configuration_id}` | Get workflow details | | POST | `/data-readiness/workflows/{configuration_id}/runs` | Trigger a workflow run | | GET | `/data-readiness/workflows/{configuration_id}/runs` | List runs for a workflow | | \* | `/data-readiness/data-assets/...` | Data asset catalog management | | \* | `/data-readiness/metrics/...` | Quality metrics and scorecards | | \* | `/data-readiness/violations` | Data quality violation tracking | | \* | `/data-readiness/glossary/terms` | Business glossary management | ## OpenAPI Specifications Each service generates an OpenAPI 3.0 specification accessible at: * Governance: `https:///api/governance/docs` (Swagger UI) or `https:///api/governance/openapi.json` * Assets: `https:///api/assets/docs` or `https:///api/assets/openapi.json` * Utils: `https:///api/utils/docs` or `https:///api/utils/openapi.json` The OpenAPI specs are used to auto-generate Python and TypeScript SDKs. ## SDKs Auto-generated SDKs are available for programmatic access: ```bash theme={null} pip install em-runtime-governance-sdk-python pip install em-runtime-assets-sdk-python pip install em-runtime-utils-sdk-python ``` ```python theme={null} from em_runtime_governance_sdk import AuthenticatedClient from em_runtime_governance_sdk.api.organizations import list_organizations client = AuthenticatedClient( base_url="https:///api/governance", token=access_token, ) orgs = await list_organizations.asyncio(client=client) ``` ```bash theme={null} npm install @emergence-ai/em-runtime-governance-sdk npm install @emergence-ai/em-runtime-assets-sdk npm install @emergence-ai/em-runtime-utils-sdk ``` ```typescript theme={null} import { Client } from '@emergence-ai/em-runtime-governance-sdk'; import { listOrganizations } from '@emergence-ai/em-runtime-governance-sdk/api'; const client = new Client({ BASE: 'https:///api/governance', TOKEN: accessToken, }); const orgs = await listOrganizations({ client }); ``` ## Pagination List endpoints support offset-based pagination using `page` and `limit` parameters: ```bash theme={null} # First page GET /assets/artifacts?limit=20&page=1 # Next page GET /assets/artifacts?limit=20&page=2 ``` The `limit` parameter controls page size (default varies by endpoint). The `page` parameter is 1-indexed. ## Next Steps Learn how to obtain and use JWT tokens for API access. Understand error response formats and troubleshooting. Use the auto-generated Python SDK for programmatic access. Use the auto-generated TypeScript SDK for frontend integration. # Execute Query Endpoint Source: https://docs.emergence.ai/api-reference/mcp/execute-query-endpoint /openapi/em-talk2data-v2.yaml post /talk2data/mcp/execute-query Execute a read-only SELECT SQL statement against a configured data connection and return the result rows. Typically called after `generate_sql`; pass the returned `rows` field directly into `generate_plotly_chart` to visualise the result. # Generate Plotly Chart Endpoint Source: https://docs.emergence.ai/api-reference/mcp/generate-plotly-chart-endpoint /openapi/em-talk2data-v2.yaml post /talk2data/mcp/generate-plotly-chart Visualise tabular results from `execute_query` as an interactive Plotly chart. Pass the `rows` from `execute_query` directly as `data`. Supports 25+ chart types (bar, line, heatmap, treemap, sankey, waterfall, parcoords, choropleth, etc.). Pick the chart type based on data shape: bar/line for categorical-vs-numeric, heatmap for matrix, treemap/sunburst for hierarchy, choropleth for geographic, sankey/funnel for flow. # Generate Sql Source: https://docs.emergence.ai/api-reference/mcp/generate-sql /openapi/em-talk2data-v2.yaml post /talk2data/mcp/generate-sql Generate SQL from a natural-language question. Returns the SQL string plus the schemas it references. Pipe the SQL into `execute_query` to fetch rows; then feed those rows into `generate_plotly_chart` to render a figure. Use this as the first step in any 'show me X' or 'what is Y' data-analysis flow. # Resolve Term Source: https://docs.emergence.ai/api-reference/mcp/resolve-term /openapi/em-talk2data-v2.yaml post /talk2data/mcp/resolve-term Resolve a business term against context pack memories. Fetches all memories from the MemoryProvider and filters client-side by term match (name or content contains the search term). Extracts structured fields (definition, related_columns, sql_pattern) from the best-matching concept/vocabulary/sample_question memories. Example: "What does 'conversion rate' mean?" will return a definition from concept/vocabulary memories, related column names from table description memories, and an example SQL pattern from sample_question memories. # Archive a memory Source: https://docs.emergence.ai/api-reference/memories/archive-a-memory /openapi/em-runtime-utils-v2.yaml post /utils/memories/{memory_id}/archive Archive a memory. # Archive Memory Source: https://docs.emergence.ai/api-reference/memories/archive-memory /openapi/em-memory-service-v2.yaml post /memories/{memory_id}/archive # Get a memory by ID Source: https://docs.emergence.ai/api-reference/memories/get-a-memory-by-id /openapi/em-runtime-utils-v2.yaml get /utils/memories/{memory_id} Get a memory by ID. # Get Memory By Id Source: https://docs.emergence.ai/api-reference/memories/get-memory-by-id /openapi/em-memory-service-v2.yaml get /memories/{memory_id} # List Memories Source: https://docs.emergence.ai/api-reference/memories/list-memories /openapi/em-memory-service-v2.yaml get /memories # Mark a memory as superseded Source: https://docs.emergence.ai/api-reference/memories/mark-a-memory-as-superseded /openapi/em-runtime-utils-v2.yaml post /utils/memories/{memory_id}/supersede/{superseded_by} Mark a memory as superseded by another memory. # Permanent Delete Memory Source: https://docs.emergence.ai/api-reference/memories/permanent-delete-memory /openapi/em-memory-service-v2.yaml post /memories/{memory_id}/permanent-delete # Permanently delete a memory Source: https://docs.emergence.ai/api-reference/memories/permanently-delete-a-memory /openapi/em-runtime-utils-v2.yaml post /utils/memories/{memory_id}/permanent-delete Permanently delete a memory. # Pin a memory Source: https://docs.emergence.ai/api-reference/memories/pin-a-memory /openapi/em-runtime-utils-v2.yaml post /utils/memories/{memory_id}/pin Pin a memory to mark it as important. # Pin Memory Source: https://docs.emergence.ai/api-reference/memories/pin-memory /openapi/em-memory-service-v2.yaml post /memories/{memory_id}/pin # Soft delete a memory Source: https://docs.emergence.ai/api-reference/memories/soft-delete-a-memory /openapi/em-runtime-utils-v2.yaml delete /utils/memories/{memory_id} Soft delete a memory (can be recovered). # Soft Delete Memory Source: https://docs.emergence.ai/api-reference/memories/soft-delete-memory /openapi/em-memory-service-v2.yaml delete /memories/{memory_id} # Supersede Memory Source: https://docs.emergence.ai/api-reference/memories/supersede-memory /openapi/em-memory-service-v2.yaml post /memories/{memory_id}/supersede/{superseded_by} # Unpin a memory Source: https://docs.emergence.ai/api-reference/memories/unpin-a-memory /openapi/em-runtime-utils-v2.yaml post /utils/memories/{memory_id}/unpin Unpin a memory. # Unpin Memory Source: https://docs.emergence.ai/api-reference/memories/unpin-memory /openapi/em-memory-service-v2.yaml post /memories/{memory_id}/unpin # Update a memory Source: https://docs.emergence.ai/api-reference/memories/update-a-memory /openapi/em-runtime-utils-v2.yaml patch /utils/memories/{memory_id} Update a memory's content, classification, or metadata. # Update Memory Source: https://docs.emergence.ai/api-reference/memories/update-memory /openapi/em-memory-service-v2.yaml patch /memories/{memory_id} # Validate a memory Source: https://docs.emergence.ai/api-reference/memories/validate-a-memory /openapi/em-runtime-utils-v2.yaml post /utils/memories/{memory_id}/validate Validate a memory with a confidence score. # Validate Memory Source: https://docs.emergence.ai/api-reference/memories/validate-memory /openapi/em-memory-service-v2.yaml post /memories/{memory_id}/validate # Bulk get metadata by FQN Source: https://docs.emergence.ai/api-reference/metadata/bulk-get-metadata-by-fqn /openapi/em-runtime-utils-v2.yaml post /utils/metadata/get Retrieve multiple metadata entities by FQN. Always returns the latest version. # Bulk get tags by FQN Source: https://docs.emergence.ai/api-reference/metadata/bulk-get-tags-by-fqn /openapi/em-runtime-utils-v2.yaml post /utils/metadata/tags/get Retrieve multiple tags by FQN. Always returns the latest non-deleted version. # Bulk soft-delete metadata Source: https://docs.emergence.ai/api-reference/metadata/bulk-soft-delete-metadata /openapi/em-runtime-utils-v2.yaml post /utils/metadata/delete Soft-delete multiple metadata entities with cascade. # Bulk soft-delete tags Source: https://docs.emergence.ai/api-reference/metadata/bulk-soft-delete-tags /openapi/em-runtime-utils-v2.yaml post /utils/metadata/tags/delete Soft-delete multiple tags by FQN. Not-found items are non-fatal. # Bulk store (upsert) metadata Source: https://docs.emergence.ai/api-reference/metadata/bulk-store-upsert-metadata /openapi/em-runtime-utils-v2.yaml post /utils/metadata/store Create or update multiple metadata entities. Items are sorted by FQN depth so ancestors are processed first. # Bulk store (upsert) tags Source: https://docs.emergence.ai/api-reference/metadata/bulk-store-upsert-tags /openapi/em-runtime-utils-v2.yaml post /utils/metadata/tags/store Create or update multiple tags. Parent categories must exist. # Clear enrichment fields on child entities Source: https://docs.emergence.ai/api-reference/metadata/clear-enrichment-fields-on-child-entities /openapi/em-runtime-utils-v2.yaml post /utils/{resource_type}/{resource_uri}/metadata/clear-enrichments Clear enrichment fields on all direct children of the given FQN. For a schema FQN: clears description and tags on all tables. For a table FQN: clears description, extension, business_definition, business_rules, enrichment flags, and tags on all columns. # Create or update a catalog tag Source: https://docs.emergence.ai/api-reference/metadata/create-or-update-a-catalog-tag /openapi/em-runtime-utils-v2.yaml post /utils/metadata/tags Upsert a catalog tag. Parent category must exist and not be deleted. Returns 201 on create, 200 on update. # Create or update a catalog tag category Source: https://docs.emergence.ai/api-reference/metadata/create-or-update-a-catalog-tag-category /openapi/em-runtime-utils-v2.yaml post /utils/metadata/catalog-tag-categories Upsert a catalog tag category. Returns 201 on create, 200 on update. # Delete a catalog tag Source: https://docs.emergence.ai/api-reference/metadata/delete-a-catalog-tag /openapi/em-runtime-utils-v2.yaml delete /utils/metadata/tags Soft-delete a catalog tag. # Delete a catalog tag category Source: https://docs.emergence.ai/api-reference/metadata/delete-a-catalog-tag-category /openapi/em-runtime-utils-v2.yaml delete /utils/metadata/catalog-tag-categories Soft-delete a catalog tag category and cascade to child tags. # Get metadata by FQN Source: https://docs.emergence.ai/api-reference/metadata/get-metadata-by-fqn /openapi/em-runtime-utils-v2.yaml get /utils/{resource_type}/{resource_uri}/metadata Retrieve a metadata asset by fully qualified name. Optionally filter by asset_type or retrieve a specific historical version. # Get or list catalog tag categories Source: https://docs.emergence.ai/api-reference/metadata/get-or-list-catalog-tag-categories /openapi/em-runtime-utils-v2.yaml get /utils/metadata/catalog-tag-categories If 'fqn' is provided, returns a single catalog tag category. Otherwise, returns a paginated list. # Get or list catalog tags Source: https://docs.emergence.ai/api-reference/metadata/get-or-list-catalog-tags /openapi/em-runtime-utils-v2.yaml get /utils/metadata/tags If 'fqn' is provided, returns a single tag. Otherwise, returns a paginated list. # List metadata entities Source: https://docs.emergence.ai/api-reference/metadata/list-metadata-entities /openapi/em-runtime-utils-v2.yaml get /utils/{resource_type}/{resource_uri}/metadata/query Returns a paginated list of metadata entities under the resource, with filtering and sorting. # List version history for an entity Source: https://docs.emergence.ai/api-reference/metadata/list-version-history-for-an-entity /openapi/em-runtime-utils-v2.yaml get /utils/{resource_type}/{resource_uri}/metadata/versions Returns enriched version history with structured diffs, per-field attribution, and contributor list. # Soft-delete metadata by FQN Source: https://docs.emergence.ai/api-reference/metadata/soft-delete-metadata-by-fqn /openapi/em-runtime-utils-v2.yaml delete /utils/{resource_type}/{resource_uri}/metadata Soft-delete a metadata entity (all versions + cascade) or a single version row. # Store (upsert) metadata Source: https://docs.emergence.ai/api-reference/metadata/store-upsert-metadata /openapi/em-runtime-utils-v2.yaml post /utils/{resource_type}/{resource_uri}/metadata Create or update a metadata asset. The asset type is determined by the type discriminator field in the request body. # Get Aggregated Scorecard Source: https://docs.emergence.ai/api-reference/metrics/get-aggregated-scorecard /openapi/em-data-readiness-v2.yaml post /api/data-readiness/metrics/scorecard Returns aggregated metric scores at the requested scope level. # Get Filter Options Source: https://docs.emergence.ai/api-reference/metrics/get-filter-options /openapi/em-data-readiness-v2.yaml get /api/data-readiness/metrics/filter-options Returns distinct filter values for cascading dashboard dropdowns, scoped by assessment type. # Get Per-Asset Metric Details Source: https://docs.emergence.ai/api-reference/metrics/get-per-asset-metric-details /openapi/em-data-readiness-v2.yaml get /api/data-readiness/metrics/details Returns per-asset metric detail for drill-down analysis. # Get Score Trend Source: https://docs.emergence.ai/api-reference/metrics/get-score-trend /openapi/em-data-readiness-v2.yaml get /api/data-readiness/metrics/trend Returns score trend over a time range, bucketed by the requested granularity. # List Dimensions Source: https://docs.emergence.ai/api-reference/metrics/list-dimensions /openapi/em-data-readiness-v2.yaml get /api/data-readiness/metrics/dimensions Returns dimension definitions with their metric definitions. # Create Model Source: https://docs.emergence.ai/api-reference/models/create-model /openapi/em-runtime-assets-v2.yaml post /assets/models Create model # Delete Model Source: https://docs.emergence.ai/api-reference/models/delete-model /openapi/em-runtime-assets-v2.yaml delete /assets/models/{resource_uri} Delete model # Get Model Source: https://docs.emergence.ai/api-reference/models/get-model /openapi/em-runtime-assets-v2.yaml get /assets/models/{resource_uri} Get model # Get Model Secret Source: https://docs.emergence.ai/api-reference/models/get-model-secret /openapi/em-runtime-assets-v2.yaml get /assets/models/{resource_uri}/secret Get model secrets # List Models Source: https://docs.emergence.ai/api-reference/models/list-models /openapi/em-runtime-assets-v2.yaml get /assets/models List models in project # Update Model Source: https://docs.emergence.ai/api-reference/models/update-model /openapi/em-runtime-assets-v2.yaml put /assets/models/{resource_uri} Update model # Verify Model Source: https://docs.emergence.ai/api-reference/models/verify-model /openapi/em-runtime-assets-v2.yaml post /assets/models/{resource_uri}/verify Verify model credentials and accessibility # Verify Model Before Save Source: https://docs.emergence.ai/api-reference/models/verify-model-before-save /openapi/em-runtime-assets-v2.yaml post /assets/models/verify Verify model credentials before creating the model # Create a new organization Source: https://docs.emergence.ai/api-reference/organizations/create-a-new-organization /openapi/em-runtime-governance-v2.yaml post /governance/organizations Creates a new tenant organization in the hub Keycloak realm (server-generated UUID), default organization groups, a database record, and OpenFGA permissions. Display `name` is provided by the caller; Keycloak assigns `id` and `alias`. Automatically creates default organization groups and sets up permissions in OpenFGA. # Get organization details Source: https://docs.emergence.ai/api-reference/organizations/get-organization-details /openapi/em-runtime-governance-v2.yaml get /governance/organizations/{org_id} Retrieves detailed information about a specific organization by its Keycloak UUID. The user must have permission to view the organization. Returns database metadata plus the Keycloak organization alias (for OIDC org switching). # List organizations for the current user Source: https://docs.emergence.ai/api-reference/organizations/list-organizations-for-the-current-user /openapi/em-runtime-governance-v2.yaml get /governance/organizations Returns Keycloak organizations the authenticated user belongs to in the hub realm. Each entry includes the organization UUID (`id`), display `name`, and Keycloak `alias` (for OIDC `scope=organization:` in the runtime-ui org switcher). # Check user permission Source: https://docs.emergence.ai/api-reference/permissions/check-user-permission /openapi/em-runtime-governance-v2.yaml get /governance/permissions/check Verifies if the authenticated user has permission to perform a specific action on a resource. Returns 200 if allowed, 403 if denied. # Delete all permissions for a resource Source: https://docs.emergence.ai/api-reference/permissions/delete-all-permissions-for-a-resource /openapi/em-runtime-governance-v2.yaml post /governance/permissions/delete-all Removes all permission tuples associated with a specific resource. Typically used during resource cleanup or deletion. Returns the count of deleted permissions. # Grant permission to user or group Source: https://docs.emergence.ai/api-reference/permissions/grant-permission-to-user-or-group /openapi/em-runtime-governance-v2.yaml post /governance/permissions/grant Grants a specific permission relation to a user or group on a resource. Used to assign roles like owner, admin, developer, operator, or viewer. # List all objects the user can read Source: https://docs.emergence.ai/api-reference/permissions/list-all-objects-the-user-can-read /openapi/em-runtime-governance-v2.yaml get /governance/permissions/accessible-objects Returns all object IDs across all resource types that the authenticated user has can_read on. Fans out to OpenFGA internally — callers make one HTTP request instead of one per resource type. # Revoke permission from user or group Source: https://docs.emergence.ai/api-reference/permissions/revoke-permission-from-user-or-group /openapi/em-runtime-governance-v2.yaml post /governance/permissions/revoke Removes a specific permission relation from a user or group on a resource. Used to revoke roles like owner, admin, developer, operator, or viewer. # Set resource parent relationship Source: https://docs.emergence.ai/api-reference/permissions/set-resource-parent-relationship /openapi/em-runtime-governance-v2.yaml post /governance/permissions/set-parent Establishes a parent-child relationship between resources for permission inheritance. For example, linking a project to an organization or a secret to a project. # Create a new project Source: https://docs.emergence.ai/api-reference/projects/create-a-new-project /openapi/em-runtime-governance-v2.yaml post /governance/projects Creates a new project within the user's organization (determined from JWT claims). The project is automatically linked to the organization and default project groups receive permissions. A UUID will be generated for the project ID unless external_id is provided. # Delete a project Source: https://docs.emergence.ai/api-reference/projects/delete-a-project /openapi/em-runtime-governance-v2.yaml delete /governance/projects/{project_id} Deletes a project and its project-scoped authorization tuples. # Get project details Source: https://docs.emergence.ai/api-reference/projects/get-project-details /openapi/em-runtime-governance-v2.yaml get /governance/projects/{project_id} Retrieves detailed information about a specific project by its ID. The user must have permission to view the project. # List projects Source: https://docs.emergence.ai/api-reference/projects/list-projects /openapi/em-runtime-governance-v2.yaml get /governance/projects List all projects in the user's organization with pagination support. # Root Source: https://docs.emergence.ai/api-reference/root/root /openapi/em-data-readiness-v2.yaml get /api/data-readiness/ # Sample Data Endpoint Source: https://docs.emergence.ai/api-reference/sample/sample-data-endpoint /openapi/em-talk2data-v2.yaml post /talk2data/v1/sample # Create a new schedule Source: https://docs.emergence.ai/api-reference/schedules/create-a-new-schedule /openapi/em-runtime-utils-v2.yaml post /utils/schedules Creates a new schedule for triggering workflows. The schedule can be either a cron-based recurring schedule or a one-time schedule. When triggered, notifications are published to the specified topic. # Delete a schedule Source: https://docs.emergence.ai/api-reference/schedules/delete-a-schedule /openapi/em-runtime-utils-v2.yaml delete /utils/schedules/{schedule_id} Permanently deletes a schedule. The schedule must belong to the authenticated user's organization. This action cannot be undone. # Delete a single schedule run Source: https://docs.emergence.ai/api-reference/schedules/delete-a-single-schedule-run /openapi/em-runtime-utils-v2.yaml delete /utils/schedules/runs/{run_id} Deletes a single schedule run by its ID. The run must belong to the authenticated user's organization. # Disable a schedule Source: https://docs.emergence.ai/api-reference/schedules/disable-a-schedule /openapi/em-runtime-utils-v2.yaml patch /utils/schedules/{schedule_id}/disable Disables a schedule and unschedules it from pg_cron. The schedule must belong to the authenticated user's organization. If the schedule is already disabled, this operation is idempotent. # Enable a schedule Source: https://docs.emergence.ai/api-reference/schedules/enable-a-schedule /openapi/em-runtime-utils-v2.yaml patch /utils/schedules/{schedule_id}/enable Enables a schedule and registers it with pg_cron. The schedule must belong to the authenticated user's organization. If the schedule is already enabled, this operation is idempotent. # Get a schedule Source: https://docs.emergence.ai/api-reference/schedules/get-a-schedule /openapi/em-runtime-utils-v2.yaml get /utils/schedules/{schedule_id} Retrieves a single schedule by its ID. The schedule must belong to the authenticated user's organization. # Get a single schedule run Source: https://docs.emergence.ai/api-reference/schedules/get-a-single-schedule-run /openapi/em-runtime-utils-v2.yaml get /utils/schedules/runs/{run_id} Retrieves a single schedule run by its ID, including the full schedule snapshot at execution time. The run must belong to the authenticated user's organization. # List schedule runs Source: https://docs.emergence.ai/api-reference/schedules/list-schedule-runs /openapi/em-runtime-utils-v2.yaml get /utils/schedules/runs Retrieves a paginated list of schedule runs for the authenticated user's organization. Results can be optionally filtered by project ID and ordered by creation time. # List schedules Source: https://docs.emergence.ai/api-reference/schedules/list-schedules /openapi/em-runtime-utils-v2.yaml get /utils/schedules Retrieves a list of all schedules associated with the authenticated user's organization. If a project ID is provided in the header, the results are filtered to only include schedules for that project. # Update a schedule Source: https://docs.emergence.ai/api-reference/schedules/update-a-schedule /openapi/em-runtime-utils-v2.yaml put /utils/schedules/{schedule_id} Updates an existing schedule. All fields in the request body are required - this is a full update. The schedule must belong to the authenticated user's organization. # Bulk index entities Source: https://docs.emergence.ai/api-reference/search/bulk-index-entities /openapi/em-runtime-search-v2.yaml post /search/bulk/index Add or update one or more entities in the global search index. Each item is validated then published to the internal event stream — embedding generation and indexing happen asynchronously. Returns 202 with the count of accepted operations. **Required fields**: ``organization_id``, ``project_id``, ``entity_type``, ``entity_id``, ``title``, ``permission_id``. **``embedded_text``**: The text used for BM25 indexing and embedding generation. For catalog entities this should be an LLM-generated contextual summary (~1000 tokens). For other entities use title + description concatenation. The embedding vector is generated asynchronously by the search handler — publishing services do not need to compute it. **``search_attributes``**: Flat ``dict[str, str]`` of fields that support structured query syntax on ``POST /search/global``. All values must be strings. Standard keys for catalog entities: ``name`` — entity name ``service_name`` — data source name (supports ``service.name:X``) ``database_name`` — database name (supports ``database.name:X``) ``schema_name`` — schema name (supports ``databaseSchema.name:X``) ``tags`` — space-separated tag FQNs (supports ``tags.tagFQN:PII*``) ``is_pii_tagged`` — 'true'/'false' (supports ``extension.is_pii_tagged:1``) ``is_sensitivity_tagged`` — 'true'/'false' ``has_description`` — 'true'/'false' (supports ``NOT _exists_:description``) ``has_owners`` — 'true'/'false' (supports ``NOT _exists_:owners``) ``has_undescribed_columns`` — 'true'/'false' (supports ``NOT _exists_:columns.description``) ``columns_tags_tagFQN`` — space-separated column tag FQNs (supports ``_exists_:columns.tags.tagFQN:PII*``) # Bulk remove entities from the index Source: https://docs.emergence.ai/api-reference/search/bulk-remove-entities-from-the-index /openapi/em-runtime-search-v2.yaml post /search/bulk/delete Remove one or more entities from the global search index. Each item is validated then published to the internal event stream — deletion happens asynchronously. Returns 202 with the count of accepted operations. # Global search Source: https://docs.emergence.ai/api-reference/search/global-search /openapi/em-runtime-search-v2.yaml post /search/global Unified search across all entity types — projects, assets, data connections, catalog entities (databases, schemas, tables, columns), files, artifacts, agents, and models. Results are filtered to entities the authenticated user can access. **Query modes** (set via the ``mode`` field): - ``semantic`` *(default)*: Plain natural language. ``q`` is run through BM25 and, when an embedding service is configured, also vectorised for hybrid RRF ranking. Best for conversational queries. - ``structured``: Elasticsearch-style query string in ``query``. Parsed by the Lark grammar and translated to SQL predicates against the ``search_attributes`` JSONB column and the ParadeDB BM25 index. Supports field:value, wildcards, boolean AND/OR/NOT, _exists_ checks, tag filtering, and extension field comparisons. Replaces the retired ``GET /search/metadata`` endpoint. - ``hybrid``: Both legs simultaneously. ``q`` drives semantic ranking; ``query`` applies structural constraints. Results are merged via Reciprocal Rank Fusion (RRF). **Pagination**: pass ``session_id`` from the first response on subsequent pages to reuse the scope session and avoid re-running the OpenFGA ListObjects fan-out. **Structured query syntax examples**: ``service.name:mysql AND tags.tagFQN:PII*`` — PII-tagged tables in a MySQL service ``NOT _exists_:owners AND entity_type:table`` — tables missing owners ``NOT _exists_:columns.description`` — tables with undescribed columns ``extension.is_pii_tagged:1 AND databaseSchema.name:public`` — PII tables in public schema ``(tags.tagFQN:SENSITIVITY_TAGS.Confidential OR tags.tagFQN:SENSITIVITY_TAGS.Restricted) AND service.name:snowflake`` — confidential or restricted Snowflake entities # Search metadata across resources Source: https://docs.emergence.ai/api-reference/search/search-metadata-across-resources /openapi/em-runtime-utils-v2.yaml get /utils/search/metadata Search catalog entities across all resources within a tenant, with filtering, sorting, and pagination. # Create a new organization secret Source: https://docs.emergence.ai/api-reference/secrets/create-a-new-organization-secret /openapi/em-runtime-governance-v2.yaml post /governance/resources/organizations/secrets Creates a new secret at the organization level. The secret is stored securely in Infisical and the creator is automatically granted owner permissions. The secret value should be provided as a JSON string matching the format required by the resource type. # Create a new project secret Source: https://docs.emergence.ai/api-reference/secrets/create-a-new-project-secret /openapi/em-runtime-governance-v2.yaml post /governance/resources/projects/secrets Creates a new secret at the project level. The secret is stored securely in Infisical and the creator is automatically granted owner permissions. The secret value should be provided as a JSON string matching the format required by the resource type. # Create a new secret Source: https://docs.emergence.ai/api-reference/secrets/create-a-new-secret /openapi/em-runtime-governance-v2.yaml post /governance/resources/{resource_type}/{resource_id}/secrets Creates a new secret within a specific resource. The secret is stored securely in Infisical and the creator is automatically granted owner permissions. The secret value should be provided as a JSON string matching the format required by the resource type. # Delete a project secret Source: https://docs.emergence.ai/api-reference/secrets/delete-a-project-secret /openapi/em-runtime-governance-v2.yaml delete /governance/resources/projects/secrets/{secret_name} Permanently deletes a project secret from Infisical and removes all associated permissions. Requires delete permissions on the secret. This action cannot be undone. # Delete a secret Source: https://docs.emergence.ai/api-reference/secrets/delete-a-secret /openapi/em-runtime-governance-v2.yaml delete /governance/resources/{resource_type}/{resource_id}/secrets/{secret_name} Permanently deletes a secret from Infisical and removes all associated permissions. Requires delete permissions on the secret. This action cannot be undone. # Delete an organization secret Source: https://docs.emergence.ai/api-reference/secrets/delete-an-organization-secret /openapi/em-runtime-governance-v2.yaml delete /governance/resources/organizations/secrets/{secret_name} Permanently deletes an organization secret from Infisical and removes all associated permissions. Requires delete permissions on the secret. This action cannot be undone. # Get organization secret value Source: https://docs.emergence.ai/api-reference/secrets/get-organization-secret-value /openapi/em-runtime-governance-v2.yaml get /governance/resources/organizations/secrets/{secret_name} Retrieves the actual secret value for a specific organization secret by name. This endpoint returns sensitive data and requires read permissions on the secret. The secret value is returned as stored in Infisical. # Get project secret value Source: https://docs.emergence.ai/api-reference/secrets/get-project-secret-value /openapi/em-runtime-governance-v2.yaml get /governance/resources/projects/secrets/{secret_name} Retrieves the actual secret value for a specific project secret by name. This endpoint returns sensitive data and requires read permissions on the secret. The secret value is returned as stored in Infisical. # Get secret value Source: https://docs.emergence.ai/api-reference/secrets/get-secret-value /openapi/em-runtime-governance-v2.yaml get /governance/resources/{resource_type}/{resource_id}/secrets/{secret_name} Retrieves the actual secret value for a specific secret by name. This endpoint returns sensitive data and requires read permissions on the secret. The secret value is returned as stored in Infisical. # List secrets for a project Source: https://docs.emergence.ai/api-reference/secrets/list-secrets-for-a-project /openapi/em-runtime-governance-v2.yaml get /governance/resources/projects/secrets Retrieves a list of all secrets associated with a specific project. Secrets are stored in Infisical and scoped by organization and project. Returns only secret metadata, not the actual secret values. # List secrets for a resource Source: https://docs.emergence.ai/api-reference/secrets/list-secrets-for-a-resource /openapi/em-runtime-governance-v2.yaml get /governance/resources/{resource_type}/{resource_id}/secrets Retrieves a list of all secrets associated with a specific resource. Secrets are stored in Infisical and scoped by organization, project, and resource. Returns only secret metadata, not the actual secret values. # List secrets for an organization Source: https://docs.emergence.ai/api-reference/secrets/list-secrets-for-an-organization /openapi/em-runtime-governance-v2.yaml get /governance/resources/organizations/secrets Retrieves a list of all secrets associated with the authenticated user's organization. Secrets are stored in Infisical and scoped by organization. Returns only secret metadata, not the actual secret values. # Update an existing organization secret Source: https://docs.emergence.ai/api-reference/secrets/update-an-existing-organization-secret /openapi/em-runtime-governance-v2.yaml put /governance/resources/organizations/secrets/{secret_name} Updates the value or description of an existing organization secret. Requires write permissions on the secret. The secret value and/or description can be updated independently. # Update an existing project secret Source: https://docs.emergence.ai/api-reference/secrets/update-an-existing-project-secret /openapi/em-runtime-governance-v2.yaml put /governance/resources/projects/secrets/{secret_name} Updates the value or description of an existing project secret. Requires write permissions on the secret. The secret value and/or description can be updated independently. # Update an existing secret Source: https://docs.emergence.ai/api-reference/secrets/update-an-existing-secret /openapi/em-runtime-governance-v2.yaml put /governance/resources/{resource_type}/{resource_id}/secrets/{secret_name} Updates the value or description of an existing secret. Requires write permissions on the secret. The secret value and/or description can be updated independently. # Service Info Source: https://docs.emergence.ai/api-reference/service-info /openapi/em-talk2data-v2.yaml get /talk2data # Service Info Source: https://docs.emergence.ai/api-reference/service-info-1 /openapi/em-talk2data-v2.yaml get /talk2data/ # Create a soft FK edge Source: https://docs.emergence.ai/api-reference/soft-fk-edges/create-a-soft-fk-edge /openapi/em-runtime-utils-v2.yaml post /utils/context-packs/{name}/soft-fk-edges Create a new Soft FK edge within a context pack. # Expand seed tables via soft FK edges to find junction tables Source: https://docs.emergence.ai/api-reference/soft-fk-edges/expand-seed-tables-via-soft-fk-edges-to-find-junction-tables /openapi/em-runtime-utils-v2.yaml post /utils/context-packs/{name}/soft-fk-expand BFS expansion to find junction tables connecting seed tables. # Get a soft FK edge by ID Source: https://docs.emergence.ai/api-reference/soft-fk-edges/get-a-soft-fk-edge-by-id /openapi/em-runtime-utils-v2.yaml get /utils/context-packs/{name}/soft-fk-edges/{edge_id} Get an FK edge by ID. # List soft FK edges Source: https://docs.emergence.ai/api-reference/soft-fk-edges/list-soft-fk-edges /openapi/em-runtime-utils-v2.yaml get /utils/context-packs/{name}/soft-fk-edges List soft FK edges within a context pack. # Soft delete a soft FK edge Source: https://docs.emergence.ai/api-reference/soft-fk-edges/soft-delete-a-soft-fk-edge /openapi/em-runtime-utils-v2.yaml delete /utils/context-packs/{name}/soft-fk-edges/{edge_id} Soft delete a soft FK edge. # Update a soft FK edge Source: https://docs.emergence.ai/api-reference/soft-fk-edges/update-a-soft-fk-edge /openapi/em-runtime-utils-v2.yaml patch /utils/context-packs/{name}/soft-fk-edges/{edge_id} Update a soft FK edge's description, confidence, or status. # Get violation Source: https://docs.emergence.ai/api-reference/violations/get-violation /openapi/em-data-readiness-v2.yaml get /api/data-readiness/violations/{violation_id} Get a single violation snapshot by its ID. # List violation definitions Source: https://docs.emergence.ai/api-reference/violations/list-violation-definitions /openapi/em-data-readiness-v2.yaml get /api/data-readiness/violations/definitions List all available violation types and their configurations. # List violations Source: https://docs.emergence.ai/api-reference/violations/list-violations /openapi/em-data-readiness-v2.yaml get /api/data-readiness/violations List all violation snapshots with optional filtering and pagination. # Update violation Source: https://docs.emergence.ai/api-reference/violations/update-violation /openapi/em-data-readiness-v2.yaml patch /api/data-readiness/violations/{violation_id} Update a violation's status (open, resolved, false_positive, acknowledged). # Cancel All Active Runs Source: https://docs.emergence.ai/api-reference/workflows/cancel-all-active-runs /openapi/em-data-readiness-v2.yaml post /api/data-readiness/workflows/{configuration_id}/runs/cancel Cancel all active (queued or running) runs for a configuration. # Cancel Workflow Run Source: https://docs.emergence.ai/api-reference/workflows/cancel-workflow-run /openapi/em-data-readiness-v2.yaml post /api/data-readiness/workflows/{configuration_id}/runs/{run_id}/cancel Cancel a specific workflow run. # Create Pending Approval Source: https://docs.emergence.ai/api-reference/workflows/create-pending-approval /openapi/em-data-readiness-v2.yaml post /api/data-readiness/workflows/runs/{run_id}/approval Create a pending approval record for a completed workflow run. This is typically called automatically when a workflow completes, but can also be called manually to create an approval for a run that doesn't have one. **Returns:** - 201: Approval created successfully - 404: Workflow run not found - 409: Approval already exists for this run # Create Workflow Configuration Source: https://docs.emergence.ai/api-reference/workflows/create-workflow-configuration /openapi/em-data-readiness-v2.yaml post /api/data-readiness/workflows Create a new workflow configuration with custom settings, schedule, and triggers. This creates a configuration that can be used to run workflows with specific settings. Required: - `workflow_definition_id`: ID of the workflow definition to configure Optional: - `config`: Configuration overrides - `schedule`: Schedule configuration for automated execution - `triggers`: Event-based triggers for workflow execution # Create Workflow Definition Source: https://docs.emergence.ai/api-reference/workflows/create-workflow-definition /openapi/em-data-readiness-v2.yaml post /api/data-readiness/workflows/definitions Create a new workflow definition (template). Workflow definitions are the deployed pipeline templates that can be configured with custom settings, schedules, and triggers. This endpoint is typically used by administrators to define new workflow types. Required: - `name`: Human-readable workflow name - `workflow_type`: Type of workflow (generate-pii-tags, generate-dq-tags, etc.) - `data_asset_types`: List of supported data asset types Optional: - `description`: Workflow description - `default_config`: Default configuration for this workflow type - `is_active`: Whether the workflow is active (default: true) # Create Workflow Run Source: https://docs.emergence.ai/api-reference/workflows/create-workflow-run /openapi/em-data-readiness-v2.yaml post /api/data-readiness/workflows/{configuration_id}/runs Trigger a new run of a workflow by configuration ID. This creates a workflow run record and triggers the associated Prefect deployment in fire-and-forget mode. The run status can be monitored via the get/list endpoints. Headers: - `Authorization`: Bearer token with JWT for authentication - `X-Project-ID`: Project resource URI (e.g., 'proj:customer-analytics') Optional: - `asset_fqn`: Fully qualified name of the data asset (e.g., 'PostgreSQL.dvdrental.public.customer') - `parameters`: Additional workflow execution parameters. For generate-table-description, use 'include' to specify a list of table FQNs to process. # Get Approval Status Source: https://docs.emergence.ai/api-reference/workflows/get-approval-status /openapi/em-data-readiness-v2.yaml get /api/data-readiness/workflows/runs/{run_id}/approval Get the approval status for a workflow run. **Returns:** - 200: Approval found - 404: Approval not found for this run # Get Pipeline Run Source: https://docs.emergence.ai/api-reference/workflows/get-pipeline-run /openapi/em-data-readiness-v2.yaml get /api/data-readiness/workflows/pipelines/{pipeline_id} Retrieve the status and progress counters for a run-all-enrichment pipeline run. # Get Task Source: https://docs.emergence.ai/api-reference/workflows/get-task /openapi/em-data-readiness-v2.yaml get /api/data-readiness/workflows/{configuration_id}/runs/{run_id}/tasks/{task_id} Get detailed information about a specific task. # Get Workflow Configuration Source: https://docs.emergence.ai/api-reference/workflows/get-workflow-configuration /openapi/em-data-readiness-v2.yaml get /api/data-readiness/workflows/{configuration_id} Retrieve detailed information about a specific workflow configuration. Returns the full configuration details including: - Configuration overrides - Schedule configuration - Trigger configurations - Workflow definition metadata (name, type, supported data asset types) The config, schedule, and triggers represent configuration-specific settings that feed into deployed pipelines without redeploying code. # Get Workflow Definition Source: https://docs.emergence.ai/api-reference/workflows/get-workflow-definition /openapi/em-data-readiness-v2.yaml get /api/data-readiness/workflows/definitions/{definition_id} Retrieve detailed information about a specific workflow definition. Workflow definitions are the deployed pipeline templates that can be configured with custom settings. This endpoint returns the full definition including supported data asset types and default configuration. # Get Workflow Run Source: https://docs.emergence.ai/api-reference/workflows/get-workflow-run /openapi/em-data-readiness-v2.yaml get /api/data-readiness/workflows/{configuration_id}/runs/{run_id} Retrieve detailed information about a specific workflow run, including all associated tasks. # Get Workflow Run Statistics Source: https://docs.emergence.ai/api-reference/workflows/get-workflow-run-statistics /openapi/em-data-readiness-v2.yaml get /api/data-readiness/workflows/{configuration_id}/runs/{run_id}/statistics Get statistics for a specific workflow run including task counts and duration. # Get Workflow Statistics Source: https://docs.emergence.ai/api-reference/workflows/get-workflow-statistics /openapi/em-data-readiness-v2.yaml get /api/data-readiness/workflows/statistics Get aggregated statistics for workflow runs. Optional filtering: - `configuration_id`: Filter statistics for a specific configuration. If not provided, returns statistics across all workflows. - `start_date`: Filter runs started after this date (ISO 8601) - `end_date`: Filter runs started before this date (ISO 8601) # List All Workflow Runs Source: https://docs.emergence.ai/api-reference/workflows/list-all-workflow-runs /openapi/em-data-readiness-v2.yaml get /api/data-readiness/workflows/runs Retrieve a paginated list of workflow runs across all workflows for the current tenant. Supports filtering by: - `asset_fqn`: Filter runs by asset FQN (prefix match) - `workflow_type`: Filter by workflow type - `status`: Comma-separated status list (e.g., 'running,failed,completed'). Valid statuses: pending, scheduled, running, completed, failed, cancelled, retrying, cached, crashed - `start_date`: Filter runs started after this date (ISO 8601) - `end_date`: Filter runs started before this date (ISO 8601) Supports sorting by: - `sort`: Multi-sort format: field:order,field:order (e.g., 'status:desc,started_at:asc'). Valid fields: workflow_name, started_at, status. Valid orders: asc, desc # List Tasks Source: https://docs.emergence.ai/api-reference/workflows/list-tasks /openapi/em-data-readiness-v2.yaml get /api/data-readiness/workflows/{configuration_id}/runs/{run_id}/tasks List tasks within a workflow run. # List Workflow Configurations Source: https://docs.emergence.ai/api-reference/workflows/list-workflow-configurations /openapi/em-data-readiness-v2.yaml get /api/data-readiness/workflows Retrieve a paginated list of workflow configurations available to the current user. Supports filtering by: - `data_asset_type`: Filter configurations linked to a specific data asset type - `workflow_type`: Filter by type (generate-pii-tags, generate-dq-tags, etc.) - `status`: Filter by configuration status - `has_schedule`: Filter configurations with active schedules - `has_triggers`: Filter configurations with active triggers - `search`: Search by workflow name/description - `is_default`: Filter by default configuration status (pass `true` to list only the canonical default per workflow type) The response includes summary schedule information and triggers. Use the Get Configuration endpoint for full configuration details. # List Workflow Definitions Source: https://docs.emergence.ai/api-reference/workflows/list-workflow-definitions /openapi/em-data-readiness-v2.yaml get /api/data-readiness/workflows/definitions Retrieve a paginated list of workflow definitions (available workflow types). Workflow definitions are the deployed pipeline templates that can be configured with custom settings, schedules, and triggers. Supports filtering by: - `workflow_type`: Filter by workflow type - `is_active`: Filter by active status - `data_asset_type`: Filter workflows that support a specific data asset type This endpoint is useful for discovering available workflow types that can be configured for execution. # Stream Workflow Run Events (SSE) Source: https://docs.emergence.ai/api-reference/workflows/stream-workflow-run-events-sse /openapi/em-data-readiness-v2.yaml get /api/data-readiness/workflows/{configuration_id}/runs/{run_id}/events/stream Real-time event stream via Server-Sent Events. **Features:** - Streams all historical events first, then live events - Supports resumable connections via `Last-Event-ID` header - Clients reconnecting will catch up from the last seen sequence number **Usage:** 1. First connection: Receives all historical events, then live events 2. Reconnection: Send `Last-Event-ID: ` header to resume from that point # Update Workflow Approval Status Source: https://docs.emergence.ai/api-reference/workflows/update-workflow-approval-status /openapi/em-data-readiness-v2.yaml patch /api/data-readiness/workflows/runs/{run_id}/approval Update the approval status of a workflow run's output. **Request Body:** - `action`: One of "approve", "reject", or "pending" - `notes`: Optional notes about the decision **Approval Flow:** - When approved, the workflow output will be applied to the data asset - When rejected, the workflow output is discarded - When set to pending, the approval is reset for re-review (clears previous decision) **Returns:** - 200: Approval updated successfully - 400: Invalid action (e.g., trying to set to pending when already pending) - 404: Approval not found for this run # Data Enrichment Source: https://docs.emergence.ai/data-governance/data-enrichment How the Data Governance solution uses LLMs to enrich metadata with descriptions, classifications, and data quality rule suggestions. # Metadata Enrichment The metadata enrichment pipeline uses **LLMs** to automatically generate descriptions, classifications, and data quality rule suggestions for profiled data assets. This transforms raw schema information into rich, governed metadata that helps data teams understand and trust their data. ## What Gets Enriched | Enrichment | Description | | --------------------------- | ---------------------------------------------------------------------- | | **Business description** | Human-readable description of the table's purpose and contents | | **Domain classification** | Business domain the table belongs to (e.g., Sales, Finance, HR) | | **Sensitivity level** | Data classification level (Public, Internal, Confidential, Restricted) | | **Data steward suggestion** | Recommended owner based on domain and content | | **Relationship summary** | Description of how the table relates to other tables | | Enrichment | Description | | ------------------------ | --------------------------------------------------------------- | | **Business description** | What the column represents in business terms | | **Data type assessment** | Whether the declared type matches actual content | | **PII classification** | Whether the column contains personally identifiable information | | **Format pattern** | Expected format (email, phone, date, currency, etc.) | | **Valid value range** | Expected min/max or enumerated values | | **DQ rule suggestions** | Recommended data quality rules for this column | ## Enrichment Pipeline ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% graph TB API["Data Governance API\n(trigger)"] FLOW["Prefect Flow\n(enrich_metadata)"] CTR["ConcurrentTaskRunner\n(parallel tables)"] CTX["prepare_tables_and_schema()\n(full schema + profiling context)"] SEM["asyncio.Semaphore\n(LLM API rate gate)"] LLM["Vertex AI\n(gemini-3.5-flash)"] VAL["Validation\n(PII cross-check,\nschema alignment)"] DB[("datareadiness DB\n(enriched metadata)")] subgraph Outputs["Generated Metadata"] DESC["Business descriptions"] PII["PII classifications"] DQR["DQ rule suggestions"] end API --> FLOW --> CTR --> CTX --> SEM --> LLM --> VAL --> Outputs --> DB classDef trigger fill:#56B4E9,stroke:#555555,color:#000 classDef orch fill:#E69F00,stroke:#555555,color:#000 classDef gate fill:#F0E442,stroke:#555555,color:#000 classDef llm fill:#CC79A7,stroke:#555555,color:#000 classDef result fill:#009E73,stroke:#555555,color:#000 classDef store fill:#0072B2,stroke:#555555,color:#fff class API trigger class FLOW,CTR,CTX orch class SEM gate class LLM,VAL llm class DESC,PII,DQR result class DB store ``` The enrichment workflow is orchestrated by Prefect and uses the OpenAI API for LLM-powered analysis: The workflow gathers schema information and profiling results for the target tables. Schema context (all tables and relationships) is assembled using `prepare_tables_and_schema()` to give the LLM full database context. Schema and profiling data are sent to the LLM with structured prompts. The LLM generates metadata enrichments based on: * Column names and types * Profiling statistics (completeness, uniqueness, patterns) * Table relationships and foreign keys * Cross-table context (schema\_info for understanding relationships) Generated metadata is validated against the schema. PII classifications are cross-checked with column patterns and profiling data. Enriched metadata is stored in the Data Governance database, linked to the corresponding data asset records. ## Schema Context for Better Enrichment The enrichment pipeline provides full schema context to the LLM for higher-quality results: ```python theme={null} from dq.utils.schema_helpers import prepare_tables_and_schema # Gather schema context schema_info, tables_info = await prepare_tables_and_schema(connection) # schema_info: all tables in the database (for relationship understanding) # tables_info: selected tables for enrichment # Pass both to the LLM prompt prompt = build_enrichment_prompt( tables=tables_info, schema_context=schema_info # LLM understands cross-table relationships ) ``` Providing full schema context allows the LLM to understand foreign key relationships and generate more accurate business descriptions. For example, it can identify that `customer_id` in the `orders` table references the `customers` table and generate appropriate descriptions. ## Concurrency Control Enrichment workflows use the same hybrid concurrency model as profiling: | Level | Mechanism | Control | | -------------- | ---------------------- | --------------------------------------------- | | **Flow-level** | `ConcurrentTaskRunner` | Max concurrent table enrichment tasks | | **Task-level** | `asyncio.Semaphore` | Max concurrent LLM API calls within each task | The task-level semaphore is particularly important for enrichment to avoid hitting LLM API rate limits: ```python theme={null} @task async def enrich_table(table_info, schema_context): semaphore = asyncio.Semaphore(config.max_concurrent_llm_calls) columns = table_info["columns"] async def enrich_column(column): async with semaphore: return await llm_enrich(column, schema_context) results = await asyncio.gather(*[enrich_column(c) for c in columns]) return results ``` ## LLM Configuration The enrichment pipeline uses LiteLLM and is provider-agnostic — defaults to Vertex AI but works with any LiteLLM-supported provider: | Setting | Source | Description | | ---------------------------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `VERTEXAI_PROJECT` / `VERTEXAI_LOCATION` | Environment variables | Vertex AI project and region (default `global`). LiteLLM also accepts the Google Cloud standards `GOOGLE_CLOUD_PROJECT` / `GOOGLE_CLOUD_LOCATION`. | | `GOOGLE_APPLICATION_CREDENTIALS` | Environment variable | Path to ADC JSON (local dev). Use Workload Identity in production. | | `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` | Environment variables | Only required if you swap the model to a non-Vertex provider | | Model | `config.yaml` | LiteLLM model identifier (e.g., `vertex_ai/gemini-3.5-flash`, `vertex_ai/claude-opus-4-8`, `openai/gpt-5.5`) | | Max tokens | `config.yaml` | Response token limit | | Temperature | `config.yaml` | Creativity level (low for factual metadata) | Enrichment prompts include database schema information and profiling statistics. Ensure your LLM provider's data handling policies are compatible with the sensitivity level of your data. For Restricted data, consider using self-hosted LLMs. ## DQ Rule Generation As part of enrichment, the LLM suggests data quality (DQ) rules for each column: | Rule Type | Example | | ---------------- | ------------------------------------------------ | | **Completeness** | "email column should be 100% non-null" | | **Format** | "phone column should match pattern XXX-XXX-XXXX" | | **Range** | "age column should be between 0 and 150" | | **Uniqueness** | "customer\_id should be unique" | | **Referential** | "product\_id should exist in products table" | | **Consistency** | "end\_date should be after start\_date" | Generated rules are stored as structured metadata and can be used by the DQ monitoring system to track quality over time. ## Enrichment Results Access enrichment results via the Data Governance API: ```json theme={null} { "table": "customers", "enriched_at": "2026-04-03T12:00:00Z", "business_description": "Customer master data containing contact information and account details for all registered customers.", "domain": "Sales", "sensitivity": "Confidential", "columns": [ { "name": "email", "business_description": "Primary email address for customer communication", "pii_classification": "PII", "suggested_rules": [ {"type": "completeness", "threshold": 1.0}, {"type": "format", "pattern": "email"} ] } ] } ``` ## Next Steps Profile your data before enrichment to provide statistical context. Learn about the Prefect orchestration system that runs enrichment. Understand PII classification in the context of GDPR. Review how enrichment outputs map to data classification levels. # Data Profiling Source: https://docs.emergence.ai/data-governance/data-profiling How the Data Governance solution profiles data assets at the column and table level to assess data quality, completeness, and statistical characteristics. # Data Profiling Data profiling is the foundation of the Data Governance solution. It analyzes connected databases to produce statistical profiles of tables and columns, identifying data quality issues, completeness gaps, and structural anomalies. ## What Data Profiling Captures For each column in a profiled table, the system computes: | Metric | Description | | -------------------------- | -------------------------------------------- | | **Completeness** | Percentage of non-null values | | **Uniqueness** | Ratio of distinct values to total rows | | **Data type distribution** | Actual types vs declared column type | | **Min / Max / Mean** | Statistical bounds for numeric columns | | **Standard deviation** | Spread of numeric values | | **Pattern analysis** | Common formats (email, phone, date patterns) | | **Top N values** | Most frequent values and their counts | | **Outlier detection** | Values beyond 2 standard deviations | For each table: | Metric | Description | | -------------------------- | ------------------------------- | | **Row count** | Total number of records | | **Column count** | Number of columns | | **Completeness score** | Average column completeness | | **Freshness** | Last update timestamp | | **Primary key analysis** | Key uniqueness and completeness | | **Foreign key validation** | Referential integrity checks | | **Duplicate detection** | Potential duplicate rows | ## Profiling Workflow ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% graph TB API["Data Governance API\n(trigger / schedule)"] FLOW["Prefect Flow\n(profile_database)"] CTR["ConcurrentTaskRunner\n(max 10 tables parallel)"] SEM["asyncio.Semaphore\n(20 column analyses / table)"] SRC[("Source Database\n(via data connection)")] subgraph Results["Results"] COL["Column metrics\n(completeness, uniqueness,\npatterns, outliers)"] TBL["Table metrics\n(row counts, freshness,\nPK/FK validation)"] end DB[("datareadiness DB\n(results + timestamps)")] SC["Scorecard"] API --> FLOW --> CTR --> SEM --> SRC SEM --> COL & TBL COL & TBL --> DB --> SC classDef trigger fill:#56B4E9,stroke:#555555,color:#000 classDef orch fill:#E69F00,stroke:#555555,color:#000 classDef gate fill:#F0E442,stroke:#555555,color:#000 classDef src fill:#0072B2,stroke:#555555,color:#fff classDef result fill:#009E73,stroke:#555555,color:#000 class API trigger class FLOW,CTR orch class SEM gate class SRC src class COL,TBL,DB,SC result ``` Data profiling is orchestrated as a Prefect workflow: An administrator or data steward selects the tables and schemas to profile via the Data Governance API. The Prefect flow launches with `ConcurrentTaskRunner` for parallel table processing. Each table is profiled as an independent task. For each table, the workflow queries the database to compute column-level statistics. Queries are optimized using sampling for large tables. Profiling results are stored in the Data Governance database (`datareadiness`) with timestamps for historical tracking. A data quality scorecard summarizes the profiling results across all tables, highlighting issues that need attention. ## Concurrency Control Profiling workflows use hybrid concurrency to efficiently process large databases without overwhelming the source: ```python theme={null} @flow(task_runner=ConcurrentTaskRunner(max_workers=10)) async def profile_database(tables: list[str]): # Process up to 10 tables concurrently tasks = [profile_table(table) for table in tables] await asyncio.gather(*tasks) @task async def profile_table(table: str): semaphore = asyncio.Semaphore(20) # Within each table, run up to 20 column analyses concurrently async with semaphore: await analyze_column(column) ``` | Level | Mechanism | Configuration | | -------------- | ------------------------------------- | ------------- | | **Flow-level** | `ConcurrentTaskRunner(max_workers=N)` | `config.yaml` | | **Task-level** | `asyncio.Semaphore(N)` | `config.yaml` | Adjust the `max_workers` and semaphore limits based on your source database capacity. Start with conservative values and increase based on observed database load. ## Integration with Data Connections Profiling uses data connections registered in the platform's Assets service: 1. The data steward selects a registered data connection 2. The profiling workflow retrieves connection credentials from the Secrets API at runtime 3. A read-only database session is established for profiling queries 4. All profiling queries use the read-only user configured during [data source setup](/guides/data-source-setup) Profiling queries can be resource-intensive on large tables. Schedule profiling runs during off-peak hours or configure sampling thresholds for tables with millions of rows. ## Profiling Results Profiling results are accessible via the Data Governance API: ```bash theme={null} # Get profiling results for a table by FQN (e.g. analytics-db.public.customers) curl -X GET \ "https:///api/data-readiness/data-assets/profiles/tables/by-fqn?fqn=" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " ``` Results include: ```json theme={null} { "table": "customers", "profiled_at": "2026-04-03T10:00:00Z", "row_count": 150000, "columns": [ { "name": "email", "completeness": 0.98, "uniqueness": 0.97, "pattern": "email", "top_values": ["gmail.com", "outlook.com", "yahoo.com"] }, { "name": "phone", "completeness": 0.72, "uniqueness": 0.95, "pattern": "phone_us", "null_count": 42000 } ], "quality_score": 0.85 } ``` ## Scheduling Profiles Profiling can be scheduled for recurring execution via the platform's scheduling system: * **Daily profiles**: Track data quality trends over time * **Weekly profiles**: Standard cadence for most datasets * **On-demand**: Triggered manually for new data sources or after schema changes ## Next Steps Enrich metadata after profiling with LLM-powered descriptions. Learn about the Prefect workflow orchestration system. Connect a database to start profiling. Understand how profiling data is classified and protected. # CRAFT Data Governance Overview Source: https://docs.emergence.ai/data-governance/overview Automated data profiling, LLM-powered metadata enrichment, data quality rule generation, and scorecard reporting orchestrated by Prefect workflows. # Data Governance Overview Assess and improve the quality and completeness of your enterprise data assets with **Data Governance**. Profile data, enrich metadata using LLMs, generate data quality (DQ) rules, and produce scorecards, all orchestrated through automated workflows. This solution implements two of the [CRAFT modules](/getting-started/concepts#craft-modules): data profiling, coverage, and policy-compliance checks implement **CRAFT Assess**; metadata enrichment, DQ rule generation, and classification implement **CRAFT Enrich**. Data Governance uses **Prefect** for workflow orchestration with `ConcurrentTaskRunner` for flow-level parallelism and `asyncio.Semaphore` for task-level concurrency control. Workflows run on a Kubernetes work pool for production-grade scalability. ## Who Is It For Profile data assets, review LLM-generated metadata enrichment, and track data quality scores across your organization's data landscape. Generate data quality rules, review scorecards, and ensure metadata standards are met across the organization for regulatory compliance. Configure data connections, manage profiling workflows, and integrate data quality checks into your data pipelines. Discover and understand data assets through enriched metadata, quality scores, and automated profiling summaries. ## Key Capabilities Automatically analyze data assets to understand structure, distributions, patterns, and anomalies. Profiling produces statistical summaries that feed into enrichment and DQ rule generation. Use LLMs to automatically enrich metadata with business descriptions, data classifications, semantic types, and governance annotations. Schema-aware prompts ensure high-quality results. Automatically generate data quality rules based on profiling results and schema analysis. LLMs produce rules that are contextually relevant to the data's business meaning and relationships. Generate data quality scorecards that track metrics, violations, and trends across data assets. Scorecards provide executive-level visibility into data health. ### Service Architecture ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% graph TB subgraph API["Data Governance API (FastAPI)"] WF["Workflows module"] MT["Metrics module"] DA["Data Assets module"] VL["Violations module"] TM["Talk to Metadata module"] end subgraph Orch["Orchestration (Prefect)"] PP["profile_database"] PE["enrich_metadata"] PD["generate_dq_rules"] end subgraph Platform["Platform Services"] GOV["Governance\n(auth + authz)"] ASS["Assets\n(data connections)"] SEC["Secrets API"] end LLM["LLM (OpenAI)"] DB[("PostgreSQL\n(datareadiness)")] API --> Orch API --> Platform PE --> LLM PD --> LLM Orch --> DB API --> DB classDef api fill:#E69F00,stroke:#555555,color:#000 classDef orch fill:#56B4E9,stroke:#555555,color:#000 classDef plat fill:#0072B2,stroke:#555555,color:#fff classDef ext fill:#CC79A7,stroke:#555555,color:#000 classDef db fill:#009E73,stroke:#555555,color:#000 class WF,MT,DA,VL,TM api class PP,PE,PD orch class GOV,ASS,SEC plat class LLM ext class DB db ``` | API Module | Purpose | | -------------------- | ------------------------------------------------------------------------------------------------------------------------- | | **Workflows** | Trigger and monitor workflow runs | | **Metrics** | Track data quality metrics | | **Data Assets** | Manage registered data assets; batch profile lookups via `POST /profiles/columns/batch` and `POST /profiles/tables/batch` | | **Violations** | Track DQ rule violations | | **Talk to Metadata** | LLM-powered metadata interaction | Each domain module follows a consistent structure: * `models/requests/` -- Pydantic request models * `models/responses/` -- Pydantic response models * `services/` -- Business logic * `routes.py` -- FastAPI router * `dependencies.py` -- Dependency injection ## Workflow Orchestration Prefect orchestrates all data processing workflows with a hybrid concurrency model: Flows use `ConcurrentTaskRunner` to run multiple table-level tasks in parallel. The concurrency limit is configurable via `config.yaml` (e.g., `DQ_RULE_GEN_MAX_CONCURRENT_TABLES`). This provides: * Parallel processing across tables * Visibility in the Prefect UI * Configurable concurrency limits Within each Prefect task, `asyncio.Semaphore` controls fine-grained concurrency for operations like LLM API calls. Limits are configurable via `config.yaml` (e.g., `DQ_RULE_GEN_MAX_CONCURRENT_RULES`). This prevents: * API rate limit exhaustion * Memory pressure from too many concurrent operations * Unpredictable costs from uncontrolled LLM calls Production workflows run on a Kubernetes work pool named `kubernetes-pool`. Flow definitions are managed as code in `scripts/prefect/deploy.py` (single source of truth). Key details: * Work pool is auto-created during setup * Docker images are tagged with git commit hashes for traceability * Secrets are injected via Helm `secretKeyRef` into pod environment variables Workflow and infrastructure errors are wrapped in a `WorkflowRuntimeError` before being returned to API consumers. This normalizes raw Prefect and infrastructure stack traces into structured error responses, making errors actionable without exposing internal implementation details. ## DQ Rule Generation DQ rule generation combines profiling results with LLM intelligence: The system loads the complete schema context (`prepare_tables_and_schema()`) including all tables, columns, foreign keys, and relationships. Users can select specific tables for rule generation, or process the entire database. When `selected_tables` is specified in event-triggered ingestion, only those tables are processed — the filter is enforced server-side. Schema context and profiling statistics are sent to the LLM, which generates data quality rules that understand business semantics and cross-table relationships. Generated rules are validated against the schema and persisted to the database. Rules include dimension (completeness, accuracy, consistency, etc.) and severity levels. ## Database Architecture | Aspect | Details | | ---------------- | ------------------------------------------------------------------------------------------------ | | **ORM** | SQLAlchemy 2.0+ with async support | | **Database** | PostgreSQL 18+ (`datareadiness` database) | | **Migrations** | Alembic with clean two-migration approach: initial schema + seed data | | **Primary keys** | UUID v4 using PostgreSQL `gen_random_uuid()` | | **Timestamps** | `TIMESTAMPTZ` (TIMESTAMP WITH TIME ZONE) | | **Seed data** | 112 records: 11 workflows + 11 configs + 12 dimensions + 16 metrics + 37 violations + 25 DQ tags | The project uses a **clean two-migration approach**: one migration for the complete schema (tables, indexes, constraints) and a second for seed data. Seed data uses idempotent `INSERT ... ON CONFLICT DO NOTHING` for safe re-runs. ## Configuration Data Governance uses **OmegaConf** for hierarchical configuration: | Source | Priority | Example | | --------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | Environment variables | Highest | `VERTEXAI_PROJECT` / `VERTEXAI_LOCATION` (LLM, CRAFT default), `REDIS_PASSWORD`; `OPENAI_API_KEY` only if a non-Vertex provider is configured | | `config.yaml` | Default | Database URLs, concurrency limits, workflow parameters | | Internal references | -- | `${.field_name}` for self-referencing config values | Configuration is loaded once at module import via `common/config.py`. Environment variables can be referenced in YAML using `${oc.env:VAR_NAME, default}` syntax. ## Supported Data Sources Data Governance connects to external data sources for profiling, enrichment, and DQ rule generation. Supported connector types: | Connector | Auth | Notes | | -------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **PostgreSQL** | Username + password | Native async (`asyncpg`) | | **Redshift** | Username + password | Via `redshift_connector` | | **Snowflake** | RSA key-pair only | `asyncio.to_thread()` wrapping (no native async driver); account identifier format (e.g., `ORGNAME-ACCOUNTNAME`); service users must be `TYPE=SERVICE` with a mounted `.p8` private key | Connector selection is determined by the data connection type registered in the Assets service. Credentials are injected via Kubernetes secrets at runtime — never stored in the database. ## Platform Integration Data Governance integrates with the em-runtime platform for shared services: | Capability | Platform Service | Integration | | -------------------- | --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | | **Authentication** | Governance (8000) | JWT validation via Keycloak | | **Authorization** | Governance (8000) | Permission checks via OpenFGA SDK | | **Data connections** | Assets (8000) | Registered data source configurations (PostgreSQL, Redshift, Snowflake) | | **Secrets** | Infisical or ESO + GCP Secret Manager, see [Secrets Management](/security/secrets-management) | Credentials for data connections and API keys | ## Tech Stack | Technology | Purpose | | --------------- | ------------------------------------------------ | | Python 3.12+ | Service and workflow implementation | | FastAPI | Async REST API framework | | Prefect | Workflow orchestration with Kubernetes work pool | | PostgreSQL 18+ | Primary data store | | Redis 7+ | Caching and messaging | | SQLAlchemy 2.0+ | Async ORM with `asyncpg` | | OmegaConf | Hierarchical YAML configuration | | OpenAI API | LLM-powered enrichment and rule generation | | Docker Compose | Local development (PostgreSQL, Redis, Prefect) | | uv | Python workspace monorepo management | | pytest + memray | Testing with memory profiling | ## Next Steps Learn how automated profiling analyzes your data assets. Understand LLM-powered metadata enrichment workflows. Configure and monitor Prefect-orchestrated data governance workflows. Connect your data sources to the platform. # Workflows Source: https://docs.emergence.ai/data-governance/workflows How the Data Governance solution uses Prefect for workflow orchestration, including profiling, enrichment, and DQ rule generation workflows. # Workflow Orchestration The Data Governance solution uses **Prefect** for orchestrating data profiling, metadata enrichment, and data quality (DQ) rule generation workflows. Prefect provides task scheduling, concurrency control, observability, and failure handling. ## Architecture | Component | Purpose | | ------------------ | -------------------------------------------- | | **Prefect Server** | Workflow API and UI (port 4200) | | **Prefect Worker** | Executes flows in Kubernetes pods | | **Work Pool** | Kubernetes work pool named `kubernetes-pool` | | **PostgreSQL** | Prefect metadata storage | All Prefect services are deployed alongside the Data Governance API via Docker Compose (local dev) or Helm (Kubernetes). Flow definitions are managed as code in `scripts/prefect/deploy.py` -- the single source of truth for all deployments. This file registers flows, configures schedules, and sets job variables. ```python theme={null} # scripts/prefect/deploy.py (simplified) from prefect import flow from prefect.deployments import Deployment deployments = [ Deployment.build_from_flow( flow=profile_database, name="profile-database", work_pool_name="kubernetes-pool", job_variables={...} ), Deployment.build_from_flow( flow=enrich_metadata, name="enrich-metadata", work_pool_name="kubernetes-pool", job_variables={...} ), ] ``` `prefect.yaml` is deprecated. All flow definitions are managed in `deploy.py`. Secrets are read from pod environment variables (injected via Helm secretKeyRef) and passed as plain strings to Prefect job\_variables. ## Available Workflows ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR TRG["API / Schedule\nTrigger"] subgraph WF["Prefect Workflows (ConcurrentTaskRunner)"] PRF["profile_database\n(Profiling)"] ENR["enrich_metadata\n(Enrichment)"] DQG["generate_dq_rules\n(DQ Generation)"] end SEM["asyncio.Semaphore\n(task-level rate gate)"] LLM["LLM API\n(OpenAI)"] DB[("PostgreSQL\n(datareadiness)")] APR{"User\nApproval"} SC["Scorecard\nUpdate"] TRG --> PRF & ENR & DQG PRF & DQG --> SEM ENR --> SEM SEM --> LLM PRF & ENR & DQG --> DB DB --> APR APR -->|"approved"| SC classDef trigger fill:#56B4E9,stroke:#555555,color:#000 classDef wf fill:#E69F00,stroke:#555555,color:#000 classDef gate fill:#F0E442,stroke:#555555,color:#000 classDef ext fill:#CC79A7,stroke:#555555,color:#000 classDef store fill:#009E73,stroke:#555555,color:#000 classDef decision fill:#F0E442,stroke:#555555,color:#000 classDef terminal fill:#009E73,stroke:#555555,color:#000 class TRG trigger class PRF,ENR,DQG wf class SEM gate class LLM ext class DB store class APR decision class SC terminal ``` ### Data Profiling Flow Analyzes connected data sources to produce statistical profiles: | Setting | Value | | --------------- | ----------------------------------------------------------------------------------------------------------------------------- | | **Flow name** | `profile-database` | | **Concurrency** | `ConcurrentTaskRunner(max_workers=10)` | | **Input** | Data connection ID, schema/table targets (database-level targets are not supported, specify schema or table), sampling config | | **Output** | Column and table-level profiling results | Profiling, Metadata Assessment, and DQ Assessment workflows require **schema-level or table-level targets**. Database-level targets (entire database) are not supported, specify individual schemas or tables to process. ### Metadata Enrichment Flow Uses LLMs to generate business descriptions and classifications: | Setting | Value | | --------------- | --------------------------------------------------------------- | | **Flow name** | `enrich-metadata` | | **Concurrency** | `ConcurrentTaskRunner(max_workers=5)` | | **Input** | Data connection ID, profiling results | | **Output** | Business descriptions, PII classifications, DQ rule suggestions | ### DQ Rule Generation Flow Generates data quality rules based on profiling data and LLM analysis: | Setting | Value | | --------------- | ----------------------------------------------------------------------- | | **Flow name** | `generate-dq-rules` | | **Concurrency** | `ConcurrentTaskRunner` with `asyncio.Semaphore` for LLM calls | | **Input** | Schema context, profiling results, enrichment metadata | | **Output** | Structured DQ rules per column | | **Config** | `DQ_RULE_GEN_MAX_CONCURRENT_TABLES`, `DQ_RULE_GEN_MAX_CONCURRENT_RULES` | DQ rule generation does **not** require a reference pack. Earlier versions passed an industry context / reference pack into the workflow; that input was removed because it caused crashes under Prefect 3.x. Generation runs purely from schema context, profiling results, and metadata. ## Workflow approval and the scorecard When you approve ("Save") a profiling or enrichment workflow run, the platform applies two changes atomically: 1. **Tombstone affected metric snapshots.** Affected `metric_snapshots` rows are written with `numerator=0, denominator=0`. This invalidates the prior score so the scorecard does not display stale numbers while the next assessment is pending. 2. **Resolve all matching OPEN violations.** Every `OPEN` violation tied to an invalidated metric is closed in the same transaction, not just the most recent. Previously the resolve step closed only the latest violation, leaving older OPEN duplicates on the dashboard. Net effect for users: scorecards update in real time. After Save, expect to see the affected scorecard cells flip to **N/A** ("pending next assessment") and any related violation entries disappear from the open-violations view. ### FQN scoping for violations Violations are filtered by `asset_fqn` using an exact-or-child boundary: a filter for `db.schema.card` matches `db.schema.card` itself and any column-level child like `db.schema.card.amount`, but does **not** match sibling tables with prefix collisions like `db.schema.cardtransaction`. Earlier versions used a plain `startswith`, which produced false matches on sibling tables sharing a name prefix. ### Ingestion triggered by `asset.created` When an ingestion run is triggered by an `asset.created` event (or kicked off manually with no schemas listed in `config`), the workflow ingests **all schemas** in the connection, scoped by `config.database`. Earlier versions only ingested the first schema returned by the connection, which led to silent data gaps when a connection exposed multiple schemas. ## Concurrency Model Workflows use a **hybrid concurrency** model combining Prefect's task runner with asyncio semaphores: ### Flow-Level Concurrency The `ConcurrentTaskRunner` controls how many table-level tasks run simultaneously: ```python theme={null} @flow(task_runner=ConcurrentTaskRunner(max_workers=10)) async def profile_database(tables: list[str]): # Up to 10 tables profiled concurrently tasks = [profile_table.submit(table) for table in tables] results = await asyncio.gather(*[t.result() for t in tasks]) return results ``` ### Task-Level Concurrency Within each task, `asyncio.Semaphore` controls fine-grained concurrency (e.g., LLM API calls): ```python theme={null} @task async def enrich_table(table_info): semaphore = asyncio.Semaphore(20) # Max 20 concurrent LLM calls async def process_column(column): async with semaphore: return await llm_client.enrich(column) results = await asyncio.gather( *[process_column(col) for col in table_info.columns] ) return results ``` | Level | Mechanism | Visibility | | ----- | ---------------------- | --------------------- | | Flow | `ConcurrentTaskRunner` | Visible in Prefect UI | | Task | `asyncio.Semaphore` | Internal to the task | Configure concurrency limits via `config.yaml` rather than hardcoding. This allows tuning based on the source database capacity and LLM API rate limits. ## Workflow Management ### Prefect UI The Prefect UI (port 4200) provides visibility into workflow execution: * **Flow runs**: Status, duration, logs for each execution * **Task runs**: Individual task status within a flow * **Deployments**: Registered flow definitions and schedules * **Work pool**: Worker status and queue depth ## Configuration Workflow configuration is managed via **OmegaConf** with environment variable overrides: ```yaml theme={null} # config.yaml profiling: max_concurrent_tables: 10 sample_threshold: 1000000 # Sample tables with more than 1M rows sample_size: 100000 enrichment: max_concurrent_tables: 5 max_concurrent_llm_calls: 20 model: "vertex_ai/gemini-3.5-flash" temperature: 0.1 dq_rules: max_concurrent_tables: 10 max_concurrent_rules: 20 ``` **Precedence:** Environment variables > `config.yaml` > defaults Environment variables can be referenced in config via OmegaConf's `${oc.env:VAR_NAME, default}` syntax. The CRAFT default is Vertex AI — common env vars are `${oc.env:VERTEXAI_PROJECT}` / `${oc.env:VERTEXAI_LOCATION, global}` (also accepts the Google Cloud SDK standards `GOOGLE_CLOUD_PROJECT` / `GOOGLE_CLOUD_LOCATION`) with `GOOGLE_APPLICATION_CREDENTIALS` for ADC. Provider-specific keys (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`) are only required when the configured model routes to a non-Vertex provider. ## Error Handling | Error Type | Handling | | --------------------------- | ----------------------------------------------------------- | | Database connection failure | Flow fails fast with descriptive error | | LLM API rate limit | Retry with exponential backoff via tenacity | | LLM API timeout | Retry up to 3 times, then skip column and continue | | Single table failure | Log error, continue processing remaining tables | | Infrastructure failure | Prefect marks flow as failed; manual retry available via UI | ## Kubernetes Deployment In Kubernetes, Prefect workflows run as pods in the `kubernetes-pool` work pool: | Component | Deployment | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Prefect Server | StatefulSet with PostgreSQL backend | | Prefect Worker | Deployment watching the kubernetes-pool | | Flow runs | Ephemeral pods created per flow execution | | Secrets | Injected via Helm `secretKeyRef`. For Vertex AI (CRAFT default): mount the ADC JSON via `GOOGLE_APPLICATION_CREDENTIALS` or use Workload Identity. For non-Vertex providers: `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / `GEMINI_API_KEY` per provider. | ## Next Steps Learn about the profiling workflow in detail. Understand the LLM-powered enrichment pipeline. Connect a database to use with governance workflows. # Analysis Agent Source: https://docs.emergence.ai/data-insights/analysis-agent How the Insights Agent performs multi-step analysis with LLM-generated Python code, reasoning loops, and tool orchestration. # Analysis Agent The **Insights Agent** is the reasoning engine at the center of the Data Insights solution. It receives natural-language questions, determines the optimal analysis approach, and orchestrates tools and sub-agents to produce comprehensive answers. ## Role in the Architecture The Insights Agent sits between the Talk2Data Service (user-facing gateway) and specialized agents like Text2SQL: ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TD T2D["Talk2Data Service (8080)
user-facing gateway"] IA["Insights Agent (8002)
reasoning engine (agentic loop)"] SQL["Text2SQL Agent (8001)
SQL generation + execution"] CODE["Coding Agent (8004)
Python code generation"] MCP["MCP Plotly Server (8000)
visualization tools"] T2D -->|A2A| IA IA -->|A2A| SQL IA -->|A2A| CODE IA -->|MCP| MCP classDef gateway fill:#56B4E9,stroke:#555555,color:#000 classDef agent fill:#E69F00,stroke:#555555,color:#000 classDef tool fill:#009E73,stroke:#555555,color:#000 class T2D gateway class IA,SQL,CODE agent class MCP tool ``` It acts as an **agentic loop** -- iteratively reasoning about the question, calling tools, analyzing intermediate results, and deciding whether to continue analysis or return a final answer. ## Capabilities For questions that can be answered with database queries, the Insights Agent delegates to the Text2SQL Agent: * "What were the top 10 products by revenue last quarter?" * "How many active users do we have per region?" * "Show me the trend of order values over the past 12 months" The Insights Agent adds value by interpreting the results, identifying patterns, and generating follow-up analysis suggestions. For complex analytical questions, the Insights Agent generates and executes Python code: * Statistical analysis (correlations, distributions, outlier detection) * Data transformations and pivots * Time series forecasting * Custom calculations not expressible in SQL ```python theme={null} # Example: LLM-generated Python for correlation analysis import pandas as pd import numpy as np df = pd.DataFrame(query_results) correlation = df[['revenue', 'marketing_spend']].corr() outliers = df[df['revenue'] > df['revenue'].mean() + 2 * df['revenue'].std()] ``` For complex questions requiring multiple data gathering and analysis steps: 1. Query database for raw data 2. Analyze initial results 3. Generate follow-up queries based on findings 4. Synthesize a comprehensive answer 5. Generate visualizations for key findings ## Agentic Loop The Insights Agent operates as an iterative reasoning loop: The agent receives the user's question along with conversation history (for multi-turn context) and the data connection schema. The LLM determines the analysis strategy: direct SQL query, multi-step analysis, Python computation, or visualization. The agent calls the appropriate tool or sub-agent. Results are captured and fed back into the reasoning loop. The LLM evaluates whether the results answer the question completely. If not, it plans additional steps. Once the analysis is complete, the agent formats the final response with text, data, and optional visualizations. ## Tool Orchestration The Insights Agent has access to multiple tools: | Tool | Purpose | Invocation | | --------------------- | --------------------------------------------------------------------------- | ------------------------------ | | **Text2SQL Agent** | Generate and execute SQL queries | A2A delegation (port 8001) | | **Coding Agent** | Run LLM-generated Python code for statistics, transformations, and analysis | A2A delegation (port 8004) | | **MCP Plotly Server** | Generate interactive Plotly charts | MCP tool call (port 8000) | | **Schema inspector** | Retrieve database schema details | Direct database metadata query | ## Event Streaming Throughout the analysis, the Insights Agent emits real-time events via the A2A protocol: ```python theme={null} # Status events keep the user informed of progress await event_queue.put(TaskStatusUpdateEvent( status="working", message="Analyzing your data..." )) # Artifact events deliver results await event_queue.put(TaskArtifactUpdateEvent( artifact=Artifact( parts=[DataPart(data=results), TextPart(text=analysis)] ) )) ``` Users see progress messages in the chat interface: "Analyzing schema...", "Running query...", "Computing statistics...", followed by the final answer. ## LLM Configuration The Insights Agent uses LiteLLM for provider-agnostic LLM access: | Setting | Configuration | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Client** | `commons.llm.LLMClient` wrapping LiteLLM | | **Model format** | `provider/model` (e.g., `vertex_ai/gemini-3.5-flash`, `vertex_ai/claude-opus-4-8`, `openai/gpt-5.5`) | | **Observability** | Langfuse LLM tracing auto-enabled when `LANGFUSE_HOST` is set, traces LLM calls, token usage, and cost. See [Langfuse Setup](/guides/langfuse-setup) for configuration. | | **Configuration** | `pydantic_settings.BaseSettings` with `.env` file support | ## Error Handling The Insights Agent handles errors at each stage of the analysis: | Error | Handling | | --------------------------- | ----------------------------------------------------------- | | Text2SQL fails | Retry with rephrased prompt or fall back to Python analysis | | Python execution error | Return error details to LLM for correction and retry | | LLM timeout or rate limit | Retry with exponential backoff via tenacity | | Data connection unavailable | Report connection error to the user | ## Conversation State The agent maintains state via the Talk2Data Service's session system: * **Sessions**: Persistent conversation containers linked to a data connection * **Messages**: Full conversation history with user and assistant messages * **Artifacts**: Query results, visualizations, and analysis outputs stored per session * **Feedback**: User feedback on answer quality for continuous improvement ## Next Steps See the full conversational analytics pipeline. Deep dive into the SQL generation agent. Learn how the Insights Agent generates charts. # Chat With Data Source: https://docs.emergence.ai/data-insights/chat-with-data How the CRAFT conversational data analytics pipeline works, from natural-language questions to SQL-generated answers with multi-turn context. # Chat With Data CRAFT's Data Insights solution enables users to ask natural-language questions against their databases and receive instant answers, visualizations, and analytical insights. The system uses a multi-agent architecture built on Google's A2A protocol to orchestrate the full pipeline from question to answer. ## How It Works When a user asks a question, the platform orchestrates a pipeline across three agents: The user types a natural-language question in the chat interface (e.g., "What were the top 10 products by revenue last quarter?"). The Talk2Data Service receives the request and creates a session context. The Insights Agent receives the question and determines the best approach. It may: * Route directly to the Text2SQL Agent for database queries * Execute a multi-step analysis plan for complex questions * Generate Python code for statistical analysis * Produce visualizations for data-oriented answers The Text2SQL Agent converts the natural-language question into SQL using schema-aware generation. It validates the SQL with `sqlglot`, executes it against the connected database, and returns the results. Results flow back through the agent chain via A2A events. The user sees real-time progress messages ("Analyzing schema...", "Generating SQL...", "Executing query...") followed by the final answer with data tables and optional visualizations. ## Architecture The Data Insights system consists of three services communicating via the A2A protocol: | Service | Port | Role | | --------------------- | ---- | ------------------------------------------------------------------------------------- | | **Talk2Data Service** | 8080 | FastAPI REST + SSE gateway. Manages sessions, chat history, and artifact storage | | **Insights Agent** | 8002 | Main agentic loop. Reasoning engine that orchestrates tool calls and agent delegation | | **Text2SQL Agent** | 8001 | Specialized agent for NL-to-SQL generation, validation, and execution | ### Communication Flow ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TD Client["Browser / API Client"] T2D["Talk2Data Service (8080)"] IA["Insights Agent (8002)"] SQL["Text2SQL Agent (8001)"] DB[("Customer Database")] Client -->|"REST + SSE"| T2D T2D -->|"A2A (JSON-RPC 2.0 / SSE)"| IA IA -->|"A2A (JSON-RPC 2.0 / SSE)"| SQL SQL -->|"SQL via data connection"| DB classDef client fill:#56B4E9,stroke:#555555,color:#000 classDef svc fill:#E69F00,stroke:#555555,color:#000 classDef data fill:#009E73,stroke:#555555,color:#000 class Client client class T2D,IA,SQL svc class DB data ``` ## Multi-Turn Conversations The system maintains conversation context across multiple turns using the A2A **Context ID** (mapped to the session ID): * Each session has a conversation history stored in PostgreSQL * Previous questions and answers are included in the LLM context window * The Insights Agent can reference earlier results ("Now show me the same data as a pie chart") * Session artifacts (query results, visualizations) persist for the duration of the session * **Session context items** (`/sessions/{id}/context`) allow storing key-value metadata per session (e.g., selected data connection, user preferences) Multi-turn context allows follow-up questions like "What about last year?" or "Break that down by region" without repeating the full original question. ## A2A Protocol Integration All inter-agent communication uses Google's A2A protocol: | Concept | Usage | | ----------------- | ---------------------------------------------------------------------------------------------- | | **Agent Cards** | Each agent exposes `/.well-known/agent-card.json` describing capabilities and skills | | **Message Parts** | `TextPart` for questions/answers, `DataPart` for datasource metadata, `FilePart` for artifacts | | **Events** | `TaskStatusUpdateEvent` for progress, `TaskArtifactUpdateEvent` for results | | **Streaming** | Server-Sent Events (SSE) for real-time updates to the frontend | ## LLM Integration The Data Insights agents use **LiteLLM** for provider-agnostic LLM access: ```python theme={null} from commons.llm import LLMClient client = LLMClient() # Model format: "provider/model" response = await client.complete( model="vertex_ai/gemini-3.5-flash", messages=[{"role": "user", "content": question}] ) ``` Supported providers include Gemini, GPT, Claude, and any LiteLLM-compatible model. LLM observability is provided by **Langfuse** when `LANGFUSE_HOST` is configured, traces every LLM call with token counts, cost, and latency. See [Langfuse Setup](/guides/langfuse-setup) for observability configuration. ## Data Connection Requirements Chat with Data requires a registered data connection in the platform: 1. Register a PostgreSQL database via the Assets API (see [Data Source Setup](/guides/data-source-setup)) 2. Create a session linked to the data connection ID 3. The Text2SQL Agent fetches the database schema and uses it for context-aware SQL generation ## Next Steps Deep dive into the NL-to-SQL generation pipeline. Learn about the Insights Agent's reasoning and analysis capabilities. Understand how charts and visualizations are generated. Connect a database to start chatting with your data. # CRAFT Data Insights Overview Source: https://docs.emergence.ai/data-insights/overview Multi-agent conversational data analytics using the A2A protocol for natural-language to SQL generation, analysis, and visualization. # Data Insights Overview Ask natural-language questions about your data and get instant SQL-generated answers, visualizations, and insights. **Data Insights** coordinates a team of AI agents that generate SQL, execute queries, analyze results, and produce visualizations -- all in real time via streaming. Data Insights is built on the **A2A (Agent-to-Agent) protocol**, an open standard for inter-agent communication using JSON-RPC 2.0 over Server-Sent Events (SSE). Each agent is independently deployable and discoverable via Agent Cards. ## Who Is It For Ask questions about your data in plain language and get instant answers, visualizations, and reports -- no SQL or technical skills required. Explore data through multi-turn conversations, generate visualizations, and export results. The AI handles SQL generation so you can focus on analysis. Get AI-powered insights from enterprise data to support strategic decisions. Generate on-demand reports and discover trends through natural-language queries. Validate SQL generation, review query plans, and configure data connections. Monitor agent performance and query execution through observability tooling. ## How It Works ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff', 'actorBkg': '#56B4E9', 'actorBorder': '#555555', 'actorTextColor': '#000000', 'actorLineColor': '#555555', 'signalColor': '#555555', 'signalTextColor': '#555555', 'noteBkgColor': '#F0E442', 'noteBorderColor': '#555555', 'noteTextColor': '#000000', 'activationBkgColor': '#E69F00', 'activationBorderColor': '#555555'}}}%% sequenceDiagram participant U as User participant SVC as Talk2Data Service participant IA as Insights Agent participant SQL as Text2SQL Agent participant PG as PostgreSQL U->>SVC: Natural-language question SVC->>IA: Route message (A2A) IA->>SQL: Delegate SQL generation (A2A) SQL->>SQL: Analyze schema + generate SQL SQL->>PG: Execute validated query PG-->>SQL: Result set SQL-->>IA: Query results (A2A artifact) IA->>IA: Analyze results, generate insights IA-->>SVC: Insights + visualizations (SSE stream) SVC-->>U: Real-time streamed response ``` A user types a natural-language question in the chat interface, such as "What were the top 10 products by revenue last quarter?" The Talk2Data Service (REST + SSE gateway) creates a session, routes the message to the Insights Agent, and establishes an SSE stream back to the client. The Insights Agent runs an agentic loop: it reasons about the question, selects tools, and delegates SQL generation to the Text2SQL Agent. The Text2SQL Agent analyzes the database schema, generates SQL using an LLM, validates it with `sqlglot`, and executes the query against the user's data connection. Query results, analysis, and visualizations stream back to the user in real time via A2A events (`TaskStatusUpdateEvent` for progress, `TaskArtifactUpdateEvent` for results). ## Architecture Building a similar solution? Use the [Solution Developer Guide › Starter Templates](/guides/solution-dev/starter-templates) — the Tier 3 template mirrors this layered agent shape with copy-pasteable scaffolds. The solution follows a layered agent architecture: ```text theme={null} Client / React UI | Talk2Data Service (REST + SSE gateway, port 8080) | Insights Agent (agentic loop, reasoning, tool orchestration, port 8002) |--- Text2SQL Agent (NL-to-SQL generation, validation, execution, port 8001) |--- Coding Agent (LLM-generated Python code execution, port 8004) |--- MCP Plotly Server (visualization tools via MCP, port 8000) ``` Text2SQL and Coding agents communicate with the Insights Agent via A2A, while the MCP Plotly Server is accessed via MCP. The Insights Agent dynamically discovers both sub-agents and MCP tools on each request. ## A2A Protocol Integration All inter-agent communication uses the A2A protocol: Each agent publishes a JSON manifest at `/.well-known/agent-card.json` that describes its identity, skills, and capabilities. Agent Cards enable automatic discovery by the Talk2Data service and Insights Agent at request time. Key fields include: * **name** -- human-readable agent name * **skills** -- list of capabilities with input/output schemas * **capabilities** -- supported features (streaming, multi-turn, etc.) * **endpoint** -- the agent's A2A service URL A2A messages contain typed parts: | Part Type | Usage | | ---------- | --------------------------------------------------------- | | `TextPart` | Natural-language text (questions, analysis, explanations) | | `DataPart` | Structured data (datasource configs, query parameters) | | `FilePart` | Binary artifacts (charts, exported files) | Agents emit events for real-time frontend updates: | Event | Purpose | | ------------------------- | -------------------------------------------------------------- | | `TaskStatusUpdateEvent` | Progress messages ("Analyzing schema...", "Generating SQL...") | | `TaskArtifactUpdateEvent` | Final results (data tables, charts, text analysis) | Every pipeline step emits a status event before starting work, providing real-time visibility into the agent's reasoning process. The A2A `context_id` maps to the session ID for multi-turn conversation state. This enables agents to maintain context across multiple questions in the same conversation. ## Pipeline Framework Agent workflows are built on the `commons.pipeline.Pipeline` state-machine framework: * **Steps** are named functions that perform a unit of work * Each step returns a `Transition` object with a `goto` target (next step, `break`, or `error`) * The pipeline supports **cooperative cancellation** for graceful shutdown * Unexpected exceptions are wrapped in `StepError` for structured error handling The Text2SQL agent uses this framework for its generate-validate-execute flow: ```text theme={null} generate_sql -> validate_sql -> execute_query -> format_results ``` ## LLM Integration Data Insights uses **LiteLLM** for provider-agnostic LLM access: | Feature | Details | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Client** | `commons.llm.LLMClient` wrapping LiteLLM | | **Model format** | `provider/model` (e.g., `vertex_ai/gemini-3.5-flash`, `vertex_ai/claude-opus-4-8`, `openai/gpt-5.5`) | | **Observability** | Langfuse LLM tracing auto-enabled when `LANGFUSE_HOST` is set. Chat dispatch and the pipeline executor are auto-instrumented with `@observe` decorators, A2A trace context propagates across services so a single conversation produces one unified trace, trace IDs are seeded deterministically from `turn_id`, and per-iteration spans are emitted inside agent loops. See [Langfuse Setup](/guides/langfuse-setup) for configuration and [Langfuse Overview](/deployment/observability/langfuse) for the full tracing model. | | **Configuration** | Per-service LLM env vars (`TALK2DATA_TEXT2SQL_LLM_MODEL`, `TALK2DATA_INSIGHTS_LLM_MODEL`, etc.) via `pydantic_settings.BaseSettings` | Credentials are never hardcoded. All API keys and connection strings are loaded from environment variables or `.env` files via `pydantic_settings.BaseSettings`. ## Database Schema The `talk2data` database schema stores conversation state and artifacts: | Table | Purpose | | -------------------------- | -------------------------------------------------------- | | **sessions** | Chat sessions with user and project context | | **conversation\_messages** | Individual messages within a session | | **artifacts** | Generated outputs (SQL queries, results, visualizations) | | **feedback** | User feedback on agent responses | * **Primary keys**: UUID strings * **Timestamps**: `DateTime(timezone=True)` with UTC * **ORM**: SQLAlchemy 2.0+ async with `asyncpg` driver * **Migrations**: Alembic in `packages/common-db/` ## REST Endpoints The Talk2Data Service exposes the following REST and SSE endpoints: | Endpoint | Purpose | | ------------------------------ | ----------------------------------------------------------- | | `GET /talk2data/chat/sessions` | List active chat sessions | | `POST /talk2data/chat/*` | Start or continue a chat session (SSE streaming) | | `POST /talk2data/v1/sample` | Preview rows from a data connection table (no LLM involved) | The `/talk2data/v1/sample` endpoint is useful for data exploration before composing a question — it fetches a configurable number of rows from a named table via an existing data connection, using a fully qualified table name (`database.schema.table`). See [Text-to-SQL](/data-insights/text-to-sql) for request and error details. ## Platform Integration Data Insights integrates with the platform layer for: | Capability | Platform Service | | ------------------ | ---------------------------------------------------------------- | | **Authentication** | Governance (JWT validation via the platform identity provider) | | **Authorization** | Governance (permission checks via the authorization service SDK) | \| **Data connections** | Assets (configured database connections) | \| **Asset management** | Assets (data connections, artifacts, files, models) | The integration uses auto-generated Python SDKs from the platform's OpenAPI specs. Data Insights never implements its own permission checks. ## Next Steps Learn how to use the conversational interface for data analysis. Understand how natural-language questions are converted to SQL queries. See how Data Insights agents are registered and discovered. Configure data connections for your databases. # Text-to-SQL Source: https://docs.emergence.ai/data-insights/text-to-sql How the Text2SQL agent converts natural-language questions into validated, executable SQL using schema-aware generation and sqlglot validation. # Text-to-SQL Agent The Text2SQL Agent is a specialized A2A agent that converts natural-language questions into SQL queries. It uses schema-aware generation, SQL validation via `sqlglot`, and safe execution against customer databases. ## Pipeline versions Two Text2SQL pipelines are available, controlled by a single configuration switch: * **Legacy pipeline (default, in production)**: the generate → validate → execute flow described below. A single schema is fetched per question, validated, and executed. * **Multi-schema pipeline (in rollout, behind a flag)**: introduces schema selection and enrichment steps so questions spanning multiple schemas are handled without requiring the user to pre-select a schema. The architecture is decided (ADRs 004/005) and the scaffolding is in place; production rollout is in progress. Switch via `text2sql.pipeline.use_new_pipeline` in `config.yaml`. The legacy pipeline is the supported path in current deployments. The multi-schema pipeline introduces: * **Schema table selection** (ADR 004): a schema-selection step narrows the catalog to the schemas relevant to the question, using semantic search over the data catalog. * **Raw DDL storage** (ADR 005): column-level DDL is stored and retrieved for the selected schemas, giving the LLM richer type and constraint context without fetching the entire catalog. * `CatalogSearchResult` model and pipeline state fields (`pruned_schema`, `enriched_schema`, etc.) carrying multi-schema context across steps. * A new `allow_multi_schema` flag on `CatalogSchemaProvider` that bypasses the single-schema guard when the pipeline is active. The rest of this page describes the legacy pipeline, which remains the default. ## Pipeline Overview ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR Q["NL Question\n(from user)"] SR["Schema\nRetrieval"] LLM["LLM\n(SQL generation)"] VAL{"sqlglot\nValidation"} EXEC["Query\nExecution\n(read-only)"] FMT["Result\nFormatting\n(DataPart)"] A2A["A2A Event\nStream"] ERR["Error\nFeedback"] Q --> SR --> LLM --> VAL VAL -->|"valid"| EXEC --> FMT --> A2A VAL -->|"invalid — retry"| ERR --> LLM classDef entry fill:#56B4E9,stroke:#555555,color:#000 classDef llm fill:#CC79A7,stroke:#555555,color:#000 classDef val fill:#F0E442,stroke:#555555,color:#000 classDef db fill:#0072B2,stroke:#555555,color:#fff classDef out fill:#009E73,stroke:#555555,color:#000 classDef err fill:#D55E00,stroke:#555555,color:#000 class Q,SR entry class LLM llm class VAL val class EXEC db class FMT,A2A out class ERR err ``` The Text2SQL agent follows a **generate, validate, execute** pipeline built on the `commons.pipeline.Pipeline` framework: The agent fetches the target database schema (tables, columns, types, relationships, foreign keys) from the data connection. This schema context is included in the LLM prompt for accurate SQL generation. The LLM generates a SQL query based on the natural-language question and schema context. The prompt includes: * Database schema (tables, columns, types) * Table relationships and foreign keys * Previous conversation context (for follow-up questions) * Database dialect (PostgreSQL) The generated SQL is validated using `sqlglot`: * **Syntax validation**: Ensures the SQL is syntactically correct * **Dialect validation**: Confirms compatibility with the target database dialect * **Safety checks**: Prevents destructive operations (DROP, DELETE, TRUNCATE, ALTER) * If validation fails, the agent retries generation with the error feedback The validated SQL is executed against the customer database via the data connection. The agent uses a read-only connection with query timeouts to prevent runaway queries. Query results are formatted as structured data (DataPart) and returned to the calling agent via A2A events. ## Schema-Aware Generation The quality of generated SQL depends heavily on the schema context provided to the LLM. The Text2SQL agent builds a comprehensive schema representation: ```python theme={null} # Schema context includes: # - Table names and descriptions # - Column names, types, and constraints # - Primary and foreign key relationships # - Sample values for enum-like columns # - Table-level comments and documentation ``` For better SQL generation quality, add comments and descriptions to your database schema. The Text2SQL agent uses these to understand the semantic meaning of tables and columns. ## SQL Validation The agent uses `sqlglot` for multi-layer validation: | Check | Purpose | | --------------------- | ------------------------------------------------- | | **Parse** | Verify the SQL is syntactically valid | | **Transpile** | Confirm compatibility with the PostgreSQL dialect | | **Safety** | Block destructive statements (DDL, DML mutations) | | **Column validation** | Verify referenced columns exist in the schema | ### Safety Rules The Text2SQL agent enforces strict safety rules: | Allowed | Blocked | | ------------------- | ------------------------------------- | | `SELECT` queries | `INSERT`, `UPDATE`, `DELETE` | | `WITH` (CTEs) | `DROP`, `ALTER`, `TRUNCATE` | | Aggregate functions | `CREATE`, `GRANT`, `REVOKE` | | Window functions | Raw `EXECUTE`, dynamic SQL | | Subqueries | Statements that modify data or schema | ## Pipeline Framework The Text2SQL agent uses the `commons.pipeline.Pipeline` state machine for managing the generate-validate-execute flow: ```python theme={null} from commons.pipeline import Pipeline, Transition, StepError class Text2SQLPipeline(Pipeline): async def generate(self, context): sql = await self.llm.generate_sql(context.question, context.schema) return Transition(goto="validate", data={"sql": sql}) async def validate(self, context): try: validated = sqlglot.transpile(context.sql, read="postgres") return Transition(goto="execute", data={"sql": validated}) except sqlglot.errors.ParseError as e: # Retry generation with error feedback return Transition(goto="generate", data={"error": str(e)}) async def execute(self, context): results = await self.db.execute(context.sql) return Transition(goto="break", data={"results": results}) ``` Steps return `Transition` objects that specify the next step (`goto`), with `"break"` signaling pipeline completion and `"error"` signaling failure. Built-in cooperative cancellation allows long-running pipelines to be interrupted gracefully. ## Event Streaming The Text2SQL agent emits A2A events at each pipeline step for real-time progress: | Step | Event | Message | | ---------------- | ------------------------- | ------------------------------ | | Schema retrieval | `TaskStatusUpdateEvent` | "Analyzing database schema..." | | SQL generation | `TaskStatusUpdateEvent` | "Generating SQL query..." | | Validation | `TaskStatusUpdateEvent` | "Validating query..." | | Execution | `TaskStatusUpdateEvent` | "Executing query..." | | Results | `TaskArtifactUpdateEvent` | Query results as DataPart | ## Error Handling The agent uses typed error hierarchies for recoverable failures: | Error Type | Handling | | ---------------------- | ------------------------------------------------------ | | `SQLValidationError` | Retry SQL generation with validation error context | | `QueryExecutionError` | Return user-friendly error message with the failed SQL | | `SchemaRetrievalError` | Check data connection health and credentials | | `QueryTimeoutError` | Suggest query simplification or filtering | | Unexpected errors | Wrapped in `StepError` by the pipeline framework | ## Agent Card The Text2SQL agent exposes its capabilities via the standard A2A Agent Card: ```json theme={null} { "name": "text2sql", "description": "Converts natural-language questions to SQL queries", "url": "http://text2sql:8001", "skills": [ { "id": "nl-to-sql", "name": "Natural Language to SQL", "description": "Generate and execute SQL from natural-language questions", "tags": ["sql", "database", "analytics"] } ] } ``` ## Sample Data Preview The Talk2Data Service exposes a `POST /talk2data/v1/sample` endpoint that returns a preview of rows from a database table without running a full analytical query. This is used by the frontend to let users inspect table contents before composing a natural-language question. **Request fields:** | Field | Type | Description | | -------------- | ------- | ------------------------------------------------------------ | | `table_fqn` | string | Fully qualified table name: `database.schema.table` | | `resource_uri` | string | Data connection resource URI (from the Assets service) | | `limit` | integer | Number of rows to return (default varies by connection type) | **Error responses:** | Status | Cause | | ------ | ------------------------------------------------------------ | | `400` | Invalid `table_fqn` format (must be `database.schema.table`) | | `404` | Data connection not found for the given `resource_uri` | | `501` | Connection type does not support row sampling | | `503` | `EM_RUNTIME_ASSETS_URL` not configured | The endpoint authenticates via the standard platform JWT and respects project-scoped data connection access. ## MCP API changes (2026-05-13) Two breaking changes shipped in em-talk2data v3.2.0 (commit `ecdc18a`): ### generate-sql: schema field is now required `GenerateSQLRequest` changed from an optional list to a required single object: ```json Before (schemas — optional list) theme={null} { "question": "How many orders shipped last month?", "database": "analytics", "schemas": [{ "tables": [...] }] } ``` ```json After (schema — required single object; database required) theme={null} { "question": "How many orders shipped last month?", "database": "analytics", "schema": { "tables": [...] } } ``` * The JSON field name is `schema` (not `schema_` — that is the Python attribute alias) * The field is **required**; omitting it returns `422 Unprocessable Entity` with an `HTTPValidationError` body ### execute-query: 422 responses now include error detail `execute-query` previously returned a generic 422 with no body schema on SQL validation failures. It now returns a structured `HTTPValidationError` body, making it easier to surface validation failure reasons in MCP clients. ### Deployment: missing env vars in local/CI Helm values `EM_RUNTIME_ASSETS_URL` and `EM_RUNTIME_UTILS_URL` were missing from the `talk2data-service` env block in `values.local.yaml` and `values.ci.yaml`. This caused `execute_query` calls to return `503`. Update local and CI overrides to include: ```yaml theme={null} talk2data-service: env: EM_RUNTIME_ASSETS_URL: "http://em-runtime-assets:8000" # local: subchart EM_RUNTIME_UTILS_URL: "http://em-runtime-utils:8000" # local: subchart ``` In GKE (`values.yaml`), the URLs use the cross-namespace service DNS (`em-runtime-assets.em-runtime.svc.cluster.local`) — these were already correct before this fix. ## Next Steps See how Text2SQL fits into the full conversational analytics pipeline. Learn how the Insights Agent orchestrates analysis beyond SQL. Understand how query results become interactive charts. Connect a database for Text2SQL to query. # Visualizations Source: https://docs.emergence.ai/data-insights/visualizations How CRAFT Insights Agent generates interactive Plotly charts from query results to visualize data patterns and trends. # Visualizations The Data Insights solution generates interactive **Plotly** visualizations as part of the conversational analytics pipeline. When a user's question benefits from a visual answer, the Insights Agent automatically selects and generates the appropriate chart type. ## How Visualizations Are Generated The Insights Agent determines when a visualization would enhance the answer and generates it as part of the agentic loop: The Insights Agent evaluates whether the question implies a visual answer. Questions about trends, comparisons, distributions, or proportions typically trigger visualization. The Text2SQL Agent generates and executes a query to retrieve the data needed for the visualization. The LLM selects the most appropriate chart type based on the data shape, question intent, and number of dimensions: | Data Pattern | Chart Type | | ------------------- | --------------------- | | Trends over time | Line chart | | Category comparison | Bar chart | | Part-to-whole | Pie / Donut chart | | Distribution | Histogram | | Correlation | Scatter plot | | Multi-dimensional | Grouped / Stacked bar | The Insights Agent generates Python code using Plotly to create the visualization: ```python theme={null} import plotly.express as px fig = px.bar( df, x="product_category", y="revenue", title="Revenue by Product Category", color="region", barmode="group" ) fig.update_layout( xaxis_title="Category", yaxis_title="Revenue ($)" ) ``` The chart is rendered and delivered as an artifact via the A2A protocol. The frontend renders it as an interactive Plotly chart with hover tooltips, zoom, and pan capabilities. ## Supported Chart Types | Chart | When Used | Plotly Function | | --------------- | ----------------------------- | ---------------- | | **Line** | Time series, trends | `px.line()` | | **Bar** | Category comparison | `px.bar()` | | **Pie / Donut** | Proportions (\< 8 categories) | `px.pie()` | | **Scatter** | Correlations, distributions | `px.scatter()` | | **Histogram** | Value distributions | `px.histogram()` | | **Area** | Cumulative trends | `px.area()` | | Chart | When Used | Plotly Function | | --------------- | ----------------------------- | ------------------------- | | **Grouped bar** | Multi-category comparison | `px.bar(barmode="group")` | | **Stacked bar** | Part-to-whole over categories | `px.bar(barmode="stack")` | | **Heatmap** | Matrix data, correlations | `px.imshow()` | | **Box plot** | Statistical distributions | `px.box()` | | **Treemap** | Hierarchical proportions | `px.treemap()` | | **Funnel** | Sequential stages | `px.funnel()` | ## Interactive Features Plotly charts rendered in the frontend support full interactivity: * **Hover tooltips**: View exact values for each data point * **Zoom and pan**: Focus on specific regions of the chart * **Legend toggle**: Show/hide individual series by clicking the legend * **Download**: Export as PNG or SVG directly from the chart toolbar * **Responsive sizing**: Charts adapt to the container width ## Chart Generation Examples **Question:** "Show me the monthly revenue trend for the past year" The Insights Agent generates a line chart with months on the x-axis and revenue on the y-axis, including trend annotations for significant changes. **Question:** "Compare sales across product categories by region" The agent generates a grouped bar chart with categories on the x-axis, sales on the y-axis, and color-coded bars for each region. **Question:** "What does the distribution of order values look like?" The agent generates a histogram showing the frequency distribution of order values, with optional statistics overlay (mean, median, standard deviation). **Question:** "What percentage of revenue comes from each channel?" The agent generates a pie or donut chart showing revenue proportions by channel, with percentage labels. ## Artifact Storage Generated visualizations are stored as session artifacts in the Talk2Data database: | Artifact Component | Storage | | ----------------------- | --------------------------------------------- | | **Chart specification** | Plotly JSON stored in the artifacts table | | **Underlying data** | Query results stored alongside the chart | | **Python code** | Generation code preserved for reproducibility | | **Metadata** | Chart type, title, axes, timestamp | Artifacts persist for the duration of the session and can be referenced in follow-up questions ("Make that a line chart instead" or "Add a trend line"). ## Customization in Follow-Up Questions Users can refine visualizations through natural-language follow-ups: | Request | Result | | ------------------------- | ------------------------------- | | "Make that a bar chart" | Chart type changed | | "Show only the top 5" | Data filtered, chart updated | | "Add a trend line" | Trend line overlay added | | "Use a logarithmic scale" | Y-axis scale changed | | "Split by region" | Color dimension added | | "Show as a percentage" | Values converted to proportions | ## Next Steps See the full conversational pipeline that produces visualizations. Understand how data is retrieved before visualization. Learn about the reasoning engine that selects chart types. Connect a database to start visualizing your data. # Helm Configuration Source: https://docs.emergence.ai/deployment/helm/configuration Configure the em-runtime Helm chart for development, staging, and production environments with external databases, S3 storage, and HTTPS. # Helm Configuration The em-runtime Helm chart packages all platform services and their dependencies into a single deployable unit. This guide covers the chart architecture, deployment modes, secret management, and production configuration. ## Chart Overview The chart deploys and manages the following components: | Component | Description | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | **em-runtime-governance** | Organizations, projects, and permissions | | **em-runtime-assets** | Artifacts, data connections, files, and models | | **em-runtime-utils** | Data catalog, scheduling, context packs, and memories | | **Keycloak** | Identity and access management (multi-tenant realms) | | **OpenFGA** | Fine-grained authorization (Zanzibar model) | | **Infisical** | Secrets management (optional; can be replaced by ESO + GCP Secret Manager, see [Secrets Management](/security/secrets-management)) | | **PostgreSQL** | Persistent data storage (optional, in-cluster) | | **Redis** | Caching and session storage (optional, in-cluster) | ## Deployment Modes All dependencies (PostgreSQL, Redis) are deployed as part of the Helm release within the cluster. ```bash theme={null} helm install em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \ --version $VERSION \ --namespace=$NAMESPACE ``` Best for development, testing, and quick setup. PostgreSQL and Redis run outside the cluster (managed services or self-hosted). The chart connects via configuration. ```bash theme={null} helm install em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \ --version $VERSION \ --namespace=$NAMESPACE \ --set postgres.enabled=false \ --set redis.enabled=false \ --set existingSecret=em-runtime-external-credentials ``` Best for production with managed database services. ## Secret Management All platform credentials are consolidated into a single chart-managed Secret named `em-runtime-secrets`. In external mode, you provide base connection details, and the chart auto-computes derived values. ### External Credentials Secret Create this before installing the chart when using external databases: ```bash theme={null} kubectl create secret generic em-runtime-external-credentials \ --namespace=$NAMESPACE \ --from-literal=postgres-password='' \ --from-literal=postgres-host='' \ --from-literal=postgres-port='5432' \ --from-literal=postgres-username='postgres' \ --from-literal=redis-password='' \ --from-literal=redis-host='' \ --from-literal=redis-port='6379' \ --from-literal=redis-scheme='redis' \ --from-literal=redis-username='default' ``` Set `redis-scheme` to `rediss` (double `s`) for TLS connections (e.g., GCP Memorystore with in-transit encryption). ### S3 Storage Credentials ```bash theme={null} kubectl create secret generic em-runtime-s3 \ --namespace=$NAMESPACE \ --from-literal=access-key-id='' \ --from-literal=secret-access-key='' ``` Skip this step if using IRSA (AWS) or GCP Workload Identity. ## External Object Storage For production, use S3-compatible object storage instead of the default PVC: ```yaml theme={null} em-runtime-assets: storage: enabled: false # Disable PVC extraEnvVars: - name: STORAGE_TYPE value: "s3" - name: S3_REGION value: "us-east-1" - name: S3_BUCKET_NAME value: "my-bucket" - name: S3_PREFIX_UPLOADED value: "uploaded" - name: S3_PREFIX_GENERATED value: "generated" - name: S3_ACCESS_KEY_ID valueFrom: secretKeyRef: { name: em-runtime-s3, key: access-key-id } - name: S3_SECRET_ACCESS_KEY valueFrom: secretKeyRef: { name: em-runtime-s3, key: secret-access-key } ``` ### Provider-Specific Notes | Provider | Configuration | | ------------------------ | ------------------------------------------------------------------------------------------------ | | **AWS S3** | No `S3_ENDPOINT_URL` needed. Set `S3_REGION` to your bucket's region. | | **Google Cloud Storage** | Use HMAC keys. Set `S3_ENDPOINT_URL` to `https://storage.googleapis.com`, `S3_REGION` to `auto`. | | **MinIO** | Set `S3_ENDPOINT_URL` to your MinIO endpoint (e.g., `http://minio:9000`). | ## Enabling HTTPS EM-Runtime uses Gateway API for routing. TLS termination is configured on the Gateway resource using cert-manager. Install cert-manager with Gateway API support enabled (`config.enableGatewayAPI=true`). Create a ClusterIssuer for your ACME provider (e.g., Let's Encrypt). Add `cert-manager.io/cluster-issuer` annotation and configure an HTTPS listener on your Gateway resource. ## Service URL Environment Variables Service-to-service communication uses canonical URL environment variables. These names were standardized as part of PE-200; legacy aliases are retained as temporary hotfixes for older application versions. | Canonical name | Replaces (legacy) | Purpose | | ----------------------- | ------------------------------------------- | ----------------------- | | `KEYCLOAK_URL` | `AUTH_URL`, `EM_RUNTIME_KEYCLOAK_URL` | Keycloak base URL | | `EM_RUNTIME_ASSETS_URL` | `RUNTIME_URL` (when used for asset routing) | Assets service base URL | | `EM_RUNTIME_UTILS_URL` | `RUNTIME_URL` (when used for utils routing) | Utils service base URL | Application versions still on the legacy names continue to work because both are set during the rollout. Once all consumer applications upgrade past the migration point, the legacy aliases will be removed. ### UI compliance and external link variables The Runtime UI surfaces compliance links (Terms of Service, Privacy Policy, DPA) and an embedded analytics script. Configure them via env vars on the `em-runtime-ui` deployment: | Variable | Purpose | | -------------------------- | ------------------------------------------------------------------- | | `YIELD_INSIGHTS_URL` | URL of the Yield Insights solution (semiconductor deployments only) | | `TERMS_OF_SERVICE_URL` | Terms of Service link surfaced in the UI footer | | `PRIVACY_POLICY_URL` | Privacy Policy link surfaced in the UI footer | | `EMERGENCE_DPA_URL` | Data Processing Agreement link | | `EMERGENCE_DPA_UPDATE_URL` | DPA update notification link | | `TERMLY_SCRIPT_SRC` | Termly script source URL for compliance banner | These are configured via the `em-runtime-ui.env` block in `values.yaml`, the same way as service URLs above. ## Environment Variable Override System Each runtime service has three layers for environment variables: | Layer | Format | Purpose | | -------------- | ----------------------- | ------------------------------------------------------------------------- | | `env` | `map[string]string` | Simple key-value pairs. **Use this to override defaults.** | | `envVars` | List of K8s env objects | Variables using `valueFrom` (secrets, config maps). Defined by the chart. | | `extraEnvVars` | List of K8s env objects | Additional variables appended after `envVars`. | ### Override Precedence ```text theme={null} env (wins) --> envVars (skipped if name exists in env) --> extraEnvVars (skipped if name exists in env) ``` **Example -- override defaults across services:** ```yaml theme={null} em-runtime-governance: env: DOCS_ENABLED: "false" # Disable API docs LOG_LEVEL: "WARNING" # Reduce log verbosity CORS_ENABLED: "false" # Disable CORS em-runtime-assets: env: DOCS_ENABLED: "false" LOG_LEVEL: "WARNING" em-runtime-utils: env: DOCS_ENABLED: "false" LOG_LEVEL: "WARNING" ``` ## Bootstrap Configuration On first startup, the platform bootstraps a default organization: ```yaml theme={null} bootstrap: organizationId: "acme" # Also used as Keycloak realm name organizationName: "ACME Corp" organizationDescription: "ACME Corporation platform" adminEmail: "admin@acme.com" ``` The bootstrap is **idempotent** -- it checks for existing resources before creating. The `organizationId` must contain only alphanumeric characters, hyphens, or underscores. ## Production Values File Below is a sample `production-values.yaml`: ```yaml theme={null} # External hostname and scheme hostname: "api.example.com" scheme: "https" # Bootstrap organization bootstrap: organizationId: "my-org" organizationName: "My Organization" adminEmail: "admin@example.com" # External databases existingSecret: "em-runtime-external-credentials" postgres: enabled: false redis: enabled: false # Keycloak keycloak: replicaCount: 2 resources: requests: { memory: "1536Mi", cpu: "1000m" } limits: { memory: "2Gi", cpu: "2000m" } # OpenFGA with HPA openfga: replicaCount: 2 autoscaling: enabled: true minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 75 # Governance with HPA em-runtime-governance: replicaCount: 2 autoscaling: enabled: true minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 75 env: LOG_LEVEL: "WARNING" DOCS_ENABLED: "false" # Assets with HPA and S3 em-runtime-assets: replicaCount: 2 autoscaling: enabled: true minReplicas: 2 maxReplicas: 10 storage: enabled: false env: LOG_LEVEL: "WARNING" DOCS_ENABLED: "false" extraEnvVars: - name: STORAGE_TYPE value: "s3" - name: S3_REGION value: "us-east-1" - name: S3_BUCKET_NAME value: "my-org-em-runtime" # Utils with HPA em-runtime-utils: replicaCount: 2 autoscaling: enabled: true minReplicas: 2 maxReplicas: 10 env: LOG_LEVEL: "WARNING" DOCS_ENABLED: "false" ``` ### Install with Production Values ```bash theme={null} helm install em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \ --version $VERSION \ --namespace production \ -f production-values.yaml ``` ## Testing The chart includes a Helm test hook that runs end-to-end tests: ```bash theme={null} helm test $RELEASE --namespace $NAMESPACE --timeout 10m ``` The test job creates a temporary organization, validates role-based access across all permission groups, and cleans up. ## Backups Back up these components together for a consistent restore: | Component | Contains | Backup Method | | ------------------------------ | ------------------------------------------- | ------------------------------------------------------- | | **PostgreSQL** | All service data | `pg_dumpall` or cloud snapshots | | **Redis** | Session and cache data | `redis-cli BGSAVE` or cloud snapshots | | **em-runtime-secrets** | All platform credentials | `kubectl get secret em-runtime-secrets -o yaml` | | **infisical-bootstrap-secret** | Machine identity token (if using Infisical) | `kubectl get secret infisical-bootstrap-secret -o yaml` | | **Assets storage** | Uploaded/generated artifacts | VolumeSnapshots or S3 bucket backup | If using Infisical, the `em-runtime-secrets` secret contains the `ENCRYPTION_KEY` and `AUTH_SECRET`. Without these, encrypted data stored by Infisical cannot be decrypted. If using ESO + GCP Secret Manager, secrets are sourced from GCP SM directly, back up the GCP Secret Manager secrets instead. ## Next Steps Complete reference for all Helm chart values. Version upgrades, migration steps, and rollback procedures. Configure telemetry for all runtime services. # em-core Chart Source: https://docs.emergence.ai/deployment/helm/em-core Helm chart for deploying the CRAFT core platform — em-platform, em-user-service, em-user-portal, Keycloak, PostgreSQL and Redis. # em-core Chart `em-core` deploys the CRAFT core platform to a Kubernetes cluster. It includes all services required to run the platform's core capabilities. ## Components | Component | Image | Purpose | | ----------------- | ------------------------------------- | ---------------------------------------- | | `em-platform` | `ghcr.io/emergenceai/em-platform` | Main backend with workflow orchestration | | `em-user-service` | `ghcr.io/emergenceai/em-user-service` | User management and authentication | | `em-user-portal` | `ghcr.io/emergenceai/em-user-portal` | Frontend web application | | `keycloak` | upstream Keycloak | Identity and access management | | `postgresql` | `postgres:alpine` | Persistent data storage | | `redis` | `redis:alpine` | In-memory caching and session storage | | `nginx` | upstream Nginx | Ingress and load balancing | ## Quick start ```bash theme={null} export HOSTNAME=platform.example.com export KEYCLOAK_ADMIN_PASSWORD= export POSTGRES_PASSWORD= ``` ```bash theme={null} kubectl create namespace em-core ``` ```bash theme={null} kubectl create secret generic em-core-credentials \ --namespace em-core \ --from-literal=KEYCLOAK_ADMIN_PASSWORD=$KEYCLOAK_ADMIN_PASSWORD \ --from-literal=POSTGRES_PASSWORD=$POSTGRES_PASSWORD ``` ```bash theme={null} helm install em-core oci://ghcr.io/emergenceai/em-charts/em-core \ --version 0.0.8 \ --namespace em-core \ --set hostname=$HOSTNAME \ --set postgresql.existingSecret=em-core-credentials \ --set redis.existingSecret=em-core-credentials ``` ## Using external PostgreSQL and Redis For production deployments, disable the bundled databases and connect to managed instances: ```yaml theme={null} # values.yaml postgresql: enabled: false redis: enabled: false postgresql: existingSecret: my-external-db-credentials # Secret must contain: POSTGRES_HOST, POSTGRES_PORT, POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD redis: existingSecret: my-external-db-credentials # Secret must contain: REDIS_HOST, REDIS_PORT ``` ## Enabling HTTPS Install [cert-manager](https://cert-manager.io) and configure a ClusterIssuer, then: ```yaml theme={null} hostname: platform.example.com schema: https ingress: tls: enabled: true issuer: letsencrypt-prod ``` ## Key values | Value | Default | Description | | ---------------------------------- | ------------------------------------- | --------------------------------------------------- | | `hostname` | `localhost` | External hostname for ingress and service discovery | | `schema` | `http` | `http` or `https` | | `em-platform.image.repository` | `ghcr.io/emergenceai/em-platform` | em-platform image | | `em-user-service.image.repository` | `ghcr.io/emergenceai/em-user-service` | em-user-service image | | `em-user-portal.image.repository` | `ghcr.io/emergenceai/em-user-portal` | em-user-portal image | | `postgresql.enabled` | `true` | Deploy bundled PostgreSQL | | `redis.enabled` | `true` | Deploy bundled Redis | The full values reference is in `charts/em-core/values.yaml` in the em-charts repository. ## Related Extends em-core with data analysis and insights capabilities. Base chart used by em-core for each component. Chart version upgrade procedures. # em-data-insights Chart Source: https://docs.emergence.ai/deployment/helm/em-data-insights Helm chart for deploying the CRAFT Data Insights solution — extends em-core with data analysis agents and Redshift integration. # em-data-insights Chart `em-data-insights` deploys the CRAFT Data Insights solution — a superset of `em-core` that adds the data analysis agent and Redshift connector. ## Components `em-data-insights` includes all [em-core](./em-core) components plus: | Component | Purpose | | ----------------------------- | ----------------------------------------------- | | `em-data-insights-agent` | Data analysis and insights generation A2A agent | | `em-redshift-query-connector` | AWS Redshift query connector service | ## Quick start ```bash theme={null} export HOSTNAME=platform.example.com export KEYCLOAK_ADMIN_PASSWORD= export POSTGRES_PASSWORD= export REDSHIFT_HOST= export REDSHIFT_USER= export REDSHIFT_PASSWORD= ``` ```bash theme={null} kubectl create namespace em-data-insights ``` ```bash theme={null} # Platform credentials (Keycloak + PostgreSQL) kubectl create secret generic em-data-insights-credentials \ --namespace em-data-insights \ --from-literal=KEYCLOAK_ADMIN_PASSWORD=$KEYCLOAK_ADMIN_PASSWORD \ --from-literal=POSTGRES_PASSWORD=$POSTGRES_PASSWORD # Redshift credentials — must be named 'redshift' with these exact keys kubectl create secret generic redshift \ --namespace em-data-insights \ --from-literal=redshift-host=$REDSHIFT_HOST \ --from-literal=redshift-port=${REDSHIFT_PORT:-5439} \ --from-literal=redshift-database=$REDSHIFT_DATABASE \ --from-literal=redshift-username=$REDSHIFT_USER \ --from-literal=redshift-password=$REDSHIFT_PASSWORD ``` ```bash theme={null} helm install em-data-insights oci://ghcr.io/emergenceai/em-charts/em-data-insights \ --version 0.0.8 \ --namespace em-data-insights \ --set em-core.hostname=$HOSTNAME \ --set em-core.schema=http ``` ## Redshift integration Configure the Redshift connector in your values: ```yaml theme={null} em-redshift-query-connector: env: REDSHIFT_DATABASE: my_database REDSHIFT_PORT: "5439" envVars: - name: REDSHIFT_HOST valueFrom: secretKeyRef: name: redshift key: redshift-host - name: REDSHIFT_USERNAME valueFrom: secretKeyRef: name: redshift key: redshift-username - name: REDSHIFT_PASSWORD valueFrom: secretKeyRef: name: redshift key: redshift-password ``` ## Using external databases Disable bundled PostgreSQL and Redis for production: ```yaml theme={null} postgresql: enabled: false redis: enabled: false postgresql: existingSecret: my-external-db-credentials redis: existingSecret: my-external-db-credentials ``` ## Key values | Value | Default | Description | | ---------------------------------------------- | ------------------------------------------------- | ---------------------------------------------- | | `em-core.hostname` | `localhost` | External hostname (passed to em-core subchart) | | `em-core.schema` | `http` | `http` or `https` | | `em-data-insights-agent.image.repository` | `ghcr.io/emergenceai/em-data-insights-agent` | Agent image | | `em-redshift-query-connector.image.repository` | `ghcr.io/emergenceai/em-redshift-query-connector` | Redshift connector image | | `postgresql.enabled` | `true` | Deploy bundled PostgreSQL | | `redis.enabled` | `true` | Deploy bundled Redis | The full values reference is in `charts/em-data-insights/values.yaml` in the em-charts repository. ## Related Core platform chart that em-data-insights extends. Data Insights solution capabilities. How the Text2SQL agent converts natural-language questions to SQL. # em-service Chart Source: https://docs.emergence.ai/deployment/helm/em-service Base Helm chart for deploying any CRAFT service on-premise — deployment, environment variables, secrets, ingress, HPA and probes. # em-service Chart `em-service` is the base Helm chart for deploying any containerised CRAFT service to a Kubernetes cluster. Every CRAFT solution chart wraps it as a subchart — one instance per component, differentiated by alias. ## Features * Deployment with configurable replicas, update strategy and pod annotations * Three-layer environment variable model with deduplication (see below) * Secret injection via `envFrom` or individual `envVars` entries with `valueFrom` * Ingress with flexible routing (standard Kubernetes Ingress and Gateway API) * Horizontal Pod Autoscaler * Liveness, readiness and startup probes * PersistentVolumeClaim support * Init containers and sidecars * RuntimeClass for specialised runtimes ## Installation ```bash theme={null} helm install my-service oci://ghcr.io/emergenceai/em-charts/em-service \ --version 0.0.15 \ --namespace my-namespace \ --set image.repository=ghcr.io/emergenceai/my-service \ --set image.tag=v1.0.0 ``` **Version pin: `0.0.15`, not `0.0.16`.** Only `0.0.15` is currently published to the OCI registry (`helm pull --version 0.0.16` returns "manifest unknown"). Bump this page when a newer tag is published. The registry is **private** — run `helm registry login ghcr.io` before pulling; access is granted to the Emergence team and Solution/Agent design partners. ## Environment variable model Three layers, applied in order, with deduplication: | Layer | Format | Purpose | | -------------- | -------------------------- | -------------------------------------------------------------------------------------- | | `env` | `map[string]string` | Plain-text variables. **Always wins** over `envVars`/`extraEnvVars` with the same name | | `envVars` | Kubernetes env object list | Full env spec — use `valueFrom` for secrets, ConfigMaps, field refs | | `extraEnvVars` | Kubernetes env object list | Additional variables appended after `envVars` | **Deduplication rule:** if a key in `env` matches the `name` of an `envVars` or `extraEnvVars` entry, the list entry is suppressed. This allows parent charts to define secret-backed defaults while letting operators override with plain values without creating duplicate env entries. ```yaml theme={null} # Typical pattern: secret default, plain override possible env: LOG_LEVEL: "INFO" # plain override envVars: - name: DATABASE_URL valueFrom: secretKeyRef: name: my-service-secrets key: database-url - name: LOG_LEVEL # suppressed — 'env' wins valueFrom: secretKeyRef: name: my-service-secrets key: log-level ``` ## Key values | Value | Default | Description | | --------------------- | -------------- | ------------------------------------------------- | | `replicaCount` | `1` | Number of replicas | | `image.repository` | `""` | Container image repository | | `image.tag` | `""` | Image tag | | `image.pullPolicy` | `IfNotPresent` | Pull policy | | `imagePullSecrets` | `[]` | Image pull secret names | | `env` | `{}` | Plain-text environment variables | | `envVars` | `[]` | Kubernetes env object list (supports `valueFrom`) | | `service.type` | `ClusterIP` | Kubernetes service type | | `service.port` | `8000` | Service port | | `ingress.enabled` | `false` | Enable Kubernetes Ingress | | `resources` | `{}` | CPU/memory requests and limits | | `autoscaling.enabled` | `false` | Enable HPA | The full values reference is in `charts/em-service/values.yaml` in the em-charts repository. ## Using as a subchart For the how-to flavour (multi-component packaging, env overlays, build-and-publish, GitOps), see [Solution Developer Guide › Package and Deploy](/guides/solution-dev/package-and-deploy). This page is the values reference; that page is the workflow. Service charts declare `em-service` as a subchart dependency in `Chart.yaml`: ```yaml theme={null} # charts/my-service/Chart.yaml apiVersion: v2 name: my-service version: 1.0.0 dependencies: - name: em-service alias: api version: 0.0.15 repository: oci://ghcr.io/emergenceai/em-charts - name: em-service # second component alias: worker version: 0.0.15 repository: oci://ghcr.io/emergenceai/em-charts ``` Configure each instance under its alias in `values.yaml`: ```yaml theme={null} api: image: repository: ghcr.io/emergenceai/my-service-api tag: v1.0.0 env: PORT: "8080" worker: image: repository: ghcr.io/emergenceai/my-service-worker tag: v1.0.0 ``` ## Related Core platform chart using em-service for platform components. Data insights solution chart extending em-core. em-runtime chart configuration and deployment modes. # Upgrades Source: https://docs.emergence.ai/deployment/helm/upgrades Version upgrade procedures, Helm chart promotion workflow, migration steps, and rollback strategies for the em-runtime platform. # Upgrades This guide covers the em-runtime Helm chart upgrade process, version promotion through environments, migration procedures, and rollback strategies. ## Upgrade Procedures ### Standard Helm Upgrade For direct Helm deployments (not using ArgoCD): ```bash theme={null} # Check current version helm list -n $NAMESPACE # Review what will change helm diff upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \ --version $NEW_VERSION \ --namespace $NAMESPACE \ -f values.yaml # Perform the upgrade helm upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \ --version $NEW_VERSION \ --namespace $NAMESPACE \ -f values.yaml ``` ## Pre-Upgrade Checklist Always back up PostgreSQL, Redis, and the `em-runtime-secrets` Secret before upgrading. Before upgrading: 1. **Review the changelog** for breaking changes and required migrations 2. **Back up databases:** ```bash theme={null} # PostgreSQL pg_dumpall -h $PG_HOST -U postgres > backup-pre-upgrade.sql # Kubernetes Secrets kubectl get secret em-runtime-secrets -n $NAMESPACE -o yaml > em-runtime-secrets-backup.yaml kubectl get secret infisical-bootstrap-secret -n $NAMESPACE -o yaml > infisical-bootstrap-backup.yaml ``` 3. **Verify current health:** ```bash theme={null} kubectl get pods -n $NAMESPACE # The gateway-routed root (/api//) returns service metadata with 200 # when the service is healthy. The Kubernetes liveness/readiness probe target # /health lives at the service root, not under the per-service prefix. curl https://$HOSTNAME/api/governance/ ``` 4. **Test in non-production first** -- promote through dev and staging before production ## Database Migrations EM-Runtime services run Alembic database migrations automatically on startup. Each service manages its own database migrations independently. ### Migration Behavior | Aspect | Behavior | | ------------- | ---------------------------------------- | | **Trigger** | Automatic on pod startup | | **Direction** | Forward only (upgrade) | | **Scope** | Per-service, per-database | | **Safety** | Idempotent -- safe to run multiple times | ### Migration Order Migrations are applied in service startup order: ```text theme={null} PostgreSQL ready -> Keycloak (schema) -> OpenFGA (schema) -> Infisical (schema) -> Governance (alembic) -> Assets (alembic) -> Utils (alembic) ``` If a migration fails, the pod will restart and retry. Check pod logs for migration errors before investigating further. ## Rollback ### Helm Rollback ```bash theme={null} # List revision history helm history em-runtime -n $NAMESPACE # Rollback to previous revision helm rollback em-runtime $PREVIOUS_REVISION -n $NAMESPACE ``` ### Database Rollback Considerations Alembic migrations are forward-only in production. If a database migration must be reversed, you must restore from backup. * **Schema-only changes** (add column, add table): Generally safe to keep during rollback; old code ignores new columns * **Data migrations**: Require database restore from pre-upgrade backup * **Destructive migrations** (drop column, rename): Require database restore ### Direct Helm Always specify `--version` in production: ```bash theme={null} helm upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \ --version 4.8.6 \ --namespace production \ -f production-values.yaml ``` ## Upgrade Troubleshooting Check pod logs for migration errors: ```bash theme={null} kubectl logs -n $NAMESPACE deploy/em-runtime-governance --previous ``` Common causes: * Database migration conflict (two migrations with same parent) * Missing database or extension * Incompatible secret format Init containers wait for dependencies (PostgreSQL, Redis, Keycloak). Check that all infrastructure services are healthy: ```bash theme={null} kubectl get pods -n $NAMESPACE -l app.kubernetes.io/component=database kubectl logs -n $NAMESPACE -l app=keycloak ``` Verify that `hostname` and `scheme` values match your Gateway and DNS configuration: ```bash theme={null} kubectl get secret em-runtime-secrets -n $NAMESPACE \ -o jsonpath='{.data.base-url}' | base64 -d ``` The CMP plugin may need to rebuild the chart dependency. Force a sync: ```bash theme={null} kubectl get application em-runtime -n argocd -o jsonpath='{.status.sync.status}' # If OutOfSync, trigger a hard refresh argocd app get em-runtime --hard-refresh argocd app sync em-runtime ``` ## Next Steps Complete reference for all Helm chart values. # Values Reference Source: https://docs.emergence.ai/deployment/helm/values-reference Complete reference for all configurable Helm chart values in the em-runtime chart, including global settings, per-service configuration, and subchart overrides. # Values Reference This page documents all configurable values in the em-runtime Helm chart. Values are organized by scope: global, per-service, infrastructure subcharts, and testing. ## Global Values | Key | Type | Default | Description | | ------------------ | ------ | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `hostname` | string | `"localhost"` | External hostname for the platform. Used in HTTPRoute hostnames, CORS origins, and Keycloak redirect URIs. | | `scheme` | string | `"http"` | URL scheme (`http` or `https`). Combined with `hostname` to form `BASE_URL`. | | `existingSecret` | string | `""` | Name of a pre-existing Secret containing external database credentials. Required when `postgres.enabled` or `redis.enabled` is `false`. | | `gatewayName` | string | `"em-runtime-gateway"` | Name of the Kubernetes Gateway resource that HTTPRoutes reference. | | `gatewayNamespace` | string | `"gateway"` | Namespace where the Gateway resource is deployed. | | `pullPolicy` | string | `"IfNotPresent"` | Default image pull policy for all containers. | | `imagePullSecrets` | list | `[{name: "em-registry-secret"}]` | Default image pull secrets for private registry access. | ## Bootstrap | Key | Type | Default | Description | | ----------------------------------- | ------ | -------------------------- | --------------------------------------------------------------------------------------------------------------------- | | `bootstrap.organizationId` | string | `"emergence"` | Default organization identifier (also used as Keycloak realm name). Must be alphanumeric with hyphens or underscores. | | `bootstrap.organizationName` | string | `"Emergence"` | Display name of the default organization. | | `bootstrap.organizationDescription` | string | `"Emergence organization"` | Description of the default organization. | | `bootstrap.adminEmail` | string | `"admin@emergence.local"` | Admin email for platform services bootstrap. | ## CORS | Key | Type | Default | Description | | -------------- | ---- | ------- | ------------------------------------------------ | | `cors.enabled` | bool | `true` | Enables CORS middleware on all runtime services. | ## HTTPRoute | Key | Type | Default | Description | | ------------------- | ---- | --------- | ------------------------------------------------------------------ | | `httpRoute.enabled` | bool | `true` | Creates an HTTPRoute resource. | | `httpRoute.rules` | list | See below | Routing rules mapping external path prefixes to internal services. | ### Default HTTPRoute Rules | Path Prefix | Service | Port | Rewrite | | ----------------- | --------------------- | ---- | ------------- | | `/keycloak` | keycloak | 8080 | None | | `/api/governance` | em-runtime-governance | 8000 | `/governance` | | `/api/assets` | em-runtime-assets | 8000 | `/assets` | | `/api/utils` | em-runtime-utils | 8000 | `/utils` | ## Service Accounts (Bootstrap) | Key | Type | Default | Description | | -------------------------- | ---- | ---------- | ------------------------------------------------------------ | | `serviceAccounts.enabled` | bool | `true` | Enables service account creation during bootstrap. | | `serviceAccounts.services` | list | 5 services | List of service accounts to create in Keycloak master realm. | Default service accounts: `svc-data-readiness`, `svc-readiness`, `svc-scheduler`, `svc-cleanup`, `svc-indexer`. ## Runtime Service Values The following values apply to each of `em-runtime-governance`, `em-runtime-assets`, and `em-runtime-utils`. Replace `` with the service name. ### Deployment | Key | Type | Default | Description | | ------------------------------ | ------ | ----------------------------------------- | ----------------------------------------------------- | | `.replicaCount` | int | `1` | Number of replicas. | | `.image.repository` | string | `"ghcr.io/emergenceai/em-runtime-"` | Container image repository. | | `.image.tag` | string | Chart version | Container image tag. | | `.image.pullPolicy` | string | `"IfNotPresent"` | Image pull policy. | | `.deploymentStrategy` | object | `{type: "Recreate"}` | Update strategy (Assets only, for PVC compatibility). | ### Autoscaling | Key | Type | Default | Description | | ------------------------------------------------------ | ---- | ------- | ----------------------------------- | | `.autoscaling.enabled` | bool | `false` | Enable Horizontal Pod Autoscaler. | | `.autoscaling.minReplicas` | int | `1` | Minimum replicas. | | `.autoscaling.maxReplicas` | int | `10` | Maximum replicas. | | `.autoscaling.targetCPUUtilizationPercentage` | int | `80` | CPU utilization target for scaling. | ### Resources | Key | Type | Default (Governance/Assets) | Default (Utils) | | ------------------------------------- | ------ | --------------------------- | --------------- | | `.resources.requests.cpu` | string | `"250m"` | `"100m"` | | `.resources.requests.memory` | string | `"512Mi"` | `"256Mi"` | | `.resources.limits.cpu` | string | `"1000m"` | `"500m"` | | `.resources.limits.memory` | string | `"1Gi"` | `"512Mi"` | ### Security Context | Key | Type | Default | Description | | ---------------------------------------------------- | ---- | --------- | ----------------------------- | | `.podSecurityContext.runAsNonRoot` | bool | `true` | Enforce non-root execution. | | `.podSecurityContext.runAsUser` | int | `1000` | User ID for the container. | | `.podSecurityContext.fsGroup` | int | `1000` | Filesystem group. | | `.securityContext.readOnlyRootFilesystem` | bool | `true` | Read-only root filesystem. | | `.securityContext.allowPrivilegeEscalation` | bool | `false` | Prevent privilege escalation. | | `.securityContext.capabilities.drop` | list | `["ALL"]` | Drop all Linux capabilities. | ### Health Probes | Key | Type | Default | Description | | ----------------------------------------- | ------ | ----------- | -------------------------------------- | | `.livenessProbe.httpGet.path` | string | `"/health"` | Liveness check path. | | `.livenessProbe.httpGet.port` | int | `8000` | Liveness check port. | | `.readinessProbe.httpGet.path` | string | `"/health"` | Readiness check path. | | `.startupProbe.failureThreshold` | int | `30` | Startup probe attempts before failure. | | `.startupProbe.periodSeconds` | int | `10` | Seconds between startup probe checks. | ### Scheduling | Key | Type | Default | Description | | ------------------------ | ------ | ------- | ----------------------------------- | | `.nodeSelector` | object | `{}` | Node label constraints. | | `.tolerations` | list | `[]` | Toleration rules for tainted nodes. | | `.affinity` | object | `{}` | Pod scheduling affinity rules. | ### Service | Key | Type | Default | Description | | ------------------------------- | ------ | ------------- | -------------------- | | `.service.type` | string | `"ClusterIP"` | Service type. | | `.service.port` | int | `8000` | Service port. | | `.service.annotations` | object | `{}` | Service annotations. | ## Default Environment Variables ### All Services | Key | Default | Description | | ----------------------------- | ------------------------------ | ---------------------------------------------------- | | `ENVIRONMENT` | `"production"` | Runtime environment identifier. | | `LOG_LEVEL` | `"INFO"` | Log verbosity (`DEBUG`, `INFO`, `WARNING`, `ERROR`). | | `LOG_FORMAT` | `"JSON"` | Log output format. | | `ROOT_PATH` | `"/api"` | API root path prefix. | | `DOCS_ENABLED` | `"true"` | Enable OpenAPI documentation endpoints. | | `OTEL_ENABLED` | `"true"` | Master switch for OpenTelemetry. | | `OTEL_EXPORTER_OTLP_ENDPOINT` | `"http://otel-collector:4317"` | OTLP collector gRPC endpoint. | | `OTEL_TRACES_ENABLED` | `"true"` | Enable distributed tracing. | | `OTEL_METRICS_ENABLED` | `"true"` | Enable metrics export. | | `OTEL_LOGS_ENABLED` | `"true"` | Enable log record export. | | `OTEL_TRACE_SAMPLE_RATE` | `"0.1"` | Trace sampling ratio (0.0-1.0). | ### Assets and Utils Only | Key | Default | Description | | -------------------- | ------------------------------------- | ---------------------------------- | | `GOVERNANCE_API_URL` | `"http://em-runtime-governance:8000"` | Internal governance service URL. | | `AUTH_CHECK_URL` | `"http://keycloak:8080/keycloak"` | Keycloak URL for token validation. | ### Secret-Backed Variables (via `envVars`) These are sourced from `em-runtime-secrets` and can be overridden by setting the same key in `env`: | Key | Services | Description | | ------------------- | -------- | ----------------------------------------------- | | `BASE_URL` | All | Platform base URL (from `hostname` + `scheme`). | | `CORS_ENABLED` | All | CORS middleware toggle. | | `POSTGRES_HOST` | All | PostgreSQL hostname. | | `POSTGRES_PORT` | All | PostgreSQL port. | | `POSTGRES_USER` | All | PostgreSQL username. | | `POSTGRES_PASSWORD` | All | PostgreSQL password. | | `POSTGRES_DB` | All | Per-service database name. | | `REDIS_HOST` | All | Redis hostname. | | `REDIS_PORT` | All | Redis port. | | `REDIS_SCHEME` | All | Redis connection scheme. | | `REDIS_PASSWORD` | All | Redis password. | ## S3 Storage Variables | Variable | Default | Description | | ---------------------- | ------------ | --------------------------------------------- | | `STORAGE_TYPE` | `local` | Storage backend: `local` or `s3`. | | `S3_ENDPOINT_URL` | None | Custom S3 endpoint (required for GCS, MinIO). | | `S3_ACCESS_KEY_ID` | None | Access key ID. | | `S3_SECRET_ACCESS_KEY` | None | Secret access key. | | `S3_REGION` | `us-east-1` | Bucket region. | | `S3_BUCKET_NAME` | `em-runtime` | Bucket name. | | `S3_PREFIX_UPLOADED` | `uploaded` | Prefix for user-uploaded artifacts. | | `S3_PREFIX_GENERATED` | `generated` | Prefix for system-generated artifacts. | ## Assets Storage (PVC) | Key | Type | Default | Description | | ---------------------------------------- | ------ | ------------------------------- | --------------------------------------- | | `em-runtime-assets.storage.enabled` | bool | `true` | Enable PVC storage. Set `false` for S3. | | `em-runtime-assets.storage.size` | string | `"10Gi"` | PVC size. | | `em-runtime-assets.storage.accessModes` | list | `["ReadWriteMany"]` | PVC access modes. | | `em-runtime-assets.storage.mountPath` | string | `"/var/lib/em-runtime/storage"` | Mount path in container. | | `em-runtime-assets.storage.storageClass` | string | `""` | StorageClass name (empty uses default). | ## Infrastructure Subcharts ### Keycloak | Key | Type | Default | Description | | ------------------------------------ | ------ | ---------- | ---------------------------- | | `keycloak.replicaCount` | int | `1` | Number of Keycloak replicas. | | `keycloak.resources.requests.cpu` | string | `"1000m"` | CPU request. | | `keycloak.resources.requests.memory` | string | `"1280Mi"` | Memory request. | | `keycloak.resources.limits.cpu` | string | `"2000m"` | CPU limit. | | `keycloak.resources.limits.memory` | string | `"2Gi"` | Memory limit. | ### OpenFGA | Key | Type | Default | Description | | ------------------------------------------------------- | ------ | --------- | --------------------------- | | `openfga.replicaCount` | int | `1` | Number of OpenFGA replicas. | | `openfga.autoscaling.enabled` | bool | `false` | Enable HPA. | | `openfga.autoscaling.targetCPUUtilizationPercentage` | int | `80` | CPU scaling target. | | `openfga.autoscaling.targetMemoryUtilizationPercentage` | int | `80` | Memory scaling target. | | `openfga.resources.requests.cpu` | string | `"100m"` | CPU request. | | `openfga.resources.requests.memory` | string | `"256Mi"` | Memory request. | ### Infisical (On-Premises / Cloud-Agnostic) Infisical is one of two supported secrets backends. For cloud deployments on GCP, ESO + GCP Secret Manager can be used instead. See [Secrets Management](/security/secrets-management) for a comparison. | Key | Type | Default | Description | | ---------------------------------------------------------- | ------ | --------- | ------------------------------------------------------------------------------- | | `infisical.enabled` | bool | `true` | Enable the Infisical subchart. Set `false` when using ESO + GCP Secret Manager. | | `infisical-standalone.infisical.replicaCount` | int | `1` | Number of Infisical replicas. | | `infisical-standalone.infisical.resources.requests.cpu` | string | `"350m"` | CPU request. | | `infisical-standalone.infisical.resources.requests.memory` | string | `"512Mi"` | Memory request. | ### PostgreSQL (In-Cluster) | Key | Type | Default | Description | | ------------------------------ | ------ | ------- | ------------------------------------------------------- | | `postgres.enabled` | bool | `true` | Enable in-cluster PostgreSQL. Set `false` for external. | | `postgres.replicaCount` | int | `1` | Number of PostgreSQL replicas. | | `postgres.persistence.enabled` | bool | `true` | Enable data persistence. | | `postgres.persistence.size` | string | `"8Gi"` | Persistent volume size. | ### Redis (In-Cluster) | Key | Type | Default | Description | | --------------------------- | ------ | ------- | -------------------------------------------------- | | `redis.enabled` | bool | `true` | Enable in-cluster Redis. Set `false` for external. | | `redis.replicaCount` | int | `1` | Number of Redis replicas. | | `redis.persistence.enabled` | bool | `true` | Enable data persistence. | | `redis.persistence.size` | string | `"8Gi"` | Persistent volume size. | ## Next Steps Deployment modes, secrets, and production setup guide. Version upgrades, migration steps, and rollback. # Networking Source: https://docs.emergence.ai/deployment/infrastructure/networking Configure Gateway API for traffic routing, cert-manager for TLS certificates, and external-dns for automated DNS record management. # Networking CRAFT uses **Gateway API** for traffic routing, **cert-manager** for TLS certificate lifecycle, and **external-dns** for automatic DNS record management. This guide covers the networking stack across cloud and on-premises environments. ## Gateway API EM-Runtime uses the [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/) (v1) for all external traffic routing. Gateway API replaces traditional Ingress with a more expressive, role-oriented model. ### Ingress Flow ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR Internet["Internet"] --> LB["Load Balancer (HTTPS)"] LB --> GW["Gateway (TLS terminates)"] GW --> Route["HTTPRoute"] Route --> Svc["Service"] Svc --> Pod["Pod"] classDef entry fill:#56B4E9,stroke:#555555,color:#000 classDef infra fill:#0072B2,stroke:#555555,color:#fff classDef app fill:#E69F00,stroke:#555555,color:#000 class Internet entry class LB,GW,Route infra class Svc,Pod app ``` ### Gateway Resource The Gateway resource defines the load balancer and listeners. It must exist before the em-runtime chart deploys HTTPRoutes. ```yaml theme={null} apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: em-runtime-gateway namespace: gateway annotations: cert-manager.io/cluster-issuer: letsencrypt-prod spec: gatewayClassName: nginx # or gke-l7-global-external-managed listeners: - name: https port: 443 protocol: HTTPS tls: mode: Terminate certificateRefs: - name: platform-tls allowedRoutes: namespaces: from: All - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: All ``` ### HTTPRoute Rules The em-runtime Helm chart creates an HTTPRoute with these default rules: | Path Prefix | Backend Service | Port | URL Rewrite | | ----------------- | --------------------- | ---- | ------------------- | | `/keycloak` | keycloak | 8080 | None (pass-through) | | `/api/governance` | em-runtime-governance | 8000 | `/governance` | | `/api/assets` | em-runtime-assets | 8000 | `/assets` | | `/api/utils` | em-runtime-utils | 8000 | `/utils` | ### Gateway Controllers by Platform | Platform | Controller | GatewayClassName | | ----------- | --------------------------------- | -------------------------------- | | **GKE** | GKE Gateway Controller (built-in) | `gke-l7-global-external-managed` | | **EKS** | AWS Load Balancer Controller | `aws-application-load-balancer` | | **AKS** | Azure Application Gateway | `azure-application-gateway` | | **On-Prem** | NGINX Gateway Fabric | `nginx` | | **On-Prem** | Envoy Gateway | `eg` | ### Configuration in Helm ```yaml theme={null} # Chart values for Gateway reference gatewayName: "em-runtime-gateway" gatewayNamespace: "gateway" hostname: "api.example.com" httpRoute: enabled: true rules: - pathPrefix: "/keycloak" serviceName: keycloak servicePort: 8080 - pathPrefix: "/api/governance" rewritePrefix: "/governance" serviceName: em-runtime-governance servicePort: 8000 ``` ## cert-manager [cert-manager](https://cert-manager.io/) automates TLS certificate provisioning and renewal. It supports ACME (Let's Encrypt) and internal CA issuers. ### Certificate Flow ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR CM["cert-manager"] -->|DNS-01 challenge| DNS["DNS Provider"] CM -->|stores| Secret["K8s Secret"] GW["Gateway"] -->|reads| Secret LE["Let's Encrypt"] -->|verifies| DNS LE -->|issues cert| CM classDef process fill:#E69F00,stroke:#555555,color:#000 classDef ext fill:#CC79A7,stroke:#555555,color:#000 classDef data fill:#009E73,stroke:#555555,color:#000 class CM,GW process class DNS,LE ext class Secret data ``` ### Installation Install cert-manager with Gateway API support: ```bash theme={null} helm install cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --set config.enableGatewayAPI=true \ --set crds.enabled=true ``` ### ClusterIssuer Configuration ```yaml theme={null} apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: server: https://acme-v2.api.letsencrypt.org/directory email: admin@example.com privateKeySecretRef: name: letsencrypt-prod-key solvers: - dns01: cloudDNS: # GCP project: your-project-id # route53: # AWS # region: us-east-1 # azureDNS: # Azure # subscriptionID: ... ``` ```yaml theme={null} apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: internal-ca spec: ca: secretName: internal-ca-key-pair --- # Create the CA key pair first apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: internal-ca namespace: cert-manager spec: isCA: true commonName: "CRAFT Internal CA" secretName: internal-ca-key-pair issuerRef: name: selfsigned kind: ClusterIssuer ``` ### DNS-01 Solver by Provider | Provider | Solver | IAM Requirement | | ------------------ | ---------------- | ----------------------------------- | | **GCP Cloud DNS** | `dns01.cloudDNS` | Workload Identity service account | | **AWS Route 53** | `dns01.route53` | IRSA role with Route 53 permissions | | **Azure DNS** | `dns01.azureDNS` | Workload Identity managed identity | | **On-Prem (BIND)** | `dns01.rfc2136` | TSIG key for dynamic DNS updates | ### Gateway Annotation Annotate the Gateway to trigger automatic certificate provisioning: ```yaml theme={null} metadata: annotations: cert-manager.io/cluster-issuer: letsencrypt-prod ``` cert-manager watches for Gateway resources with this annotation and provisions TLS certificates for each HTTPS listener. ## external-dns [external-dns](https://github.com/kubernetes-sigs/external-dns) synchronizes Kubernetes resources (Gateway, Service, Ingress) with DNS providers, automatically creating A/CNAME records. ### How It Works 1. external-dns watches Gateway and Service resources for hostname annotations 2. When a Gateway gets an external IP, external-dns creates a DNS A record 3. Records are updated or removed as resources change ### Configuration by Provider ```yaml theme={null} provider: google google: project: your-project-id domainFilters: - runtime.dev.emergence.ai policy: sync sources: - gateway-httproute - service ``` Requires a Workload Identity service account with `dns.admin` role. ```yaml theme={null} provider: aws aws: region: us-east-1 domainFilters: - runtime.dev.emergence.ai policy: sync sources: - gateway-httproute - service ``` Requires an IRSA role with Route 53 `ChangeResourceRecordSets` permission. ```yaml theme={null} provider: rfc2136 rfc2136: host: dns.example.com port: 53 zone: example.com tsigKeyname: external-dns tsigSecretAlg: hmac-sha256 policy: sync sources: - gateway-httproute - service ``` Requires a TSIG key configured on the BIND DNS server. ## TLS Requirements | Boundary | Requirement | | ----------------------- | ---------------------------------- | | Client to load balancer | TLS 1.2+ (certificate required) | | Load balancer to pods | HTTP (in-cluster; TLS optional) | | Services to PostgreSQL | SSL recommended (private network) | | Services to Redis | TLS supported (`rediss://` scheme) | ## Network Policies For on-premises deployments, configure Kubernetes NetworkPolicies for defense in depth: ```yaml theme={null} # Restrict data-insights egress to LLM API endpoints apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: talk2data-egress namespace: em-runtime spec: podSelector: matchLabels: app: data-insights policyTypes: - Egress egress: - to: - ipBlock: cidr: 0.0.0.0/0 ports: - port: 443 protocol: TCP ``` Restrict data-insights egress to allowlisted LLM API endpoints (OpenAI, Anthropic, Vertex AI) for SSRF mitigation. ## Firewall Rules Ensure the following traffic is permitted: | Direction | Port | Protocol | Purpose | | -------------------- | ---- | -------- | --------------------------------- | | **Inbound** | 443 | TCP | HTTPS from clients | | **Inbound** | 80 | TCP | HTTP redirect to HTTPS | | **Cluster to DB** | 5432 | TCP | PostgreSQL | | **Cluster to Cache** | 6379 | TCP | Redis | | **Cluster to S3** | 443 | TCP | Object storage | | **Cluster to LLM** | 443 | TCP | LLM API endpoints (data-insights) | ## DNS Architecture One DNS A record pointing the platform hostname to the load balancer IP/CNAME: ```text theme={null} api.example.com → A → ``` For GKE deployments, external-dns creates this automatically when the Gateway receives an IP address. For on-prem, configure the DNS record manually or use external-dns with RFC2136. ## Troubleshooting ```bash theme={null} # Check cert-manager logs kubectl logs -n cert-manager -l app=cert-manager # Check certificate and challenge status kubectl describe certificate -n gateway-system kubectl get challenges -n gateway-system ``` ```bash theme={null} # Check Gateway status kubectl describe gateway em-runtime-gateway -n gateway # Verify the gateway controller is running kubectl get pods -n gateway-system ``` ```bash theme={null} # Check external-dns logs kubectl logs -l app.kubernetes.io/name=external-dns # Verify IAM / Workload Identity kubectl describe sa external-dns -n external-dns ``` ## Next Steps Complete networking prerequisites and firewall rules. # Secrets Infrastructure Source: https://docs.emergence.ai/deployment/infrastructure/secrets Deploy and configure ESO + GCP Secret Manager or Infisical for secrets management in CRAFT. # Secrets Infrastructure This guide covers deploying and configuring the secrets management backend for CRAFT. Two options are supported, choose based on your deployment environment. For a comparison of the two backends and guidance on which to choose, see [Secrets Management](/security/secrets-management). *** ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% graph TB subgraph OptionA["Option A: ESO + GCP Secret Manager"] GCP["GCP Secret Manager\n(Terraform provisioned)"] CSS["ClusterSecretStore\n(Workload Identity auth)"] ES["ExternalSecret\n(K8s resource, refresh interval)"] KS["Kubernetes Secret"] SR["Stakater Reloader\n(watches annotations)"] POD1["Pod restart\n(rolling update)"] GCP --> CSS --> ES --> KS --> SR --> POD1 end subgraph OptionB["Option B: Infisical"] INF["Infisical Server\n(Helm subchart)"] MI["Machine Identity\n(credentials in Helm values)"] POD2["Pods fetch secrets\nat startup via env injection"] INF --> MI --> POD2 end classDef gcp fill:#0072B2,stroke:#555555,color:#fff classDef k8s fill:#56B4E9,stroke:#555555,color:#000 classDef infra fill:#E69F00,stroke:#555555,color:#000 classDef pod fill:#009E73,stroke:#555555,color:#000 class GCP gcp class CSS,ES,KS,SR k8s class INF,MI infra class POD1,POD2 pod ``` ## Option A: ESO + GCP Secret Manager For GKE and GCP-based deployments. Infrastructure is provisioned by Terraform; Kubernetes resources are managed by ArgoCD. ### Prerequisites * GKE cluster with Workload Identity enabled * Terraform-provisioned GCP infrastructure * ArgoCD deployed and managing the cluster ### 1. Terraform Provisions GCP Secrets The `gke` Terraform module automatically creates the required secrets in GCP Secret Manager when you run `terraform apply`. No manual secret creation is needed for infrastructure secrets, the module provisions them with placeholder values that you update after provisioning. After running Terraform: 1. Navigate to [GCP Secret Manager](https://console.cloud.google.com/security/secret-manager) in your project 2. Update each provisioned secret with its actual value 3. ESO will sync the updated values into Kubernetes on the next sync cycle ### 2. Install External Secrets Operator ESO can be installed as an ArgoCD-managed application or deployed manually: ```bash theme={null} helm install external-secrets external-secrets/external-secrets \ --namespace external-secrets \ --create-namespace \ --set installCRDs=true ``` ### 3. ClusterSecretStore ESO requires a `ClusterSecretStore` that configures how to authenticate to GCP Secret Manager. This is provisioned by Terraform/ArgoCD: ```yaml theme={null} apiVersion: external-secrets.io/v1beta1 kind: ClusterSecretStore metadata: name: gcp-secret-manager spec: provider: gcpsm: projectID: "your-project-id" # Replace with your GCP project auth: workloadIdentity: clusterLocation: us-central1 clusterName: "your-cluster-name" serviceAccountRef: name: external-secrets namespace: external-secrets ``` ### 4. ExternalSecret Resources Each service has `ExternalSecret` resources that map GCP SM secrets to K8s Secrets. These are managed by ArgoCD in the `kubernetes/em-runtime/` directory: ```yaml theme={null} apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: em-runtime-credentials namespace: em-runtime annotations: reloader.stakater.com/auto: "true" # Enable auto-restart on change spec: refreshInterval: 1h secretStoreRef: name: gcp-secret-manager kind: ClusterSecretStore target: name: em-runtime-credentials creationPolicy: Owner data: - secretKey: postgres-password remoteRef: key: em-runtime # GCP SM secret name property: postgres-password - secretKey: redis-password remoteRef: key: em-runtime property: redis-password ``` ### 5. Install Stakater Reloader Reloader watches K8s Secrets for changes and triggers rolling restarts of annotated pods: ```bash theme={null} helm install reloader stakater/reloader \ --namespace em-runtime \ --set reloader.watchGlobally=false ``` Pods are automatically enrolled when their Deployment or the Secret has the annotation: ```yaml theme={null} metadata: annotations: reloader.stakater.com/auto: "true" ``` ### 6. Verify Sync ```bash theme={null} # Check ExternalSecret status kubectl get externalsecrets -n em-runtime # Check the synced K8s Secret exists kubectl get secret em-runtime-credentials -n em-runtime # Check Reloader is watching kubectl logs -n em-runtime -l app.kubernetes.io/name=reloader | head -20 ``` ### Manual Sync (Immediate Rotation) To force an immediate sync without waiting for the refresh interval: ```bash theme={null} kubectl annotate externalsecret em-runtime-credentials \ --namespace em-runtime \ force-sync="$(date +%s)" \ --overwrite ``` *** ## Option B: Infisical For on-premises and cloud-agnostic deployments. Infisical runs as a Helm subchart within em-runtime. ### Prerequisites * Kubernetes cluster (any CNCF-conformant distribution) * Helm chart deployed with Infisical subchart enabled ### 1. Enable Infisical in Helm Values Infisical is enabled by default. Verify in your values file: ```yaml theme={null} infisical: enabled: true resources: requests: cpu: "250m" memory: "512Mi" limits: cpu: "1000m" memory: "1Gi" ``` ### 2. Bootstrap Infisical On first deployment, Infisical initializes with the admin email configured in Helm: ```yaml theme={null} bootstrap: adminEmail: "admin@your-domain.com" ``` Access the Infisical UI and complete initial setup: ```bash theme={null} kubectl port-forward svc/em-runtime-infisical -n em-runtime 8080:80 # Open http://localhost:8080 ``` ### 3. Create Machine Identity Create a Kubernetes machine identity in Infisical for each service that needs secret access: 1. In the Infisical UI, navigate to **Project → Access Control → Machine Identities** 2. Create a machine identity with the Kubernetes authentication method 3. Configure the service account name and namespace ### 4. Inject Credentials Infisical credentials are passed to services via environment variables in Helm values: ```yaml theme={null} governance: extraEnvVars: - name: INFISICAL_URL value: "http://em-runtime-infisical.em-runtime.svc.cluster.local" - name: INFISICAL_CLIENT_ID valueFrom: secretKeyRef: name: infisical-machine-identity key: clientId - name: INFISICAL_CLIENT_SECRET valueFrom: secretKeyRef: name: infisical-machine-identity key: clientSecret ``` ### 5. Verify ```bash theme={null} # Check Infisical pod is running kubectl get pods -n em-runtime -l app.kubernetes.io/name=infisical # Check governance service logs for Infisical connection kubectl logs -n em-runtime -l app=em-runtime-governance | grep -i infisical ``` *** ## Backup and Recovery ### ESO + GCP Secret Manager GCP Secret Manager has built-in version history. To recover a previous secret value: ```bash theme={null} # List secret versions gcloud secrets versions list your-secret-name --project=your-project-id # Access a specific version gcloud secrets versions access VERSION_ID --secret=your-secret-name --project=your-project-id ``` ### Infisical Back up the Infisical PostgreSQL database (`infisical` database) using standard PostgreSQL backup tools: ```bash theme={null} kubectl exec -n em-runtime deployment/em-runtime-infisical -- \ pg_dump -U infisical infisical > infisical-backup-$(date +%Y%m%d).sql ``` Include this backup in your regular database backup rotation. See [Backup and Restore](/guides/backup-restore) for the full backup procedure. ## Next Steps Concepts, architecture, and backend comparison. Full backup procedures including secrets backends. # Langfuse Source: https://docs.emergence.ai/deployment/observability/langfuse Deploy and configure Langfuse for LLM observability in CRAFT, traces, evaluations, and prompt management. # Langfuse [Langfuse](https://langfuse.com) is an LLM observability platform used by CRAFT to trace LLM calls, manage prompts, and run evaluations. CRAFT services are auto-instrumented to emit traces in the Langfuse format whenever a Langfuse host is configured. Langfuse is **optional**. The platform operates normally without it. When configured, it enables LLM call tracing, evaluation, and prompt versioning. For the upstream Langfuse documentation, see [langfuse.com/docs](https://langfuse.com/docs). ## LLM Observability vs. Infrastructure Observability | | Langfuse | OpenTelemetry | | ----------------------- | ------------------------------------------------------------------- | -------------------------------------------- | | **Purpose** | LLM call tracing, evaluation, cost | Infrastructure metrics, request traces, logs | | **What it instruments** | LLM API calls (tokens, latency, cost, quality) | Application services (HTTP, DB, Redis) | | **Integration** | LiteLLM callbacks plus `@observe` decorators on solution code paths | OTLP exporter | | **When active** | `LANGFUSE_HOST` configured | `OTEL_ENABLED=true` | See [LLM Observability](/deployment/observability/llm-observability) for a conceptual overview of how these complement each other. ## Auto-Enabled Integration Langfuse is auto-enabled when `LANGFUSE_HOST` is set in any solution's environment. The integration works through **LiteLLM callbacks** plus `@observe` decorators on key code paths. No code changes are needed in the solution services. When enabled, Langfuse captures: * Every LLM API call (model, prompt, completion, tokens, cost, latency) * Agent session traces (multi-turn conversation flows) * Evaluation results from scheduled and manual evaluations * Prompt template versions and their usage ## Cross-service tracing Recent solution updates introduced a unified tracing model so a single user conversation produces a single, end-to-end trace, even when it crosses agent boundaries: * **Auto-instrumented entry points.** Chat dispatch and the pipeline executor are decorated with `@observe`, so every conversation turn opens a trace span automatically. * **A2A trace context propagation.** When one agent calls another via the A2A protocol, the trace context propagates through the A2A metadata so the receiving agent's spans become children of the originating trace, not a new root. * **Deterministic trace IDs from `turn_id`.** Trace IDs are seeded from the conversation turn ID. Reprocessing the same turn (e.g., a retried evaluation) emits to the same trace rather than creating a duplicate. * **Per-iteration spans inside agent loops.** Multi-step agent loops emit one span per iteration so you can see how many planning steps a question took, how many tool calls each step made, and where time was spent. The net effect: the Langfuse UI shows one trace per user question, regardless of how many services or LLM calls were involved. ## Deployment Langfuse is self-hosted. Choose the deployment model that fits your infrastructure. For single-VM, demo, or local deployments, use the public upstream `langfuse/langfuse` Docker image: ```bash theme={null} # Get the upstream docker-compose.yml curl -L -o docker-compose.yml https://raw.githubusercontent.com/langfuse/langfuse/main/docker-compose.yml # Configure environment (see langfuse.com/docs for full reference) # Required: NEXTAUTH_SECRET, SALT, database credentials docker compose up -d ``` This starts the standard Langfuse stack: Web (Next.js), Worker, PostgreSQL, ClickHouse, Redis, and a blob store. See [langfuse.com/docs/deployment/self-host](https://langfuse.com/docs/deployment/self-host) for the upstream self-hosting guide and configuration reference. For production Kubernetes deployments, use the upstream Helm chart: ```bash theme={null} helm repo add langfuse https://langfuse.github.io/langfuse-k8s helm install langfuse langfuse/langfuse \ --namespace langfuse \ --create-namespace \ --set langfuse.nextauth.secret="" \ --set langfuse.salt="" \ --set postgresql.auth.password="" ``` See [langfuse.com/docs/deployment/self-host/kubernetes](https://langfuse.com/docs/deployment/self-host/kubernetes) for full Helm values. ## Connecting CRAFT solutions to Langfuse After deploying Langfuse (upstream or internal fork), configure each solution to send traces. ### Data Insights (em-talk2data) ```yaml theme={null} # Helm values or environment em-talk2data: extraEnvVars: - name: LANGFUSE_HOST value: "https://langfuse.example.com" - name: LANGFUSE_PUBLIC_KEY valueFrom: secretKeyRef: name: langfuse-credentials key: public-key - name: LANGFUSE_SECRET_KEY valueFrom: secretKeyRef: name: langfuse-credentials key: secret-key ``` ### Data Governance (em-data-readiness) Same environment variables: `LANGFUSE_HOST`, `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`. ### Semiconductor (em-semi) Same environment variables applied to the Backend API and AI Agent services. ## Required Environment Variables | Variable | Description | | --------------------- | -------------------------------------------------------------------- | | `LANGFUSE_HOST` | URL of your Langfuse instance (e.g., `https://langfuse.example.com`) | | `LANGFUSE_PUBLIC_KEY` | Project public key from Langfuse dashboard | | `LANGFUSE_SECRET_KEY` | Project secret key from Langfuse dashboard | ## Accessing Langfuse After deployment, access the Langfuse UI at your configured host URL. 1. Create an account or log in. 2. Create a **project** for each solution you want to trace. 3. Copy the **Public Key** and **Secret Key** from the project settings. 4. Configure these in the solution's environment (see above). ## Related Why LLM observability matters and how Langfuse and OTel complement each other. Step-by-step setup including all configuration options. Infrastructure-level observability (metrics, traces, logs). # LLM Observability Source: https://docs.emergence.ai/deployment/observability/llm-observability Understanding LLM-specific observability for CRAFT, why it differs from infrastructure observability and how Langfuse and OpenTelemetry complement each other. # LLM Observability CRAFT uses two complementary observability systems that operate at different layers: * **OpenTelemetry**, infrastructure-level: HTTP requests, database queries, Redis, service health * **Langfuse**, LLM-level: model calls, token usage, cost, prompt quality, evaluation results Both are valuable; neither replaces the other. ## Why LLM Observability Is Different Traditional observability (metrics, traces, logs) was designed for deterministic systems. LLM-powered applications introduce unique observability challenges: | Challenge | Why it matters | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------- | | **Non-determinism** | The same prompt can produce different outputs. Standard error rates don't capture quality degradation. | | **Token economics** | Cost is proportional to token usage, not request count. A single call can cost $0.001 or $1.00 depending on prompt length. | | **Prompt engineering** | Changing a prompt is a configuration change, not a code change, but it can dramatically affect output quality. | | **Evaluation** | "Is this response correct?" requires domain-specific evaluation, not just latency or error rate checks. | | **Multi-turn context** | A user session spans multiple LLM calls. Standard tracing doesn't capture the logical conversation flow. | ## What Each System Captures ### OpenTelemetry (Infrastructure) ```text theme={null} Service → [HTTP/DB/Redis spans] → OTel Collector → Grafana ``` | Signal | Examples | | ----------- | ----------------------------------------------------- | | **Metrics** | Request rate, P99 latency, error rate, pod CPU/memory | | **Traces** | End-to-end request flow through services | | **Logs** | Service logs, error messages, audit events | Answers: "Is the service up? Is it slow? Are there errors?" ### Langfuse (LLM) ```text theme={null} LLM call → [LiteLLM callback] → Langfuse ``` | Signal | Examples | | --------------- | ----------------------------------------------------------- | | **Traces** | Complete LLM session with all turns, tools, and context | | **Metrics** | Tokens used, estimated cost, latency per model call | | **Evaluations** | LLM-as-a-judge quality scores, human labels, rubric results | | **Prompts** | Version history, usage statistics, A/B comparison | Answers: "Is the AI producing quality output? What's it costing? Which prompt version is better?" ## Architecture ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TB subgraph App["Application Services"] Svc["em-talk2data / em-semi"] LLM["LiteLLM (LLM proxy)"] end subgraph OTel["Infrastructure Observability"] OTelC["OTel Collector"] Prom["Metrics Backend"] Grafana["Grafana"] end subgraph LFuse["LLM Observability"] LF["Langfuse"] end Svc -->|HTTP spans| OTelC Svc --> LLM LLM -->|LiteLLM callbacks| LF LLM -->|LLM API calls| ExtLLM["Gemini / GPT / Claude"] OTelC --> Prom --> Grafana LF --> LFDash["Langfuse Dashboard"] classDef app fill:#E69F00,stroke:#555555,color:#000 classDef otel fill:#009E73,stroke:#555555,color:#000 classDef lf fill:#0072B2,stroke:#555555,color:#fff classDef ext fill:#CC79A7,stroke:#555555,color:#000 class Svc,LLM app class OTelC,Prom,Grafana otel class LF,LFDash lf class ExtLLM ext ``` ## Integration Pattern The platform uses **LiteLLM** as a provider-agnostic LLM proxy. LiteLLM natively supports Langfuse as a callback handler, requiring no changes to application code. When `LANGFUSE_HOST` is set, LiteLLM automatically: 1. Records each LLM API call to Langfuse (prompt, completion, model, tokens, cost) 2. Groups calls into sessions by conversation ID 3. Reports evaluation scores if evaluators are configured ## Langfuse Trace Anatomy A Langfuse trace for a Data Insights session might look like: ```text theme={null} Session: "user-query-schema-explain" └─ Trace: /api/chat (3.2s) ├─ Span: schema_lookup (0.3s) [DB query] ├─ Span: LLM call (2.5s) │ Model: gemini-3.5-flash │ Input tokens: 1,240 │ Output tokens: 380 │ Cost: $0.0018 │ Quality score: 0.87 (LLM-as-judge) └─ Span: format_response (0.4s) ``` ## Cost Tracking Langfuse aggregates LLM costs across all calls, enabling: * Cost per conversation / session / user * Cost trends over time * Model comparison (cost vs. quality tradeoff) * Budget alerting (configurable thresholds) ## Related Deploy and configure Langfuse. Infrastructure observability with OTel. # OpenTelemetry Source: https://docs.emergence.ai/deployment/observability/opentelemetry Configure OpenTelemetry for distributed tracing, metrics, and log export across all em-runtime services using the vendor-neutral OTLP protocol. # OpenTelemetry All em-runtime services export traces, metrics, and logs via [OpenTelemetry](https://opentelemetry.io/) using the vendor-neutral OTLP protocol. The platform ships with built-in instrumentation -- no application code changes are needed. ## Architecture ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TB subgraph Apps["Applications (OTel Built-in)"] GOV["governance"] AST["assets"] UTL["utils"] T2D["Data Insights"] DRY["Data Governance"] KC["Keycloak (/metrics)"] end subgraph Collection["Collection"] OTel["OpenTelemetry Collector"] end subgraph Storage["Storage"] Prom["Metrics Backend"] Tempo["Grafana Tempo (Traces)"] Loki["Loki (Logs)"] end subgraph Viz["Visualization"] Grafana["Grafana"] AM["Alertmanager"] end GOV & AST & UTL & T2D & DRY -->|OTLP gRPC/HTTP| OTel KC -->|metrics scrape| Prom OTel -->|Remote Write| Prom OTel -->|OTLP| Tempo Prom & Tempo & Loki --> Grafana Prom --> AM classDef app fill:#E69F00,stroke:#555555,color:#000 classDef collect fill:#56B4E9,stroke:#555555,color:#000 classDef storage fill:#009E73,stroke:#555555,color:#000 classDef viz fill:#0072B2,stroke:#555555,color:#fff class GOV,AST,UTL,T2D,DRY,KC app class OTel collect class Prom,Tempo,Loki storage class Grafana,AM viz ``` The OTLP collector is **not** included in the em-runtime Helm chart. Deploy your preferred collector separately. Any OTLP-compatible backend works: Grafana Cloud, Datadog, Splunk, New Relic, Honeycomb, or self-hosted. ## OpenTelemetry vs Langfuse These two observability systems are **complementary**, not overlapping: | | OpenTelemetry | Langfuse | | ----------------- | ------------------------------------------------ | ----------------------------------------------------- | | **Instruments** | HTTP requests, DB queries, Redis, service health | LLM API calls (model, prompt, tokens, cost, quality) | | **Answers** | "Is the service healthy? Is it slow?" | "Is the AI producing good output? What does it cost?" | | **Integration** | OTLP exporter (always active) | LiteLLM callbacks (when `LANGFUSE_HOST` is set) | | **Storage** | PromQL-compatible metrics backend / Tempo / Loki | Langfuse's own PostgreSQL + ClickHouse | | **Visualization** | Grafana | Langfuse UI | Use both together: OTel for infrastructure health, Langfuse for LLM quality and cost. See the Langfuse setup guide for LLM-specific observability. ## Configuration Telemetry is controlled via environment variables in each service's `env` block in the Helm values: | Variable | Default | Description | | ----------------------------- | ------------------------------ | ----------------------------------------------------------------------------------------- | | `OTEL_ENABLED` | `"true"` | Master switch for all telemetry. | | `OTEL_EXPORTER_OTLP_ENDPOINT` | `"http://otel-collector:4317"` | OTLP collector gRPC endpoint. | | `OTEL_TRACES_ENABLED` | `"true"` | Enable distributed tracing. | | `OTEL_METRICS_ENABLED` | `"true"` | Enable metrics export. | | `OTEL_LOGS_ENABLED` | `"true"` | Enable log record export via OTLP. | | `OTEL_TRACE_SAMPLE_RATE` | `"1.0"` | Trace sampling ratio (0.0-1.0). Default is 100%; override via Helm values for production. | ### Point to Your Collector Override the endpoint in your values file: ```yaml theme={null} em-runtime-governance: env: OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector.monitoring:4317" em-runtime-assets: env: OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector.monitoring:4317" em-runtime-utils: env: OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector.monitoring:4317" ``` ### Disable Telemetry For test environments without a collector: ```yaml theme={null} em-runtime-governance: env: OTEL_ENABLED: "false" ``` Individual signals can also be toggled independently. Telemetry initialization is non-fatal. If the collector is unreachable, services log a warning and continue operating normally. ## Auto-Instrumented Libraries The following libraries are automatically instrumented with no code changes: | Library | What It Captures | | -------------- | ----------------------------------------------------- | | **FastAPI** | Inbound HTTP request spans (excludes `/health`) | | **SQLAlchemy** | Database query spans and connection metrics | | **Redis** | Redis command spans | | **httpx** | Outbound HTTP request spans (inter-service SDK calls) | ## Telemetry Signals ### Traces Distributed traces follow the [W3C Trace Context](https://www.w3.org/TR/trace-context/) format. Traces propagate across service boundaries automatically via httpx instrumentation. Key trace fields: * `service.name` -- identifies the emitting service * `http.method`, `http.url`, `http.status_code` -- HTTP span attributes * `db.system`, `db.statement` -- database query details * `trace_id` -- correlates logs and traces ### Metrics Application metrics are exported via OTLP and include: * HTTP request duration histograms * HTTP request counts by status code * Database connection pool utilization * Redis command latency ### Logs Structured JSON logs are written to stdout and optionally exported via OTLP: | Field | Description | | ----------- | --------------------------------------- | | `level` | Log level (DEBUG, INFO, WARNING, ERROR) | | `message` | Log message | | `timestamp` | ISO 8601 timestamp | | `service` | Service name | | `trace_id` | W3C trace ID for log-trace correlation | ## OTel Collector Configuration Deploy the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) to receive, process, and export telemetry data. ### Recommended Setup ```yaml theme={null} # otel-collector-config.yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: timeout: 5s send_batch_size: 1024 exporters: # Traces otlp/tempo: endpoint: tempo.monitoring:4317 tls: insecure: true # Metrics prometheusremotewrite: endpoint: http://prometheus.monitoring:9090/api/v1/write # Logs loki: endpoint: http://loki.monitoring:3100/loki/api/v1/push service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp/tempo] metrics: receivers: [otlp] processors: [batch] exporters: [prometheusremotewrite] logs: receivers: [otlp] processors: [batch] exporters: [loki] ``` ### Helm Installation ```bash theme={null} helm install otel-collector open-telemetry/opentelemetry-collector \ --namespace monitoring \ --create-namespace \ -f otel-collector-config.yaml ``` ## Metric Sources | Source | Endpoint | Protocol | | --------------------------- | --------------------------------- | ---------------------------- | | Application services (OTel) | OTLP gRPC (4317) / HTTP (4318) | OpenTelemetry | | Keycloak | `/keycloak/metrics` (port 8080) | Prometheus exposition format | | Kubernetes | kube-state-metrics, node-exporter | Prometheus exposition format | | PostgreSQL | pg\_exporter (optional) | Prometheus exposition format | | Redis | redis\_exporter (optional) | Prometheus exposition format | ## Backend Options | Component | Purpose | | ----------------- | ------------------------------ | | **Prometheus** | Metrics storage and querying | | **Grafana Tempo** | Distributed trace storage | | **Loki** | Log aggregation | | **Grafana** | Dashboards and visualization | | **Alertmanager** | Alert routing and notification | Deploy via the [Grafana LGTM Helm chart](https://grafana.com/docs/lgtm/latest/get-started/install/kubernetes/) or individual component charts. Point the OTel Collector exporters to Grafana Cloud endpoints: ```yaml theme={null} exporters: otlp/grafana: endpoint: tempo-.grafana.net:443 headers: Authorization: "Basic " ``` Use the Datadog Agent as an OTLP receiver: ```yaml theme={null} exporters: datadog: api: key: ${DD_API_KEY} ``` Use the AWS Distro for OpenTelemetry (ADOT) Collector: ```yaml theme={null} exporters: awsxray: {} awsemf: {} ``` ## Next Steps Telemetry environment variables in Helm values. Infrastructure requirements for the observability stack. # CRAFT Deployment Overview Source: https://docs.emergence.ai/deployment/overview Deploy CRAFT on any Kubernetes cluster using Helm charts and ArgoCD GitOps. Supports GKE, EKS, AKS, and on-premises environments. # Deployment Overview Run CRAFT on any Kubernetes cluster. The platform uses **Helm charts** for application packaging and **ArgoCD** for GitOps-based continuous deployment across four environments: dev, staging, demo, and production. CRAFT deploys on **your infrastructure**. The platform runs on any CNCF-conformant Kubernetes cluster -- GKE, EKS, AKS, or self-managed -- without cloud-specific dependencies. You choose where to run; the same Helm chart deploys across all environments, while provisioning steps vary by environment. ## Supported Environments Deploy on GKE Autopilot or Standard clusters. Terraform modules automate cluster, VPC, IAM, and Secret Manager provisioning. Deploy on Amazon EKS with Helm charts. Terraform modules for automated provisioning are planned. Deploy on Azure AKS with Helm charts. Terraform modules for automated provisioning are planned. Deploy on any CNCF-conformant cluster (v1.28+). No cloud-specific services required -- use self-hosted PostgreSQL, Redis, and MinIO. ## Helm Chart The em-runtime Helm chart packages all platform services and their dependencies: ### Included Components | Component | Type | Purpose | | -------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------- | | **Governance** | Deployment | Identity, authorization, secrets management (port 8000) | | **Assets** | Deployment | Artifacts, data connections, files, models (port 8000) | | **Utils** | Deployment | Data catalog, search, scheduling, context packs, memories (port 8000) | | **PostgreSQL** | StatefulSet / External | Primary data store for all services | | **Redis** | StatefulSet / External | Caching, messaging, event streaming | | **Keycloak** | StatefulSet / External | Multi-realm JWT authentication | | **OpenFGA** | Deployment / External | Relationship-based access control | | **Infisical** | Deployment / External | Secrets management, one of two supported backends. See [Secrets Management](/security/secrets-management). | ### Deployment Modes All dependencies (PostgreSQL, Redis, Keycloak, OpenFGA) are deployed as part of the Helm release within the Kubernetes cluster. Best for: * Development and testing environments * Standalone deployments * Quick setup with minimal external dependencies PostgreSQL and Redis run outside the Kubernetes cluster (managed services or Docker). The Helm chart connects to them via configuration values. Best for: * Production environments with managed database services * Existing infrastructure reuse * Separation of stateful and stateless workloads ### Solution Integration Solutions declare `em-runtime` as a **Helm subchart dependency** in their `Chart.yaml`. This ensures: * Platform services are co-deployed with the solution * Consistent Keycloak, OpenFGA, and database configuration * Single `helm install` deploys the complete stack * Version-pinned dependencies for reproducible deployments ## Service Startup Order The Helm chart respects the required startup order via init containers and readiness probes: ```text theme={null} PostgreSQL -> Redis -> Keycloak -> OpenFGA -> [Infisical] -> Governance -> Assets/Utils -> Solutions ``` Governance must complete its bootstrap process (initializing Keycloak realms and OpenFGA schema) before Assets, Utils, or any solution can start. If Governance is down, downstream services return 403 on all permission-checked endpoints. ## Observability Stack The deployment includes a full observability stack via the Grafana LGTM bundle: | Component | Purpose | | ----------------- | ------------------------------------------------------ | | **OpenTelemetry** | OTLP-based traces, metrics, and logs from all services | | **Loki** | Log aggregation and querying | | **Grafana** | Dashboards and visualization | | **Tempo** | Distributed tracing | | **Mimir** | Metrics storage and querying | ## Next Steps Customize the Helm chart values for your environment. Set up OpenTelemetry and the Grafana LGTM stack. Gateway API, cert-manager, and DNS configuration. Configure ESO + GCP Secret Manager or Infisical. # Prerequisites Source: https://docs.emergence.ai/deployment/prerequisites Infrastructure, networking, database, and access requirements for deploying CRAFT on Kubernetes. # Prerequisites Before deploying EM-Runtime, ensure your environment meets the infrastructure, database, networking, and access requirements listed below. ## Kubernetes Cluster **1.28+** required for Gateway API v1 support. **3 nodes** for HA pod distribution across failure domains. **4 vCPU, 16 GB RAM** per node. **8 vCPU, 32 GB RAM** per node for HPA burst headroom. | Requirement | Specification | | ------------------------- | -------------------------------------------------------------------------------- | | **Cluster type** | Managed (GKE, EKS, AKS) or self-managed (RKE2, k3s, OpenShift) | | **CNI plugin** | Calico, Cilium, or cloud-native CNI | | **Gateway API** | Gateway API v1 controller (NGINX Gateway Fabric, Envoy Gateway, or cloud-native) | | **cert-manager** | v1.x for TLS certificate lifecycle | | **external-dns** | Optional, for automatic DNS record creation | | **Image registry access** | Pull access to `ghcr.io/emergenceai/*` (private) | ### IP Address Planning Allocate three non-overlapping CIDR ranges: | Range | Purpose | Recommended Size | | ------------ | ------------------- | --------------------- | | Node CIDR | Kubernetes node IPs | /20 (4,096 addresses) | | Pod CIDR | Pod network | /19 (8,192 addresses) | | Service CIDR | ClusterIP services | /22 (1,024 addresses) | ## Compute Resources ### Per-Component Resource Allocation The table shows **Helm chart defaults** (suitable for dev/staging) and **production overrides** (used in GCP deployments). | Component | Min Replicas | CPU Request (Default / Prod) | CPU Limit | Memory Request (Default / Prod) | Memory Limit | | | --------------------- | ------------- | ---------------------------- | --------- | ------------------------------- | ------------ | ------------------------------------------ | | Keycloak | 2 | 1000m / 1000m | 2000m | 1280Mi / 1536Mi | 2Gi | | | OpenFGA | 2 (HPA: 2-10) | 100m / 200m | 500m | 256Mi / 256Mi | 512Mi | | | Infisical | 2 | 350m / 500m | 1000m | 512Mi / 512Mi | 1Gi | (if using Infisical backend; omit for ESO) | | em-runtime-governance | 2 (HPA: 2-10) | 250m / 500m | 1000m | 512Mi / 512Mi | 1Gi | | | em-runtime-assets | 2 (HPA: 2-10) | 250m / 500m | 1000m | 512Mi / 512Mi | 1Gi | | | em-runtime-utils | 2 (HPA: 2-10) | 100m / 250m | 500m | 256Mi / 256Mi | 512Mi | | ### Aggregate Totals (Core Only, Helm Defaults) | Scenario | CPU Request | CPU Limit | Memory Request | Memory Limit | | ------------------------------------ | ----------- | --------- | -------------- | ------------ | | **Minimum (all at min replicas)** | 4.1 vCPU | 12.0 vCPU | 6.5 GiB | 12.0 GiB | | **Maximum (HPA services at max 10)** | 9.7 vCPU | 36.0 vCPU | 18.5 GiB | 36.0 GiB | Add 10-20% overhead for system pods (kube-system, cert-manager, external-dns, gateway controller, monitoring agents). Companion solution services (Data Insights, Data Governance) add additional resources. ## Infrastructure Overview ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% graph TB subgraph Cluster["Kubernetes Cluster (3+ nodes, 4vCPU/16GB each)"] GW["Gateway API v1\n(HTTPRoute + TLS)"] subgraph Services["Platform Services"] KC["Keycloak 26\n(2 replicas)"] OF["OpenFGA 1.11\n(2–10 replicas)"] GV["Governance\n(2–10 replicas)"] AS["Assets\n(2–10 replicas)"] UT["Utils\n(2–10 replicas)"] end end subgraph Data["Data Layer"] PG[("PostgreSQL 18+\n9 databases\n2vCPU / 4GB / 100GB SSD")] RD[("Redis 8.4.2\n5GB / noeviction")] end OBJ[("Object Storage\n(S3-compatible, private)")] CERT["cert-manager\n(TLS certificates)"] GW --> Services Services --> PG & RD Services --> OBJ CERT --> GW classDef cluster fill:#56B4E9,stroke:#555555,color:#000 classDef svc fill:#E69F00,stroke:#555555,color:#000 classDef data fill:#0072B2,stroke:#555555,color:#fff classDef ext fill:#009E73,stroke:#555555,color:#000 class GW cluster class KC,OF,GV,AS,UT svc class PG,RD data class OBJ,CERT ext ``` ## Database Requirements ### PostgreSQL | Parameter | Specification | | ----------------------- | ------------------------------- | | **Version** | PostgreSQL 18 | | **Instance sizing** | 2 vCPU, 4 GB RAM (minimum) | | **High availability** | Multi-AZ / regional replication | | **Initial disk** | 100 GB SSD | | **Maximum disk** | 1,000 GB (auto-resize) | | **Backup retention** | 7 days minimum | | **Required extensions** | `pg_cron` (on `utils` database) | #### Databases (9 total, single shared instance) | Database | Used By | | --------------- | ---------------------------------------------------------------------------------- | | `keycloak` | Keycloak IAM | | `openfga` | OpenFGA authorization | | `infisical` | Infisical secrets management (provisioned by default; unused if using ESO backend) | | `governance` | Platform Governance service | | `assets` | Platform Assets service | | `utils` | Platform Utilities service (+ pg\_cron) | | `prefect` | Prefect workflow orchestration (Data Governance) | | `datareadiness` | Data quality and metadata (Data Governance) | | `talk2data` | Natural language query engine (Data Insights) | ### Redis | Parameter | Specification | | --------------------- | ---------------------------------------------------------- | | **Version** | Redis 8.4.2 (self-managed) / Redis 7.0+ (managed services) | | **Memory** | 5 GB | | **High availability** | Multi-AZ replica (STANDARD\_HA) | | **Authentication** | Password-protected (AUTH required) | | **TLS** | Supported (`rediss://` scheme) | | **Eviction policy** | `noeviction` (reject writes when full) | ### Object Storage (S3-Compatible) | Parameter | Specification | | ------------------- | ------------------------------------------- | | **Protocol** | S3-compatible API (AWS S3, GCS HMAC, MinIO) | | **Access control** | Private (no public access) | | **Authentication** | Access key + secret key (HMAC) | | **Bucket name** | Configurable (default: `em-runtime`) | | **Max file upload** | 1,000 MB (configurable) | ## Networking ### Service Ports (Internal) | Service | Port | Protocol | | | --------------------- | ---- | -------------------------- | ---------------------------- | | em-runtime-governance | 8000 | HTTP | | | em-runtime-assets | 8000 | HTTP | | | em-runtime-utils | 8000 | HTTP | | | Keycloak | 8080 | HTTP | | | OpenFGA | 8080 | HTTP + gRPC (health: 8081) | | | Infisical | 8080 | HTTP | (if using Infisical backend) | | PostgreSQL | 5432 | TCP | | | Redis | 6379 | TCP | | ### External Ingress (HTTPRoute Rules) | Path | Backend | Port | Notes | | ----------------- | --------------------- | ---- | ------------------------- | | `/keycloak` | keycloak | 8080 | Pass-through (no rewrite) | | `/api/governance` | em-runtime-governance | 8000 | Rewrite to `/governance` | | `/api/assets` | em-runtime-assets | 8000 | Rewrite to `/assets` | | `/api/utils` | em-runtime-utils | 8000 | Rewrite to `/utils` | ### TLS Requirements | Boundary | Requirement | | ----------------------- | ---------------------------------- | | Client to load balancer | TLS 1.2+ (certificate required) | | Load balancer to pods | HTTP (in-cluster; TLS optional) | | Services to PostgreSQL | SSL recommended (private network) | | Services to Redis | TLS supported (`rediss://` scheme) | ### Firewall Rules Ensure the following traffic is allowed: * **Inbound:** HTTPS (443) * **From cluster:** PostgreSQL (5432), Redis (6379), S3 (443) * **LLM API egress:** OpenAI, Anthropic, Vertex AI endpoints (for Data Insights and Data Governance) ## Required Secrets Create these credentials before deployment: | Secret | Description | | ------------------------ | --------------------------------------- | | `POSTGRES_PASSWORD` | PostgreSQL superuser password | | `POSTGRES_HOST` | PostgreSQL hostname or IP | | `REDIS_PASSWORD` | Redis AUTH password | | `REDIS_HOST` | Redis hostname or IP | | `S3_ACCESS_KEY_ID` | Object storage access key | | `S3_SECRET_ACCESS_KEY` | Object storage secret key | | `S3_ENDPOINT_URL` | Object storage endpoint | | `S3_BUCKET_NAME` | Bucket name | | OCI registry credentials | Pull secret for `ghcr.io/emergenceai/*` | ## Required Tools | Tool | Version | Purpose | | --------- | ------- | -------------------------------------- | | `kubectl` | Latest | Kubernetes cluster management | | `helm` | 3.x | Helm chart installation | | `gh` | Latest | GitHub CLI for registry authentication | ```bash theme={null} # Authenticate GitHub CLI with required scopes gh auth login --scopes read:packages,repo,workflow ``` ## Pre-Deployment Checklist * Kubernetes cluster provisioned (1.28+, 3+ nodes, 4+ vCPU / 16+ GB RAM each) * Gateway API controller installed and operational * cert-manager installed * kubectl access configured with cluster-admin or equivalent * PostgreSQL 18 instance provisioned with multi-AZ * 9 databases created * `pg_cron` extension installed on `utils` database * Redis instance provisioned (5 GB, AUTH enabled, `noeviction` policy) * S3-compatible bucket created (private access) * HMAC access key and secret key provisioned * DNS record planned for platform hostname * TLS certificate available * All database and storage credentials available * Container registry credentials for `ghcr.io/emergenceai/*` * Helm 3.x and `gh` CLI installed and authenticated * Access to `oci://ghcr.io/emergenceai/charts` verified ## Next Steps Configure the Helm chart for your environment. # CRAFT Architecture Source: https://docs.emergence.ai/getting-started/architecture The 3-layer architecture of CRAFT: Solutions, Platform, and Infrastructure layers with repo mapping and dependency model. # Architecture CRAFT uses a **3-layer architecture**. Your domain-specific **Solutions** build on a shared **Platform**, which runs on managed **Infrastructure**. ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TD subgraph Solutions["Solutions Layer"] T2D["Data Insights"] DR["Data Governance"] SEMI["Semiconductor"] end subgraph Platform["Platform Layer (em-runtime)"] GOV["Governance
/api/governance
Auth, Orgs, Permissions"] ASS["Assets
/api/assets
Data Connections, Models"] UTL["Utils
/api/utils
Catalog, Scheduling, Memories"] end subgraph Infra["Infrastructure Layer"] K8S["Kubernetes"] PG["PostgreSQL"] RED["Redis"] KC["Keycloak"] FGA["OpenFGA"] end T2D --> GOV T2D --> ASS DR --> GOV DR --> ASS SEMI --> GOV GOV --> PG GOV --> KC GOV --> FGA ASS --> PG ASS --> RED UTL --> PG UTL --> RED style Solutions fill:#E69F00,stroke:#555555,color:#000 style Platform fill:#56B4E9,stroke:#555555,color:#000 style Infra fill:#0072B2,stroke:#555555,color:#fff ``` ## The Three Layers Domain-specific applications that solve customer problems. Each solution is a standalone product with its own data model, agents, and UI, but shares platform services for identity, authorization, secrets, and resource management. | Solution | Description | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Data Insights** | Multi-agent conversational data analytics using the A2A protocol. Users ask natural-language questions; agents generate SQL, execute queries, analyze results, and produce visualizations. | | **Data Governance** | Metadata enrichment, data profiling, DQ rule generation, and scorecard reporting. Orchestrated by Prefect workflows with LLM-powered enrichment. | | **Semiconductor** | Semiconductor fab analytics with AI-powered analysis agents and context graph. | Shared services consumed by all solutions. Provides identity, authorization, resource management, and a unified UI shell. | Service | Port | Responsibility | | -------------- | ---- | ------------------------------------------------------------------------------------------- | | **Governance** | 8000 | Organizations, projects, users, Keycloak multi-realm auth, OpenFGA ReBAC, Secrets API | | **Assets** | 8000 | Artifacts, data connections, files, models | | **Utils** | 8000 | Data catalog, search, scheduling, context packs, memories, operational utilities | | **Runtime UI** | -- | React microfrontend shell using Module Federation 2.0; hosts solution UIs as remote modules | All platform services share a common auth flow: solutions forward user JWT tokens to Governance for permission validation via auto-generated SDKs. Cloud deployment, networking, and GitOps automation. | Component | Tool | Responsibility | | ------------------------------- | --------- | --------------------------------------------------------------------- | | **Infrastructure Provisioning** | Terraform | Kubernetes clusters, networking, IAM, secrets backend | | **GitOps Delivery** | ArgoCD | Continuous deployment from Helm charts to 4 environments | | **Helm Charts** | Helm | Application chart templates with environment-specific values overlays | ## How Solutions Depend on the Platform Solutions consume platform services through two integration mechanisms: ### 1. Helm Subchart Dependency Each solution's Helm chart declares `em-runtime` as a subchart dependency. At deploy time, the platform services (Keycloak, OpenFGA, PostgreSQL, Redis) are co-deployed or connected via shared infrastructure values. This ensures every solution gets consistent identity, authorization, and secrets management without duplicating infrastructure. ### 2. SDK-Based Runtime Integration em-runtime auto-generates Python and TypeScript SDKs from its OpenAPI specs. Solutions import these SDKs to: * **Authenticate users** -- forward JWT tokens to Governance for permission validation * **Manage resources** -- CRUD operations on agents, artifacts, data connections via the Assets SDK * **Check permissions** -- OpenFGA relationship-based access control via the Governance SDK Solutions never implement their own permission checks. They delegate to the platform via SDK calls. If Governance is down, downstream services return 403. ## Open Standards and Portable Components CRAFT is built on open standards and portable, open-source components at every layer: ### Standards and Protocols | Layer | Standard | Purpose | | ----------------------- | ------------------------ | ---------------------------------------------------------------------- | | **Agent Communication** | A2A (JSON-RPC 2.0 / SSE) | Open protocol for inter-agent messaging | | **Tool Integration** | MCP | Open protocol for connecting agents to external tools and data sources | | **Identity** | OIDC / PKCE | Standards-based authentication | | **Observability** | OTLP | Vendor-neutral telemetry export to any compatible backend | ### Open-Source Components | Layer | Component | Purpose | | ------------------ | --------- | ------------------------------------------------------------------------------------ | | **Authorization** | OpenFGA | Relationship-based access control (open-source engine) | | **Object Storage** | obstore | Portable storage across GCS, Azure Blob, S3-compatible stores, and local filesystems | | **LLM Access** | LiteLLM | Provider-agnostic gateway -- use Gemini, GPT, Claude, or any supported model | Every protocol and component listed above is an open standard or open-source project. The platform does not require any proprietary service -- you can deploy on any infrastructure that supports these standards. ## Next Steps Learn about the core entities: organizations, projects, agents, and data connections. Deep dive into the em-runtime services that power the platform layer. Explore the multi-agent data analytics solution. Learn how to deploy the platform on Kubernetes. # CRAFT Core Concepts Source: https://docs.emergence.ai/getting-started/concepts Core concepts of CRAFT: organizations, projects, agents, data connections, schedules, and multi-tenancy. # Key Concepts This page introduces the foundational concepts you need to work with CRAFT. Understanding these concepts helps you build and operate any solution on the platform. ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% erDiagram Organization ||--o{ Project : "contains" Organization ||--o{ User : "has members" Project ||--o{ Agent : "registers" Project ||--o{ DataConnection : "configures" Project ||--o{ Schedule : "manages" Project ||--o{ ContextPack : "contains" Organization { string org_id PK string name string keycloak_realm } Project { string project_id PK string org_id FK string name } Agent { string agent_id PK string project_id FK string protocol } DataConnection { string connection_id PK string project_id FK string type } ``` ## Core Entities An **organization** is the top-level tenant boundary in CRAFT. Every resource in the platform belongs to exactly one organization. **Key characteristics:** * Each organization maps to a single **Keycloak realm** (the realm ID equals the org ID) * Organizations are created during the Governance service bootstrap process * All users, projects, and resources are scoped to an organization * Cross-organization data access is prevented at the database query level **Identity flow:** When you authenticate, your JWT token contains the `org_id` derived from the Keycloak realm. All subsequent API calls are scoped to that organization automatically. **Related:** [Organizations](/platform/organizations) | [Authentication](/platform/authentication) A **project** is a logical grouping of resources within an organization. Projects enable teams to organize agents, data connections, and other assets into separate workspaces. **Key characteristics:** * Projects belong to an organization and inherit its permission structure * The `project_id` is passed via the `X-Project-ID` HTTP header or as a path parameter -- it is never stored in the JWT * Users can have different roles across different projects within the same organization * All list queries filter by both `org_id` (from JWT) and `project_id` (from header) **Permission inheritance:** Projects inherit permissions from their parent organization via OpenFGA. A user who is an `admin` of the organization automatically has admin-level computed permissions on all projects within it. **Related:** [Projects](/platform/projects) | [Authorization](/platform/authorization) | [Authenticate Users (for solution developers)](/guides/solution-dev/authenticate-users) An **agent** is a registered AI service that can be discovered, invoked, and monitored through the platform. CRAFT supports three agent ecosystems through a unified registry: | Protocol | Description | Discovery | | -------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------ | | **A2A** (Agent-to-Agent) | Open protocol for inter-agent communication using JSON-RPC 2.0 over SSE | Agent Cards at `/.well-known/agent-card.json` | | **MCP** (Model Context Protocol) | Protocol for providing context to LLMs through tool definitions and resource access | MCP server metadata with tool/resource/prompt indexing | | **agentskills.io** | Portable skill format for packaging agent capabilities | Skill packages linked to Agent Card registrations | Agents are registered in the **Assets** service (port 8000) and can be searched across all three protocols with a single query. **Agent lifecycle stages:** EXPERIMENTAL, STABLE, DEPRECATED, RETIRED **Related:** [Agent Registry](/platform/agents) | [Data Insights Agents](/data-insights/overview) A **data connection** is a configured link to an external data source registered in the platform. Solutions use data connections to access customer data without managing credentials directly. **Key characteristics:** * Registered in the Assets service (port 8000) * Support databases, APIs, and file stores as data sources * Credentials are managed through the platform Secrets API (Infisical or ESO + GCP Secret Manager) * Scoped to an organization and project for multi-tenant isolation * Used by Data Insights for querying customer databases * Used by Data Governance for profiling and enrichment **Security:** Data connection credentials are never exposed in API responses. They are stored via the platform Secrets API and injected at runtime only when a solution needs to establish a connection. **Related:** [Data Connections](/platform/data-connections) | [Data Source Setup Guide](/guides/data-source-setup) A **schedule** defines recurring execution of tasks such as data profiling runs, agent health checks, or report generation. Schedules are managed by the **Utils** service (port 8000). **Key characteristics:** * Cron-based scheduling with timezone support * Scoped to an organization and project * Integrated with the platform's permission system (requires `can_execute` permission) * Execution history and status tracking **Related:** [Schedules](/platform/schedules) **Webhooks are planned for an upcoming release.** The webhook system will provide HTTP callbacks to notify external systems when platform events occur. Planned capabilities include HMAC-signed payloads, configurable event filters, automatic retry with exponential backoff, and SSRF protection. **Related:** [Webhooks (Planned)](/platform/webhooks) Multi-tenancy in CRAFT provides complete isolation between organizations while allowing flexible access control within each organization. **Isolation model:** * **Authentication isolation:** Each organization is a separate Keycloak realm with its own user directory, identity providers, and SSO configuration * **Authorization isolation:** OpenFGA stores per-organization relationship tuples; cross-organization access is structurally impossible * **Data isolation:** Each service owns its own database; all queries filter by `org_id` from the JWT token * **Secrets isolation:** Secret access is scoped per organization (Infisical namespaces or GCP SM IAM bindings) **Security invariants:** * The `org_id` always comes from the authenticated JWT, never from request parameters * List operations always filter by `org_id` AND `project_id` at the database level * Write operations always stamp the `org_id` from the JWT onto new resources * No cross-service foreign keys exist between databases **Related:** [Multi-Tenancy](/platform/multi-tenancy) | [Security](/security/overview) ## Permission Model The platform uses **Relationship-Based Access Control (ReBAC)** via OpenFGA. Permissions are computed from relationships rather than statically assigned. ### Roles | Role | Description | | ----------- | ------------------------------------------------------------------- | | `owner` | Full control over the organization and all its resources | | `admin` | Administrative access to manage users, projects, and settings | | `member` | Standard access to project resources | | `developer` | Access to create and modify agents, data connections, and workflows | | `operator` | Access to deploy, schedule, and monitor resources | | `viewer` | Read-only access to all resources | ### Computed Permissions Permissions are derived from role relationships at query time: | Permission | Description | | -------------------- | ------------------------------------------------------- | | `can_read` | View a resource and its metadata | | `can_write` | Create or update a resource | | `can_delete` | Remove a resource | | `can_execute` | Run an agent, trigger a schedule, or execute a workflow | | `can_manage_secrets` | Access and manage secrets for data connections | ### Inheritance Permissions flow downward through the hierarchy: ```text theme={null} Organization -> Project -> Resource (Agent, Data Connection, etc.) ``` A user with the `admin` role on an organization automatically receives `can_read`, `can_write`, and `can_delete` computed permissions on all projects and resources within that organization. ## CRAFT Modules CRAFT is one product, composed of three modules. Module names are stable nomenclature; capitalize as shown. | Module | Status | Purpose | | ----------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **CRAFT Assess** | Available | Evaluates data and surfaces what blocks agent-readiness. Implemented today by the Data Insights and Data Governance solutions (data profiling, coverage analysis, policy compliance checks). | | **CRAFT Enrich** | Available | Automatically enriches metadata, generates data quality rules, and classifies data assets. Implemented today by the Data Governance solution (LLM-powered metadata enrichment, DQ rule generation, classification). | | **CRAFT Toolkit** | Planned | Verification certificates and auto-formalization tools for developers extending the platform. Custom data connectors are not part of Toolkit; they are added on-demand to the platform's data-connections capability. | See [Introduction](/getting-started/introduction#how-craft-is-composed) for the user-facing framing. ## Glossary of Key Terms | Term | Definition | | -------------- | --------------------------------------------------------------------------------------------------- | | **A2A** | Agent-to-Agent. Open protocol for inter-agent communication (JSON-RPC 2.0 over SSE). | | **Agent Card** | A JSON manifest at `/.well-known/agent-card.json` describing an A2A agent's capabilities and skills | | **MCP** | Model Context Protocol -- a protocol for providing context to LLMs through tool definitions | | **ReBAC** | Relationship-Based Access Control -- fine-grained authorization based on entity relationships | | **OIDC** | OpenID Connect -- the identity layer used by Keycloak for authentication | | **PKCE** | Proof Key for Code Exchange -- OAuth 2.0 extension for secure browser-based auth flows | | **OTel** | OpenTelemetry -- vendor-neutral observability framework for traces, metrics, and logs | | **LGTM** | Loki, Grafana, Tempo, Mimir -- the Grafana observability stack | | **SSE** | Server-Sent Events -- unidirectional HTTP streaming used by the A2A protocol | ## Next Steps Explore the platform services that implement these concepts. Deep dive into OpenFGA and the ReBAC permission model. Learn about the unified agent registry across A2A, MCP, and agentskills.io. Understand the full security architecture including authentication, authorization, and secrets. # Introduction to CRAFT Source: https://docs.emergence.ai/getting-started/introduction What CRAFT is, the three modules that compose it, and how the platform fits into your stack. # Welcome to CRAFT CRAFT is an enterprise intelligence platform. Agents reason across your enterprise data to drive day-to-day operational decisions and long-horizon strategic ones, inside a verified envelope of constraints, policies, and proofs. What that looks like in practice: root-cause analysis that traces problems to their source, anomaly detection that catches what humans miss, scenario modeling that stress-tests decisions before they're made, and Verified Talk-to-Data that lets anyone in your organization query enterprise data in plain English and trust the answer. CRAFT deploys on **your infrastructure**, in your cloud, in your data center, or both. Every component runs on any CNCF-conformant Kubernetes cluster without cloud-specific dependencies. You retain full control over your data, policies, and execution. ## How CRAFT is composed CRAFT is one product, composed of three modules that target different stages of the path from raw data to verified action. Evaluates your data and surfaces what is blocking agent-readiness. Identifies gaps in quality, coverage, and policy compliance before autonomous agents are deployed. Automatically enriches metadata, generates data quality rules, and classifies data assets so agents can reason on them reliably. Planned. Verification certificates and auto-formalization tools for developers extending the platform. Custom data connectors are not part of Toolkit; they live in CRAFT data connections and are added on-demand. Built on Emergence's neuro-formal approach: mathematical proof is embedded into the architecture, so bounded autonomy is structural by design rather than a layered-on feature. ## What is CRAFT (technically)? Under the modules, CRAFT is a **3-layer architecture**: 1. **Solutions** are domain-specific applications (Data Insights, Data Governance, Semiconductor Analytics) that solve customer problems using AI agents, LLMs, and workflow orchestration. 2. **Platform** provides shared services for identity, authorization, secrets management, asset management, and a unified UI shell. Every solution inherits these capabilities automatically. 3. **Infrastructure** provides Kubernetes-based provisioning, GitOps delivery, and Helm-based deployment across four environments (dev, staging, demo, prod). The two views are orthogonal: Assess and Enrich are *what* you do with CRAFT; the 3-layer architecture is *how* CRAFT is built. ## Who is it for? Get AI-powered insights from your enterprise data without writing code. Ask questions in natural language, explore dashboards, and make data-driven decisions faster. Query your organization's data through conversational AI. Generate reports, discover trends, and share findings without technical skills. Ask natural-language questions against your databases and get instant SQL-generated answers, visualizations, and insights through multi-agent orchestration. Profile data assets, enrich metadata with LLMs, generate data quality rules, and produce scorecards through automated workflows. Deploy and operate the platform on any Kubernetes cluster with Helm charts, ArgoCD GitOps, and full observability via OpenTelemetry and the Grafana LGTM stack. ## Key Capabilities * **Multi-agent orchestration** using the A2A protocol (JSON-RPC 2.0 over SSE) * **Asset Management** for data connections, artifacts, files, and models * **Pipeline framework** for stateful, multi-step agent workflows with cooperative cancellation * **Provider-agnostic LLM integration** via LiteLLM (Gemini, GPT, Claude, and more) * **Verified Talk-to-Data** with schema-aware natural-language to SQL, validation, and execution * **Data profiling and enrichment** powered by LLMs and orchestrated by Prefect workflows * **Data quality rules and scorecards** with automated generation and tracking * **Conversational analytics** with multi-turn context and visualization support * **Multi-tenant authentication** with OIDC/PKCE and single sign-on support * **Fine-grained authorization** with computed permissions across organizations and projects * **Secrets management** for cloud and self-hosted deployments. See [Secrets Management](/security/secrets-management) * **Multi-tenancy** with organization and project-level isolation * **Helm charts** with subchart dependency model for consistent deployments * **ArgoCD** GitOps across dev, staging, demo, and production environments * **OpenTelemetry** traces, metrics, and logs with the Grafana LGTM stack * **Auto-generated SDKs** (Python and TypeScript) from OpenAPI specs ## Platform at a Glance | Layer | Components | Key Technologies | | ------------------ | --------------------------------------------- | ----------------------------------------------------- | | **Solutions** | Data Insights, Data Governance, Semiconductor | FastAPI, A2A Protocol, LiteLLM, Prefect | | **Platform** | Governance, Assets, Utils, Runtime UI | Keycloak, OpenFGA, Secrets API, Module Federation 2.0 | | **Infrastructure** | Infrastructure Provisioning, GitOps, Helm | Terraform, ArgoCD, Kubernetes, Helm | ## Design Principles Deploy on any infrastructure you choose: your cloud, your data center, or both. The core platform components are open-source and self-hostable, and cloud-managed services are optional rather than required. For example, Redis Streams handles event streaming instead of cloud-specific message brokers, and `obstore` provides object storage across GCS, Azure Blob, S3-compatible stores, and local filesystems instead of provider-specific SDKs. The platform already includes Keycloak, OpenFGA, PostgreSQL, and Redis, plus optional secrets backends. Before adding a new dependency, the team first evaluates whether an existing one can serve the need. Every addition is justified against the cost of operational complexity. All design and implementation decisions follow these principles rigorously. Simple, focused solutions are preferred over over-engineered abstractions. ## Next Steps Understand the 3-layer architecture and how solutions depend on the platform. Learn about organizations, projects, agents, data connections, and multi-tenancy. Build, package, and deploy a new solution on CRAFT — start with the 30-minute quickstart. Explore the services that power the platform layer. Deploy CRAFT on any Kubernetes cluster, cloud-managed or self-hosted. # A2A Protocol Primer Source: https://docs.emergence.ai/guides/agent-author/a2a-protocol-primer Agent Cards, JSON-RPC methods, SSE streaming, and the task lifecycle — everything you need to understand agent communication on CRAFT. # A2A Protocol Primer Google's A2A protocol (now under the Linux Foundation) is the communication standard that connects all agents on CRAFT. Understanding A2A is prerequisite knowledge for every agent author, regardless of which framework you choose — it defines the contract your agent must honour at its network boundary. ## What A2A Is A2A is an open, vendor-neutral protocol for agent-to-agent communication. Key properties: * **Transport**: JSON-RPC 2.0 over HTTP, with Server-Sent Events (SSE) for streaming responses * **Discovery**: Agent Cards served at a well-known URL describe capabilities * **Framework-agnostic**: works with Google ADK, Pydantic AI, LangGraph, or any custom implementation * **Task-centric**: every interaction creates a Task with a defined lifecycle A2A defines the *protocol*, not the implementation. CRAFT agents are free to use any framework internally; they must speak A2A at their boundary. CRAFT's reference agents currently serve `protocolVersion: "0.3.0"` (verified against the live text2sql agent's `/.well-known/agent-card.json`). The [A2A specification](https://a2a-protocol.org/latest/specification/) reached **v1.0** in early 2026 — the platform's `a2a-sdk` Python library (`pip install a2a-sdk`) will track upstream as the SDK catches up. The `a2a-sdk` provides the canonical types and server utilities (`A2AStarletteApplication`, `DefaultRequestHandler`, `AgentCard`, etc.). ## Agent Cards — Identity and Discovery An Agent Card is a JSON document that describes what an agent does, what it can accept, and how to reach it. Every CRAFT agent publishes its card at: ``` GET /.well-known/agent-card.json ``` This endpoint is **unauthenticated** — any orchestrator can fetch it to understand the agent's capabilities before invoking it. ### Agent Card Structure ```json theme={null} { "name": "Insights Agent", "description": "Main analytics agent — plans queries, coordinates sub-agents, analyzes data.", "url": "https://", "version": "1.4.0", "protocolVersion": "0.3.0", "provider": { "organization": "EmergenceAI", "url": "https://emergence.ai" }, "capabilities": { "streaming": true, "pushNotifications": false, "stateTransitionHistory": false }, "skills": [ { "id": "data_query", "name": "Data Query", "description": "Query databases using natural language", "tags": ["sql", "query", "database"], "examples": [ "Show me Q4 revenue by region", "What are the top 10 customers by sales?" ] }, { "id": "data_analysis", "name": "Data Analysis", "description": "Analyze data, generate charts, and produce insights", "tags": ["analysis", "insights", "charts"] } ], "defaultInputModes": ["text", "data", "file"], "defaultOutputModes": ["text", "data", "file"], "securitySchemes": { "bearer": { "type": "http", "scheme": "bearer" } }, "security": [{ "bearer": [] }] } ``` ### Capabilities | Capability | Effect when `true` | | ------------------------ | -------------------------------------------------------------------------- | | `streaming` | Agent supports `message/stream` (SSE); callers may use real-time streaming | | `pushNotifications` | Agent supports webhook push for task completion | | `stateTransitionHistory` | Agent returns full task state transition history in responses | Clients should read capabilities from the Agent Card before calling methods that depend on them. Calling `message/stream` against an agent with `streaming: false` returns `UnsupportedOperationError`. ### Skills Skills are discrete capability declarations within an Agent Card. They help orchestrators understand what tasks the agent can handle and inform LLM routing decisions. ```python theme={null} # Pydantic AI / a2a SDK — define skills in code from a2a.types import AgentSkill skill = AgentSkill( id="data_analysis", name="Data Analysis", description="Performs data analysis tasks related to semiconductors.", tags=["semiconductor", "data analysis"], examples=["analyze wafer data", "predict defect rates"], ) ``` **Claude Agent SDK and LangGraph** also require explicit skill authoring — add `AgentSkill` objects to your `AgentCard` regardless of framework: ```python theme={null} # Claude Agent SDK / LangGraph — same AgentSkill definition from a2a.types import AgentSkill skill = AgentSkill( id="enterprise-analysis", name="Enterprise Data Analysis", description="Answers business questions by querying governed data sources and generating SQL.", tags=["analytics", "sql", "enterprise"], examples=["What is our MRR by region?", "Show top 10 customers by revenue last quarter"], ) ``` **Google ADK and `to_a2a()`**: ADK's `to_a2a()` constructs a minimal Agent Card from the agent's `name`, `description`, and `instruction` — but does not populate `AgentSkill` definitions. For production CRAFT registrations, author explicit `AgentSkill` objects even when using ADK, to give orchestrators rich capability descriptions for routing decisions. **Claude Agent SDK — Wrapping as an A2A Agent:** The Claude Anthropic SDK does not natively produce A2A `AgentSkill` definitions. Wrap a Claude call inside a standard A2A server: ```python theme={null} from a2a.types import AgentSkill import anthropic client = anthropic.AsyncAnthropic() skill = AgentSkill( id="data-analysis", name="Data Analysis", description="Analyzes enterprise data using Claude and CRAFT's data catalog.", tags=["analytics", "sql", "enterprise"], examples=["What is our MRR by region?", "Identify anomalies in yesterday's orders"], ) # The A2A server calls the Claude API when tasks arrive — see streaming-idioms.mdx # for the full TaskUpdater pattern async def handle_task(message: str) -> str: response = await client.messages.create( model="claude-opus-4-8", max_tokens=1024, messages=[{"role": "user", "content": message}], ) return response.content[0].text ``` **LangGraph — Wrapping as an A2A Agent:** ```python theme={null} from a2a.types import AgentSkill from langgraph.prebuilt import create_react_agent skill = AgentSkill( id="graph-workflow", name="Graph Workflow", description="Executes multi-step analysis workflows using LangGraph state machines.", tags=["workflow", "stateful", "multi-step"], ) # Build the LangGraph agent; expose via A2AStarletteApplication (see streaming-idioms.mdx # for the TaskUpdater + astream integration pattern) graph = create_react_agent("anthropic:claude-opus-4-8", tools=[...]) ``` ## JSON-RPC Methods The A2A protocol exposes these JSON-RPC methods on your agent's root endpoint (`POST /`): | Method | Use Case | Response | | ------------------- | -------------------------------------------------- | ----------------------------------- | | `message/send` | Fire-and-forget, await a single complete response | JSON-RPC response with Task | | `message/stream` | Real-time SSE stream of status and artifact events | SSE event stream | | `tasks/get` | Poll for current task state | Task object | | `tasks/cancel` | Cancel an in-progress task | Acknowledgment | | `tasks/resubscribe` | Reconnect to a dropped SSE stream | SSE event stream from current state | Most CRAFT agents use `message/stream` exclusively — it is the preferred method for all production interactions because it delivers progressive status updates to the user. ## Message Structure Every request to a CRAFT agent is a **Message** containing one or more **Parts**: ```json theme={null} { "jsonrpc": "2.0", "method": "message/stream", "params": { "message": { "role": "user", "messageId": "", "contextId": "", "parts": [ { "kind": "text", "text": "Show me Q4 revenue by region" }, { "kind": "data", "data": { "type": "datasource", "resource_uri": "data:acme-corp:ml-project:analytics-db", "datasource_type": "database", "datasource_name": "Analytics Database", "selected_schemas": [ { "schema_name": "public", "schema_fqn": "analytics-db.analytics_db.public" } ] } } ] }, "contextId": "" }, "id": "req-1" } ``` The A2A spec uses camelCase for protocol fields (`messageId`, `contextId`, `taskId`). The platform's `a2a-sdk` server also accepts snake\_case input (`message_id`, `context_id`) and normalizes, but responses are always camelCase — match the spec to keep request/response symmetric. ### Part Types | Part Type | `kind` value | Use Case | | ---------- | ------------ | ---------------------------------------------------------------------------- | | `TextPart` | `"text"` | Natural language input or agent response | | `DataPart` | `"data"` | Structured JSON — datasource context, artifact references, selection context | | `FilePart` | `"file"` | Binary content referenced by URI (`asset://artifacts/chart.png`) | The `context_id` groups messages into a conversation session. CRAFT agents use this to retrieve conversation history from the task store and maintain coherent multi-turn dialogue. ## Task Lifecycle Every `message/send` or `message/stream` call creates or resumes a **Task**. Tasks follow a strict state machine: ``` submitted ──► working ──► completed ──► failed ──► input-required ──► canceled ──► rejected ``` Terminal states (`completed`, `failed`, `canceled`, `rejected`) are final — a task in a terminal state cannot be restarted. To continue a conversation, the client sends a new message with the same `context_id`. ## SSE Streaming — Event Sequence When a client calls `message/stream`, the server responds with `Content-Type: text/event-stream` and pushes events as the agent works. Each `data:` line is a complete JSON-RPC 2.0 response object. ### Event Types | Result Type | When Used | | ------------------------- | ---------------------------------------------------- | | `Task` | First event — the created task (state = `submitted`) | | `TaskStatusUpdateEvent` | Lifecycle state changes and intermediate messages | | `TaskArtifactUpdateEvent` | Completed artifacts (results, charts, files) | ### Protocol Rules 1. The first event MUST be a `Task` or a `Message` 2. Intermediate events are `TaskStatusUpdateEvent` (state=`working`) and `TaskArtifactUpdateEvent` 3. The final event MUST be `TaskStatusUpdateEvent` with `final: true` and a terminal state 4. The SSE connection closes after the final event ### Wire Example — Text2SQL Stream The following is a real-world event sequence from the CRAFT text2sql agent: ``` data: {"kind":"status-update","task_id":"t1","final":false, "status":{"state":"working","message":{"role":"agent", "parts":[{"kind":"text","text":"Analyzing database schema"}]}}} data: {"kind":"status-update","task_id":"t1","final":false, "status":{"state":"working","message":{"role":"agent", "parts":[{"kind":"text","text":"Running query"}]}}} data: {"kind":"artifact-update","task_id":"t1", "artifact":{"artifact_id":"abc","name":"query_summary", "parts":[{"kind":"text","text":"Top 10 customers by revenue"}]}} data: {"kind":"artifact-update","task_id":"t1","last_chunk":true, "artifact":{"artifact_id":"def","name":"query_results", "parts":[{"kind":"data","data":{"type":"artifact", "uri":"data:acme:proj:results-parquet", "resource_type":"parquet", "metadata":{"row_count":10,"columns":["customer_name","total_revenue"]}}}]}} data: {"kind":"status-update","task_id":"t1","final":true, "status":{"state":"completed"}} ``` ### Chunked Artifact Streaming Large artifacts can be delivered in chunks using the `append` and `lastChunk` flags: ```json theme={null} {"artifactUpdate": {"artifact": {"artifactId": "a1", "parts": [...chunk1...], "append": false, "lastChunk": false}}} {"artifactUpdate": {"artifact": {"artifactId": "a1", "parts": [...chunk2...], "append": true, "lastChunk": false}}} {"artifactUpdate": {"artifact": {"artifactId": "a1", "parts": [...chunk3...], "append": true, "lastChunk": true}}} ``` ## Agent-to-Agent Communication Flow The following sequence diagram shows how an orchestrator delegates a task to a sub-agent using the A2A protocol. ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% sequenceDiagram participant UI as UI / Client participant Orch as Orchestrator Agent participant Reg as Assets Registry participant Sub as Sub-Agent (e.g. text2sql) UI->>Orch: POST / message/stream note over Orch: Task submitted Orch->>Reg: GET /assets/agents (resolve sub-agent URL) Reg-->>Orch: agent_card.json (url, capabilities, skills) Orch->>Sub: POST / message/stream (A2A JSON-RPC) Sub-->>Orch: SSE: Task{state=submitted} Sub-->>Orch: SSE: StatusUpdate{state=working, "Analyzing schema"} Sub-->>Orch: SSE: ArtifactUpdate{name=query_results, last_chunk=true} Sub-->>Orch: SSE: StatusUpdate{state=completed, final=true} Orch-->>UI: SSE: StatusUpdate{state=working, "Running analysis"} Orch-->>UI: SSE: ArtifactUpdate{chart + summary} Orch-->>UI: SSE: StatusUpdate{state=completed, final=true} ``` ## Authentication Agent Cards declare supported authentication schemes in `securitySchemes`. CRAFT agents use JWT bearer tokens — the orchestrator injects them when calling sub-agents. ```json theme={null} { "securitySchemes": { "bearer": { "type": "http", "scheme": "bearer", "bearerFormat": "JWT" } }, "security": [{ "bearer": [] }] } ``` Agents validate the token, extract the `user_id` and `org_id`, and use them to scope all database queries and audit logs. A small `UserContextBuilder` that reads the `Authorization` header and exposes a typed context object to your dependencies is the canonical pattern for Pydantic AI + `a2a` server integrations. ## Reconnection If an SSE connection drops mid-stream, the client can reconnect using `tasks/resubscribe`: ```json theme={null} { "jsonrpc": "2.0", "method": "tasks/resubscribe", "params": { "taskId": "task-123" } } ``` The server resumes the event stream from the task's current state. Your agent implementation does not need to handle this explicitly — `A2AStarletteApplication` and `DefaultRequestHandler` manage reconnection automatically. ## Next Steps Build your first agent using Google ADK, Pydantic AI, or another supported framework. Orchestration, delegation, and fan-out across multiple A2A agents. # Debugging Agents Source: https://docs.emergence.ai/guides/agent-author/debugging-agents Trace inspection, prompt iteration, tool-call analysis, and symptom-keyed troubleshooting for CRAFT agents. # Debugging Agents Agent failures are rarely obvious from error messages alone. This page covers how to inspect traces, analyse tool calls, iterate on prompts, and diagnose the most common failure modes. ## Toolbox Before debugging, confirm you have access to these tools: | Tool | Purpose | | ----------------------------------------------- | ---------------------------------------------------------------------------- | | **Langfuse** | Trace inspection, span hierarchy, token usage, LLM input/output | | **ADK Web** (`adk web`) | Interactive ADK agent debug UI — replay conversations, inspect tool calls | | **curl / jq** | Direct A2A JSON-RPC invocation for isolated testing | | **Agent Card** (`/.well-known/agent-card.json`) | Verify capabilities and skills are declared correctly | | **Application logs** | Structured JSON logs with `task_id`, `context_id`, `user_id` for correlation | ## Reading a Trace in Langfuse Every agent request produces a trace in Langfuse. The span hierarchy for a Pydantic AI agent looks like: ``` agent.execute # outer span (task_id, context_id, user_id) agent.llm_stream [attempt=0] # LLM call span mcp.tool.get_schema # MCP tool span mcp.tool.execute_sql # MCP tool span mcp.tool.upload_artifact # MCP tool span ``` Key attributes to check: * `agent.response_length` — if 0, the agent produced no output (likely an error) * `mcp.tool.status` — `success` or error classification * `mcp.tool.result_size_bytes` — large values (>50KB) indicate context bloat risk * `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` — token budget ## Symptom Index **Most likely causes:** 1. **Task validation failure** — the request is missing `task_id`, `context_id`, or authenticated user context. Check application logs for `ValueError: Task ID and Context ID must be provided`. 2. **Authentication error** — the JWT token is missing or invalid. If your executor validates the `Authorization` header before processing the task, a missing or invalid token raises before emitting any events. 3. **MCP connection failure** — the `StreamableHttpTransport` cannot connect to the MCP server. Check that `MCP_SERVER_URL` is set and the MCP pod is healthy. **Diagnostics:** ```bash theme={null} # Substitute :8003 with whatever port your A2A agent binds locally — # each agent listens on its own port (unlike em-runtime services, which # all bind container port 8000 and are fronted by the gateway). For the # reference em-talk2data agents these are :8001 text2sql, :8002 insights, # :8003 coding, :8080 talk2data-service. # # Check agent card — if this fails, the agent isn't running curl http://localhost:8003/.well-known/agent-card.json # Send a minimal test request curl -X POST http://localhost:8003/ \ -H "Content-Type: application/json" \ -H "x-user-id: test-user" \ -d '{ "jsonrpc":"2.0","method":"message/send", "params":{"message":{"role":"user","message_id":"m1","context_id":"c1", "parts":[{"kind":"text","text":"hello"}]}}, "id":"1" }' # Check agent logs kubectl logs -l app=my-agent --tail=100 | grep ERROR ``` **Most likely causes:** 1. **MCP server returning errors** — the tool call reaches the MCP server but returns a structured error. Check `mcp.tool.status` in the trace span. 2. **Iteration limit hit** — the agent has exceeded `max_code_failures`. Look for log line: `Iteration limit reached for task {task_id}`. 3. **Forbidden operation** — the tool call uses a restricted pattern (e.g., filesystem access, blocked import). Look for `LINT_ERROR` in the tool result. 4. **Tool schema mismatch** — the LLM is passing incorrect argument types. Check the `mcp.tool.param_fingerprint` across calls — if it's consistent and always failing, the tool schema is wrong. **Diagnostics:** ```python theme={null} # Test the tool directly, bypassing the agent from fastmcp import Client from fastmcp.client.transports import StreamableHttpTransport async def test_tool(): client = Client(transport=StreamableHttpTransport("https://craft.emergence.ai/mcp", headers={"Authorization": "Bearer ", "X-Project-ID": ""})) async with client: result = await client.call_tool( name="get_schema", arguments={"table_name": "orders", "schema_fqn": "db.schema.public"}, ) print(result) ``` **Symptoms:** Agent responses become shorter, less accurate, or the LLM refuses to call tools. Token usage approaches the model's context limit. **Most likely causes:** 1. **Tool results too large** — a tool is returning large payloads (DataFrames, Plotly figures) directly to the LLM. Your toolset should strip large fields and store them in the Assets API, passing only the resource URI. 2. **Conversation history too long** — the task store is loading the full conversation history. Check `agent.context_metrics.history_messages` in the trace. 3. **System prompt too large** — the instruction builder is including too much context. Check `agent.context_metrics.instruction_length`. **Diagnostics:** ```bash theme={null} # Check token usage in Langfuse — look for input_tokens near model limit # Gemini Flash: 1M tokens; Gemini Pro: 2M tokens; Claude Sonnet: 200K tokens # Check tool result sizes in the trace # mcp.tool.result_size_bytes > 50000 is a warning sign ``` **Fix:** Implement side payload interception in your toolset. Strip large data blobs (DataFrames, images, Plotly figures) before they reach the LLM context window; store them in the Assets API and pass only the resource URI. **Symptoms:** A single agent request consumes 10x the expected tokens. The LLM is looping on tool calls or generating excessively long responses. **Most likely causes:** 1. **Missing iteration limit** — no `max_code_failures` guard on code execution tool. The LLM keeps trying different code variations. 2. **Tool always returning errors** — the LLM keeps retrying a broken tool. Check `mcp.tool.status` across the trace — all `error` with the same tool name is a signal. 3. **Infinite delegation loop** — two agents are delegating to each other. Check the orchestrator's `sub_agents` list for circular references. 4. **Large system prompt being rebuilt per turn** — the instruction builder is fetching context on every LLM round-trip. Check `agent.context_metrics.instruction_build_duration_s`. **Diagnostics:** ```bash theme={null} # Count tool calls per task from Langfuse # agent.tool_calls_per_request > 20 is suspicious for most agents ``` **Fix:** ```python theme={null} # Add hard iteration limits if ctx.deps._code_failure_count >= settings.max_code_failures: return { "success": False, "error": "Execution limit reached. Ask the user for clarification.", } # Set max_tokens to bound output cost model_settings = ModelSettings( temperature=0, max_tokens=4096, # bound output ) ``` **Symptoms:** `state=failed` with message "No datasource DataPart found" or "resource\_uri resolution failed". **Most likely causes:** 1. **Missing DataPart in the message** — the caller did not include a `DataPart` with `type: "datasource"`. The A2A message must include both a `TextPart` and a `DataPart`. 2. **Wrong resource\_uri format** — the `resource_uri` must be in the full four-segment format: `data:{org_id}:{project_id}:{name}`. The simplified format (`data:my-db`) is not accepted. 3. **Missing selected\_schemas** — `selected_schemas` is empty or absent. Text2SQL requires exactly one schema entry. **Diagnostics:** ```bash theme={null} # Inspect the incoming A2A message for DataParts kubectl logs -l app= | grep "datasource" # Test with a complete message curl -X POST https:/// \ -H "Content-Type: application/json" \ -d '{ "jsonrpc":"2.0","method":"message/send", "params":{"message":{"role":"user","message_id":"m1","context_id":"c1","parts":[ {"kind":"text","text":"count rows in orders"}, {"kind":"data","data":{ "type":"datasource", "resource_uri":"data:acme:proj:analytics-db", "datasource_type":"database", "datasource_name":"Analytics DB", "selected_schemas":[{"schema_name":"public","schema_fqn":"db.db.public"}] }} ]}},"id":"1" }' ``` ## Prompt Iteration The fastest way to improve agent quality is iterating on the system prompt. Use ADK Web or direct A2A calls to test prompt changes without redeploying. ### ADK Web — Interactive Replay (Google ADK) ```bash theme={null} # Start your agent in development mode uvicorn my_agent.agent:a2a_app --host 0.0.0.0 --port 8003 --reload # Open the ADK Web UI in a second terminal adk web packages/my_agent # Navigate to http://localhost:8000 and replay test conversations ``` ADK Web shows each tool call, its arguments, and the LLM's reasoning before and after. Use it to observe exactly how the prompt influences routing and tool selection. ### Claude Agent SDK — Prompt Replay For Claude-based agents, replay prompt variations using the Anthropic SDK directly without running the full A2A server: ```python theme={null} import anthropic client = anthropic.Anthropic() response = client.messages.create( model="claude-opus-4-8", max_tokens=1024, system="[paste your agent's system prompt here]", messages=[{"role": "user", "content": "[failing input]"}], tools=[...], # paste your tool definitions ) print(response.content) ``` Trace inspection: look for `anthropic.messages.create` spans in Langfuse. Tool use blocks appear as child spans with `input` and `output` fields. ### LangGraph — Debug with `debug=True` LangGraph's `astream` supports verbose debug output and LangSmith/Langfuse tracing: ```python theme={null} # Enable verbose node-level output during development async for step in graph.astream( {"messages": [{"role": "user", "content": "..."}]}, stream_mode="updates", debug=True, ): for node_name, output in step.items(): print(f"[{node_name}]", output) ``` For production tracing, pass the `langfuse_handler` callback (see [Eval Harness](/guides/agent-author/eval-harness)) — spans appear as `langgraph:node:` entries in Langfuse. ### Minimal Repro with Direct curl For non-ADK agents, replay failing conversations directly: ```bash theme={null} # Capture a failing trace from Langfuse (input field) # Replay it directly against the agent curl -X POST http://localhost:8003/ \ -H "Content-Type: application/json" \ -H "x-user-id: debug-user" \ -d @failing_trace_input.json | python3 -m json.tool ``` ### Prompt Change Checklist Before changing the system prompt: 1. Identify the specific behaviour to change (use a Langfuse trace as evidence) 2. Write a test case that captures the failure 3. Make the minimum prompt change needed to fix the test case 4. Run the full regression suite to check for new regressions 5. Re-check token usage — prompt changes can inflate or deflate input token cost ## Structured Logging for Correlation All CRAFT agents log structured JSON with `task_id` and `context_id`. Use these to correlate logs with Langfuse traces: ```python theme={null} logger.info( "Tool call completed", extra={ "tool_name": name, "task_id": task_id, "context_id": context_id, "duration_s": duration, "status": "success", }, ) ``` ```bash theme={null} # Find the Langfuse trace for the same task # Search by task_id in Langfuse UI filter: trace.metadata.task_id = "task-uuid-here" ``` ## Next Steps Build regression suites to catch issues before they reach production. Configure Langfuse tracing for your agent. # Eval Harness Source: https://docs.emergence.ai/guides/agent-author/eval-harness Langfuse evaluators, regression suites, and golden traces for CRAFT agents. # Eval Harness Agent quality degrades silently. A prompt change, a model upgrade, or a new tool can alter behaviour across hundreds of evaluation dimensions without surfacing as an exception. A structured eval harness catches regressions before they reach users. CRAFT uses Langfuse for observability and evaluation. Before setting up your eval harness, complete the [Langfuse Setup Guide](/guides/langfuse-setup) to connect your agent to your Langfuse project. ## What to Evaluate For each agent, define evaluations in three categories: | Category | Description | Example | | -------------------------- | ----------------------------------------------------------- | ----------------------------------------------------------------------------- | | **Functional correctness** | Does the agent produce the right output for a given input? | SQL matches expected query, chart type matches request | | **Tool-call accuracy** | Does the agent call the right tools with correct arguments? | Calls `get_schema` before `execute_sql`; passes correct `schema_fqn` | | **Conversation behaviour** | Does the agent handle edge cases gracefully? | Asks for clarification when datasource is missing; does not loop indefinitely | ## Setting Up Langfuse Tracing All agent frameworks on CRAFT support Langfuse tracing via the OTEL exporter. ### Pydantic AI Configure Langfuse via the standard OTEL exporter. Add this at application startup: ```python theme={null} from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter provider = TracerProvider() provider.add_span_processor( BatchSpanProcessor( OTLPSpanExporter( endpoint=f"https://{LANGFUSE_HOST}/api/public/otel/v1/traces", headers={"Authorization": f"Basic {langfuse_basic_auth}"}, ) ) ) trace.set_tracer_provider(provider) ``` Pydantic AI emits OTEL spans automatically when a `TracerProvider` is configured. Every agent run, tool call, and MCP request appears as a structured span. ### Google ADK Configure Langfuse via the standard OTEL exporter: ```python theme={null} from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter provider = TracerProvider() provider.add_span_processor( BatchSpanProcessor( OTLPSpanExporter( endpoint=f"https://{LANGFUSE_HOST}/api/public/otel/v1/traces", headers={"Authorization": f"Basic {langfuse_basic_auth}"}, ) ) ) trace.set_tracer_provider(provider) ``` ADK emits OTEL spans for each agent turn, tool invocation, and LLM call. Look for `google.adk.agent` root spans in Langfuse. ### Claude Agent SDK The Anthropic SDK emits OTEL spans automatically when a `TracerProvider` is configured — the same setup as Pydantic AI and ADK works without additional configuration: ```python theme={null} from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from anthropic import Anthropic provider = TracerProvider() provider.add_span_processor( BatchSpanProcessor( OTLPSpanExporter( endpoint=f"https://{LANGFUSE_HOST}/api/public/otel/v1/traces", headers={"Authorization": f"Basic {langfuse_basic_auth}"}, ) ) ) trace.set_tracer_provider(provider) # Anthropic SDK auto-emits spans (anthropic.messages.create root) when TracerProvider is set client = Anthropic() ``` Look for `anthropic.messages.create` root spans in Langfuse, with `tool_use` child spans for each tool invocation. ### LangGraph LangGraph integrates with Langfuse via the LangChain callback handler (requires `langfuse>=3.0`): ```python theme={null} # Requires: pip install langfuse>=3.0 from langfuse.langchain import CallbackHandler # v3 import path (NOT langfuse.callback) from langgraph.prebuilt import create_react_agent langfuse_handler = CallbackHandler() # reads LANGFUSE_PUBLIC_KEY/SECRET_KEY from env graph = create_react_agent("anthropic:claude-opus-4-8", tools=[...]) # Pass handler per-invocation result = await graph.ainvoke( {"messages": [{"role": "user", "content": "..."}]}, config={"callbacks": [langfuse_handler]}, ) ``` Spans appear in Langfuse as `langgraph:node:` entries. Each node execution, LLM call, and tool invocation is captured separately. ## Golden Traces A golden trace is a recorded agent execution that represents the correct behaviour for a given input. Store golden traces in your test fixtures and replay them to catch regressions. ### Recording a Golden Trace ```python theme={null} import json from langfuse import Langfuse langfuse = Langfuse() # Fetch a specific trace by ID from Langfuse trace = langfuse.fetch_trace("trace-id-from-langfuse-ui") # Serialize to a fixture file golden = { "input": trace.input, "expected_output": trace.output, "tool_calls": [ {"name": span.name, "input": span.input, "output": span.output} for span in trace.observations if span.type == "SPAN" and span.name.startswith("mcp.tool.") ], } with open("tests/fixtures/golden_revenue_query.json", "w") as f: json.dump(golden, f, indent=2) ``` ### Replaying Golden Traces ```python theme={null} import pytest import json from my_agent.executor import MyAgentExecutor from tests.harness import MockEventQueue, MockRequestContext @pytest.mark.parametrize("fixture_path", [ "tests/fixtures/golden_revenue_query.json", "tests/fixtures/golden_top_customers.json", "tests/fixtures/golden_missing_datasource.json", ]) async def test_golden_trace(fixture_path): with open(fixture_path) as f: golden = json.load(f) executor = MyAgentExecutor(...) event_queue = MockEventQueue() context = MockRequestContext(user_input=golden["input"]) await executor.execute(context, event_queue) # Check final output final_event = event_queue.get_final_event() assert golden["expected_output"] in final_event.text # Check tool calls actual_tool_calls = event_queue.get_tool_calls() for expected in golden["tool_calls"]: assert any( tc["name"] == expected["name"] for tc in actual_tool_calls ), f"Expected tool call '{expected['name']}' not found" ``` ## Langfuse Evaluators Langfuse evaluators run scoring functions against traces. CRAFT agents use two types: ### LLM-as-Judge Evaluators Use a lightweight LLM to score agent outputs on dimensions like correctness, helpfulness, and groundedness: ```python theme={null} from langfuse import Langfuse from langfuse.model import ModelUsage langfuse = Langfuse() def create_sql_correctness_evaluator(): """Create a Langfuse evaluator that scores SQL correctness.""" return langfuse.create_score_config( name="sql_correctness", data_type="NUMERIC", min_value=0, max_value=1, ) def score_sql_output(trace_id: str, generated_sql: str, expected_sql: str): """Score a trace using an LLM judge.""" import anthropic client = anthropic.Anthropic() response = client.messages.create( model="claude-haiku-4-5", max_tokens=256, messages=[{ "role": "user", "content": f"""Score the SQL query correctness from 0 to 1. Expected SQL: {expected_sql} Generated SQL: {generated_sql} Return only a JSON object: {{"score": 0.95, "reason": "..."}}""" }], ) result = json.loads(response.content[0].text) langfuse.score( trace_id=trace_id, name="sql_correctness", value=result["score"], comment=result["reason"], ) ``` ### Deterministic Evaluators For structured outputs where correctness can be computed without an LLM: ```python theme={null} def score_tool_call_order(trace_id: str, tool_calls: list[dict]): """Score whether the agent called tools in the correct order.""" names = [tc["name"] for tc in tool_calls] # get_schema must precede execute_sql if "execute_sql" in names and "get_schema" in names: schema_idx = names.index("get_schema") sql_idx = names.index("execute_sql") score = 1.0 if schema_idx < sql_idx else 0.0 elif "execute_sql" in names: score = 0.0 # called execute_sql without schema check else: score = 1.0 # no SQL — correct for non-SQL queries langfuse.score( trace_id=trace_id, name="tool_call_order", value=score, ) ``` ## Regression Suite Structure Organise your test fixtures by scenario type. For the text2sql agent, a well-structured regression suite includes: ```text theme={null} tests/ ├── fixtures/ │ ├── happy_path/ │ │ ├── simple_count.json │ │ ├── revenue_by_quarter.json │ │ └── top_customers.json │ ├── edge_cases/ │ │ ├── missing_datasource.json # Agent should return state=failed │ │ ├── unsupported_dialect.json │ │ └── empty_result_set.json │ └── cancellation/ │ └── cancel_mid_execution.json ├── test_golden_traces.py ├── test_tool_call_order.py └── harness.py # MockEventQueue, MockRequestContext ``` ## Running Evals in CI Add eval runs to your CI pipeline to catch regressions on pull requests: ```yaml theme={null} # .github/workflows/eval.yml name: Agent Eval on: pull_request: paths: - "packages/my_agent/**" jobs: eval: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Run regression suite env: LANGFUSE_SECRET_KEY: ${{ secrets.LANGFUSE_SECRET_KEY }} LANGFUSE_PUBLIC_KEY: ${{ secrets.LANGFUSE_PUBLIC_KEY }} GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }} run: | uv run pytest tests/test_golden_traces.py -v \ --tb=short \ -x # fail fast on first regression ``` ## Cost-Aware Evaluation Track token usage per eval run to catch cost regressions alongside quality regressions: ```python theme={null} # After each eval run, log usage to Langfuse langfuse.score( trace_id=trace_id, name="input_tokens", value=usage.input_tokens, data_type="NUMERIC", ) langfuse.score( trace_id=trace_id, name="output_tokens", value=usage.output_tokens, data_type="NUMERIC", ) langfuse.score( trace_id=trace_id, name="tool_call_count", value=toolset.tool_call_count, data_type="NUMERIC", ) ``` Set budget alerts in Langfuse if token usage per task exceeds a threshold — this catches prompt regressions that inflate input context or cause tool call loops. ## Next Steps Inspect traces and diagnose failures when evals surface regressions. # Multi-Agent Patterns Source: https://docs.emergence.ai/guides/agent-author/multi-agent-patterns Delegation, supervision, and parallel fan-out across CRAFT agents using the A2A protocol. # Multi-Agent Patterns CRAFT's multi-agent coordination happens at the protocol layer through A2A. Agents discover each other via the registry, delegate work via JSON-RPC, and stream results via SSE — without sharing process memory or tight framework coupling. This page covers the three main coordination patterns used in production CRAFT deployments. ## The CRAFT Multi-Agent Architecture CRAFT's analytics solution is a good reference for multi-agent patterns. A typical layout chains three specialized agents: 1. **Orchestrator** — receives user requests, plans the workflow, delegates to sub-agents 2. **Text2SQL** — converts natural language to SQL, executes against the target database, returns query results as Parquet artifacts 3. **Insights** — analyses query results, generates charts, produces natural-language insights and visualizations Each agent is an independently deployed microservice. They communicate exclusively through the A2A protocol. ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR UI["User / UI"] -->|"REST + SSE"| Facade["Analytics Service\n(REST facade)"] Facade -->|"A2A message/stream"| Orch["Orchestrator Agent\n(Google ADK)"] Orch -->|"A2A delegate"| T2S["Text2SQL Agent\n(ADK)"] Orch -->|"A2A delegate"| Ins["Insights Agent\n(ADK)"] Orch -->|"A2A delegate"| Future["Future Agents\n(any framework)"] T2S -->|"SSE: sql_plan, query_results"| Orch Ins -->|"SSE: chart, summary"| Orch T2S --- DB[("Database")] Ins --- Artifacts[("Artifact Store")] classDef entry fill:#56B4E9,stroke:#555555,color:#000 classDef process fill:#E69F00,stroke:#555555,color:#000 classDef infra fill:#0072B2,stroke:#555555,color:#fff classDef external fill:#CC79A7,stroke:#555555,color:#000 classDef data fill:#009E73,stroke:#555555,color:#000 class UI entry class Facade process class Orch,T2S,Ins process class Future external class DB,Artifacts,Artifacts data ``` ## Pattern 1: Sequential Delegation The orchestrator delegates tasks to sub-agents one at a time, using the output of one as the input to the next. This is the most common pattern on CRAFT. ### Google ADK — `RemoteA2aAgent` Sub-Agents ADK makes sequential delegation transparent: `RemoteA2aAgent` instances look like local sub-agents. The orchestrator LLM routes to them via the `transfer_to_agent` tool. Resolve sub-agent URLs from the CRAFT Assets API rather than hardcoding hostnames. ```python theme={null} from google.adk.agents.remote_a2a_agent import RemoteA2aAgent from google.adk.agents.llm_agent import Agent # Resolve agent card URLs from the Assets API at startup text2sql = RemoteA2aAgent( name="text2sql_agent", description="Converts natural language to SQL queries and executes them.", agent_card="https:///.well-known/agent-card.json", ) insights = RemoteA2aAgent( name="insights_agent", description="Analyses query results, generates charts, and produces insights.", agent_card="https:///.well-known/agent-card.json", ) orchestrator = Agent( model="gemini-3.5-flash", name="orchestrator", instruction=""" You are a data analysis orchestrator. When the user asks a data question: 1. Delegate to text2sql_agent to retrieve the data 2. Delegate to insights_agent to analyse and visualise it 3. Synthesize the results for the user Always pass the datasource context (resource_uri, schema) to text2sql_agent. """, sub_agents=[text2sql, insights], ) ``` ### Claude Agent SDK — Sequential Delegation ```python theme={null} import httpx import anthropic from a2a.client import A2AClient from a2a.types import SendMessageRequest, MessageSendParams, Message, Part, TextPart, Role from uuid import uuid4 claude = anthropic.Anthropic() async def claude_orchestrator(user_question: str, sub_agent_url: str, context_id: str) -> str: async with httpx.AsyncClient() as http: a2a = A2AClient(http, url=sub_agent_url) # Delegate to sub-agent via A2A request = SendMessageRequest( id=str(uuid4()), params=MessageSendParams( message=Message( role=Role.user, message_id=str(uuid4()), context_id=context_id, parts=[Part(root=TextPart(text=user_question))] ) ) ) result = await a2a.send_message(request) sub_result = "" if result.root.result and result.root.result.artifacts: for artifact in result.root.result.artifacts: for part in artifact.parts: if hasattr(part.root, "text"): sub_result += part.root.text # Claude synthesises the final response response = claude.messages.create( model="claude-opus-4-8", max_tokens=1024, messages=[{ "role": "user", "content": f"Original question: {user_question}\nSub-agent result: {sub_result}\n\nSynthesize a final answer." }], ) return response.content[0].text ``` ### LangGraph — Sequential Delegation ```python theme={null} from langgraph.graph import StateGraph, END from typing import TypedDict, Annotated import operator import httpx from a2a.client import A2AClient from a2a.types import SendMessageRequest, MessageSendParams, Message, Part, TextPart, Role from uuid import uuid4 class OrchestratorState(TypedDict): question: str context_id: str sub_agent_result: str final_answer: str async def delegate_to_sub_agent(state: OrchestratorState) -> OrchestratorState: async with httpx.AsyncClient() as http: a2a = A2AClient(http, url="https://") request = SendMessageRequest( id=str(uuid4()), params=MessageSendParams( message=Message( role=Role.user, message_id=str(uuid4()), context_id=state["context_id"], parts=[Part(root=TextPart(text=state["question"]))] ) ) ) result = await a2a.send_message(request) sub_text = "" if result.root.result and result.root.result.artifacts: for artifact in result.root.result.artifacts: for part in artifact.parts: if hasattr(part.root, "text"): sub_text += part.root.text return {"sub_agent_result": sub_text} builder = StateGraph(OrchestratorState) builder.add_node("delegate", delegate_to_sub_agent) builder.set_entry_point("delegate") builder.add_edge("delegate", END) graph = builder.compile() ``` ### How ADK Delegation Works When the orchestrator LLM decides to delegate: 1. LLM emits a `transfer_to_agent` tool call with the target agent name and message 2. ADK resolves the remote agent's card (lazy, cached after first fetch) 3. ADK constructs an A2A `message/stream` JSON-RPC request from the session context 4. Sends via `httpx.AsyncClient` to the sub-agent's URL 5. Processes the SSE response stream back into ADK internal events 6. Returns the final artifact to the orchestrator as if it were a local sub-agent result ### Passing Context Between Agents The Text2SQL agent requires a `DataPart` with the datasource context. The orchestrator must forward this from the original user message: ```json theme={null} { "method": "message/stream", "params": { "message": { "parts": [ { "kind": "text", "text": "Show me top 10 customers by revenue" }, { "kind": "data", "data": { "type": "datasource", "resource_uri": "data:acme-corp:ml-project:analytics-db", "datasource_type": "database", "datasource_name": "Analytics Database", "selected_schemas": [ { "schema_name": "public", "schema_fqn": "analytics-db.analytics_db.public" } ] } } ] } } } ``` The `resource_uri` must be in the full four-segment format: `data:{org_id}:{project_id}:{name}`. The simplified format (`data:my-db`) will fail — the Assets API requires all four segments for connection resolution. ## Pattern 2: Parallel Fan-Out Multiple sub-agents are invoked concurrently and their results are merged. Use this when sub-tasks are independent and latency matters. ### Manual Fan-Out with asyncio For frameworks other than ADK (or for orchestrators that need explicit concurrency control): ```python theme={null} import asyncio import httpx from a2a.client import A2AClient async def fan_out_query( context_id: str, user_message: str, agent_urls: list[str], ) -> list[dict]: """Invoke multiple agents in parallel and collect their results.""" async def invoke_agent(url: str) -> dict: async with httpx.AsyncClient() as http: a2a = A2AClient(http, url=url) # Full shape: see a2a-protocol-primer.mdx for the SendMessageRequest + Message/Part structure # Simplified for readability — replace with: # request = SendMessageRequest(id=str(uuid4()), params=MessageSendParams( # message=Message(role=Role.user, message_id=str(uuid4()), # context_id=context_id, parts=[Part(root=TextPart(text=user_message))]))) result = await a2a.send_message( message=user_message, context_id=context_id, ) return result tasks = [invoke_agent(url) for url in agent_urls] results = await asyncio.gather(*tasks, return_exceptions=True) # Filter out failures — fan-out should be resilient to partial failures return [r for r in results if not isinstance(r, Exception)] ``` ### ADK Fan-Out via Parallel Sub-Agents Google ADK supports parallel sub-agent invocation when the orchestrator instruction explicitly requests concurrent delegation: ```python theme={null} orchestrator = Agent( model="gemini-3.5-flash", name="orchestrator", instruction=""" You can delegate to multiple agents simultaneously when tasks are independent. For data analysis requests that need both SQL results and market context, invoke text2sql_agent and context_agent in parallel, then synthesize results. """, sub_agents=[text2sql, context_agent], ) ``` ADK's `RemoteA2aAgent` A2A support is currently **experimental**. Parallel invocation of multiple `RemoteA2aAgent` instances in the same turn may produce duplicate events (tracked in [ADK issue #3207](https://github.com/google/adk-python/issues/3207)). Test thoroughly before using in production. ## Pattern 3: Supervision and Retry An orchestrator supervises sub-agent results and retries failed tasks with corrected inputs. ```python theme={null} orchestrator = Agent( model="gemini-3.5-flash", name="orchestrator", instruction=""" When text2sql_agent returns a failed task: 1. Inspect the error message 2. If it's a schema error, call text2sql_agent again with corrected schema context 3. If it's a permission error, ask the user to check their database access 4. Maximum 2 retries before reporting failure to the user Never retry more than twice — report the issue instead of looping indefinitely. """, sub_agents=[text2sql], ) ``` ### Detecting Sub-Agent Failures A sub-agent that fails emits a `TaskStatusUpdateEvent` with `state=failed` and an error message. ADK surfaces this as an error event in the orchestrator's event stream. The orchestrator LLM sees the failure message in its context and can decide to retry or escalate. ## Registering Sub-Agents with the Orchestrator When adding a new agent to the platform, you must: 1. **Register it with the Assets API** — gives it a `resource_uri` and makes it discoverable 2. **Add it as a `RemoteA2aAgent` to the orchestrator** (if using Google ADK) — and redeploy 3. **Update the orchestrator instruction** — explain what the new agent does and when to use it ```python theme={null} # Add to orchestrator's sub_agents list new_agent = RemoteA2aAgent( name="my_new_agent", description="Describe what this agent does and when the orchestrator should use it.", agent_card="https:///.well-known/agent-card.json", ) orchestrator = Agent( ... sub_agents=[text2sql, insights, new_agent], # add here ) ``` The `description` field on `RemoteA2aAgent` is what the orchestrator LLM reads to decide routing. Write it as a capability statement: "Use this agent when the user needs X, Y, or Z." ## Carrying Artifacts Across Agent Boundaries Artifacts (Parquet files, charts, SQL queries) produced by sub-agents are stored in the Assets API and referenced by `resource_uri`. When the orchestrator passes results from one agent to another, it passes the `resource_uri` — not the raw data. ```python theme={null} # Text2SQL produces query_results as an ArtifactFilePart { "type": "artifact", "uri": "data:acme-corp:ml-project:query-results-uuid", "resource_type": "parquet", "metadata": {"row_count": 100, "columns": ["customer", "revenue"]} } # Orchestrator passes this URI to Insights Agent as a DataPart { "kind": "data", "data": { "type": "artifact_context", "uri": "data:acme-corp:ml-project:query-results-uuid", "resource_type": "parquet" } } ``` ## Context Propagation Every A2A message must include a `context_id` — the conversation session identifier. Sub-agents use this to look up conversation history from the task store. In a multi-agent architecture, the orchestrator's `context_id` flows through to each sub-agent call. This ensures that sub-agents can retrieve prior turns when needed for multi-step workflows. ```python theme={null} # ADK handles context_id propagation automatically via RemoteA2aAgent. # For manual A2A calls, always forward the parent context_id: response = await a2a_client.send_message( message=user_message, context_id=parent_context_id, # always forward ) ``` ## Next Steps SSE streaming, partial results, and cooperative cancellation. Review the A2A wire format and task lifecycle. # Agent Author Guide Source: https://docs.emergence.ai/guides/agent-author/overview How to build, register, and operate A2A-compatible agents on the CRAFT platform. # Agent Author Guide CRAFT is a multi-agent platform built on the A2A protocol. Every capability in CRAFT — from natural-language SQL generation to semiconductor yield analysis — runs as a discrete, independently deployable agent. **The short version:** CRAFT gives you managed LLMs, tool discovery, and an agent registry. Your agent gets all three by pointing its LLM client at the CRAFT gateway and making one API call to register. Start with the [tutorial](/guides/agent-author/your-first-agent) if you want to be running in under 20 minutes. This overview covers what it means to be an agent on CRAFT, when to build one, how to choose a framework, and how the platform integrates with your agent at runtime. ## What is an agent on CRAFT? A CRAFT agent is a service that: 1. Exposes an **Agent Card** at `/.well-known/agent-card.json` describing its identity, skills, and capabilities 2. Accepts work via **A2A JSON-RPC** (`message/send` or `message/stream`) 3. Streams progress and results back as **SSE events** 4. Registers with the **Assets API** so other agents and the orchestrator can discover and invoke it Agents are microservices first. They can be authored in any language, deployed anywhere, and call any tools — as long as they speak the A2A protocol at their boundary. ## When should you build an agent? * You have a bounded capability with a clear input/output contract (e.g., "turn natural language into SQL", "generate yield analysis charts") * The capability needs to be independently scalable or deployable * The capability will be invoked by other agents or orchestrators * You want the capability to appear in the platform's agent registry and be discoverable by the orchestrator * The capability is owned by a separate team with its own release cadence * The capability is a simple function call used only within a single agent — add it as a tool instead * The capability is a data transformation pipeline with no LLM reasoning — use a standard service * You need real-time collaboration between multiple LLMs sharing memory — use sub-agents within the same ADK process * The capability is a UI-layer concern — use CRAFT's REST API directly ## Framework choice CRAFT supports four agent frameworks. Choose based on your team's constraints and the nature of your agent. | Framework | Language | Best For | Pros | Cons | | -------------------- | ------------------ | ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | | **Google ADK** | Python | Analytics, multi-agent orchestration, data pipelines | Native A2A (`to_a2a()`), `RemoteA2aAgent` for sub-agent delegation, model-agnostic via LiteLLM, ADK Web for debugging | A2A support still experimental; Python-only | | **Claude Agent SDK** | Python, TypeScript | Reasoning-heavy tasks, document analysis, complex multi-step workflows | Extended thinking, strong tool-use, multi-turn conversation handling | No native A2A server — wrap with `a2a` library manually | | **Pydantic AI** | Python | Type-safe agents, data-structured outputs, MCP toolsets | FastMCPToolset for MCP integration, strict typing, clean dependency injection via `deps_type` | Less mature ecosystem than ADK | | **LangGraph** | Python | Stateful workflows, graph-based logic, conditional branching | Explicit state machine, fine-grained control over execution flow | More boilerplate; no native A2A — wrap with `a2a` library | **CrewAI and role-based frameworks are not supported on CRAFT.** Multi-agent coordination on CRAFT happens at the protocol layer through A2A — agents discover each other via the registry and communicate via JSON-RPC. Role-based crew frameworks duplicate this coordination mechanism and create tight coupling that breaks independent deployability. Do not use them. ### Framework decision flowchart ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TD Start["New agent on CRAFT"] --> Q1{"Primary need?"} Q1 -->|"Multi-agent orchestration\nor data analytics"| Q2{"Team already uses\nGoogle ADK?"} Q1 -->|"Reasoning-heavy,\nlong documents"| Claude["Claude Agent SDK"] Q1 -->|"Strict typed outputs,\nMCP toolsets"| Pydantic["Pydantic AI"] Q1 -->|"Complex stateful\nworkflow / branching"| LangGraph["LangGraph"] Q2 -->|"Yes or no preference"| ADK["Google ADK"] Q2 -->|"No — already on\nPydantic AI"| Pydantic ADK --> A2A_native["to_a2a() + RemoteA2aAgent\n— A2A built in"] Claude --> A2A_wrap["Wrap with a2a library\n+ A2AStarletteApplication"] Pydantic --> A2A_wrap LangGraph --> A2A_wrap A2A_native --> Register["Register in Assets API"] A2A_wrap --> Register classDef entry fill:#56B4E9,stroke:#555555,color:#000 classDef decision fill:#F0E442,stroke:#555555,color:#000 classDef framework fill:#E69F00,stroke:#555555,color:#000 classDef infra fill:#0072B2,stroke:#555555,color:#fff classDef terminal fill:#009E73,stroke:#555555,color:#000 class Start entry class Q1,Q2 decision class ADK,Claude,Pydantic,LangGraph framework class A2A_native,A2A_wrap infra class Register terminal ``` ## Platform integration points Every agent on CRAFT integrates with three platform services at runtime. ### Assets API — agent registry The Assets API is the source of truth for agent discovery. Before your agent can be invoked by an orchestrator, you must register it: ```bash theme={null} curl -X POST "https:///api/assets/agents" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " \ -H "Content-Type: application/json" \ -d '{ "agent_card": { ... }, "tags": ["analytics"], "visibility": "TENANT_ONLY" }' ``` The registry stores the Agent Card, assigns a `resource_uri` (`agent:::`), and makes the agent discoverable to orchestrators and other platform consumers. ### LiteLLM gateway — model access All LLM calls from agents on CRAFT route through the shared LiteLLM gateway. This provides: * **Unified model access** — use Gemini, Claude, GPT-5.5, and others with a single OpenAI-compatible endpoint * **Cost tracking** — per-org, per-project token usage recorded automatically * **Rate limiting** — enforced per tenant to prevent runaway costs * **Model aliasing** — `LLM_PRO`, `LLM_LIGHT` constants resolve to the current best model for each tier Configure your agent to point at the gateway: ```python theme={null} # Google ADK — via the LiteLlm wrapper pointed at the gateway from google.adk.agents import Agent from google.adk.models.lite_llm import LiteLlm import os root_agent = Agent( name="my_agent", # The "openai/" prefix tells LiteLLM to use the OpenAI-compatible endpoint; # the actual model string is whatever the gateway routes (gemini-3.5-flash here). model=LiteLlm( model="openai/gemini-3.5-flash", api_base=os.environ["CRAFT_GATEWAY_URL"], api_key=os.environ["CRAFT_TOKEN"], ), ) # Pydantic AI — point the OpenAI-compatible provider at the gateway from pydantic_ai import Agent from pydantic_ai.models.openai import OpenAIChatModel from pydantic_ai.providers.openai import OpenAIProvider model = OpenAIChatModel( "gemini-3.5-flash", provider=OpenAIProvider( base_url=os.environ["CRAFT_GATEWAY_URL"], api_key=os.environ["CRAFT_TOKEN"], ), ) agent = Agent(model, system_prompt="...") ``` See [Your First Agent › Step 1](/guides/agent-author/your-first-agent) for the same pattern across all four supported frameworks (ADK, Claude SDK / OpenAI client, Pydantic AI, LangGraph). ### em-runtime-mcp — tool access CRAFT agents access data and platform capabilities through the `em-runtime-mcp` MCP server. Tools available include database schema inspection, query execution, artifact upload/download, and context graph queries. ```python theme={null} # Pydantic AI — FastMCPToolset connects to em-runtime-mcp from pydantic_ai.toolsets.fastmcp import FastMCPToolset from fastmcp import Client from fastmcp.client.transports import StreamableHttpTransport mcp_client = Client( transport=StreamableHttpTransport(settings.mcp_server_url), ) toolset = FastMCPToolset(mcp_client) ``` Create a fresh MCP client and toolset per agent request. Do not reuse a single long-lived client across requests — MCP sessions can expire or the MCP server pod can restart. ## Guide contents Agent Cards, JSON-RPC methods, SSE streaming, and the task lifecycle. Framework-by-framework tutorial: Google ADK, Claude Agent SDK, Pydantic AI, LangGraph. Function tools, MCP tools via FastMCPToolset, schema discipline. Delegation, supervision, and parallel fan-out via A2A. SSE streaming, partial results, cooperative cancellation. Langfuse evaluators, regression suites, golden traces. Trace inspection, prompt iteration, common failure modes. # Streaming Idioms Source: https://docs.emergence.ai/guides/agent-author/streaming-idioms SSE streaming, partial results, cooperative cancellation, and reconnection patterns for CRAFT agents. # Streaming Idioms All CRAFT agents stream their responses via Server-Sent Events (SSE). This page covers the practical patterns for producing well-formed SSE streams, handling partial results, and implementing cooperative cancellation. **A2A SSE vs. MCP Streamable HTTP**: This page covers A2A streaming — the JSON-RPC events that flow between agents via Server-Sent Events (SSE). This is different from the MCP Streamable HTTP transport used to connect to the CRAFT tool gateway. A2A's `message/stream` uses SSE; MCP tool connections use the MCP Streamable HTTP transport (e.g., ADK's `StreamableHTTPConnectionParams`, FastMCP's `StreamableHttpTransport`, the MCP Python SDK's `streamablehttp_client`, or LangGraph's `MultiServerMCPClient` with `transport="http"`). Both A2A SSE and MCP Streamable HTTP are current standards — they serve different protocol layers. ## The Streaming Contract The A2A protocol requires every `message/stream` response to follow a strict event ordering: 1. First event: `Task` (state = `submitted`) 2. Zero or more: `TaskStatusUpdateEvent` (state = `working`) with intermediate progress messages 3. Zero or more: `TaskArtifactUpdateEvent` with result artifacts 4. Final event: `TaskStatusUpdateEvent` with `final: true` and a terminal state (`completed`, `failed`, `canceled`) Every event is a JSON-RPC 2.0 response object sent as a `data:` line in the SSE stream. Never close the SSE connection without emitting a final event with `final: true`. Clients that receive an unexpected connection close will attempt to resubscribe via `tasks/resubscribe`, and if the task store has no record of the task, they will error. Always emit a final status event before the connection closes, even on error. ## Streaming Text Output ### Pydantic AI — `run_stream` with `stream_text()` The following pattern demonstrates streaming text output from a Pydantic AI agent. Each text chunk is forwarded as a `TaskStatusUpdateEvent` with `state=working`: ```python theme={null} response_message = updater.new_agent_message( parts=[Part(root=TextPart(text=""))] ) async with self.agent.run_stream( user_message, deps=agent_deps, toolsets=[toolset], ) as run: async for text_content in run.stream_text(): # Update the message in place — each chunk replaces the previous response_message.parts[0].root = TextPart(text=text_content) await updater.update_status( TaskState.working, message=response_message, ) # Get the final complete output and emit the terminal working event result = await run.get_output() response_message.parts[0].root = TextPart(text=result) await updater.update_status( TaskState.working, message=response_message, final=True, metadata={"timestamp": datetime.now(timezone.utc).isoformat()}, ) await updater.complete() ``` `stream_text()` returns the **accumulated** text at each yield, not a delta. Each chunk is the full response so far. This means the client always has a coherent partial response it can display immediately, without needing to concatenate chunks. ### Google ADK — SSE via `to_a2a()` When using `to_a2a()`, ADK handles all SSE event emission automatically. Your agent code just runs normally: ```python theme={null} root_agent = Agent( model="gemini-3.5-flash", name="my_agent", instruction="...", tools=[my_tool], ) # to_a2a() wraps the agent with SSE streaming support. # Intermediate LLM output appears as TaskStatusUpdateEvent (state=working). # Final output appears as TaskArtifactUpdateEvent. a2a_app = to_a2a(root_agent, port=8003) ``` ADK's `to_a2a()` uses `TaskStatusUpdateEvent` with `state=working` for streaming intermediate text chunks, then emits a `TaskArtifactUpdateEvent` for the final text. This differs from the Pydantic AI pattern (which uses only status events). Both are valid A2A; clients must handle both patterns. ### Claude Agent SDK — Streaming via A2A The Claude Agent SDK doesn't wrap natively in A2A. Build a thin A2A executor that streams from the Anthropic API and emits the correct A2A events: ```python theme={null} import anthropic from uuid import uuid4 from a2a.server.tasks import TaskUpdater from a2a.types import TaskState, Part, TextPart async def claude_a2a_handler(task_updater: TaskUpdater, user_message: str): client = anthropic.AsyncAnthropic() # AsyncAnthropic for use inside async def await task_updater.update_status( TaskState.working, message=task_updater.new_agent_message([Part(root=TextPart(text="Starting..."))]), final=False, ) full_response = "" async with client.messages.stream( model="claude-opus-4-8", max_tokens=1024, messages=[{"role": "user", "content": user_message}], ) as stream: async for text_delta in stream.text_stream: full_response += text_delta # Throttle: emit status every ~200 chars if len(full_response) % 200 < len(text_delta): await task_updater.update_status( TaskState.working, message=task_updater.new_agent_message( [Part(root=TextPart(text=full_response[:120] + "..."))] ), final=False, ) await task_updater.add_artifact( parts=[Part(root=TextPart(text=full_response))], artifact_id=str(uuid4()), name="response", last_chunk=True, ) await task_updater.update_status( TaskState.completed, message=task_updater.new_agent_message([Part(root=TextPart(text=full_response))]), final=True, ) ``` ### LangGraph — Streaming via A2A Use `stream_mode="messages"` which yields `(chunk, metadata)` tuples for token-level streaming. Use `stream_mode="updates"` for node-completion events instead. ```python theme={null} from uuid import uuid4 from a2a.server.tasks import TaskUpdater from a2a.types import TaskState, Part, TextPart async def langgraph_a2a_handler(task_updater: TaskUpdater, graph, user_message: str): await task_updater.update_status( TaskState.working, message=task_updater.new_agent_message([Part(root=TextPart(text="Running graph..."))]), final=False, ) full_output = "" # stream_mode="messages" yields (AIMessageChunk, metadata) tuples async for chunk, metadata in graph.astream( {"messages": [{"role": "user", "content": user_message}]}, stream_mode="messages", ): if hasattr(chunk, "content") and chunk.content: full_output += chunk.content if len(full_output) % 200 < len(chunk.content): await task_updater.update_status( TaskState.working, message=task_updater.new_agent_message( [Part(root=TextPart(text=full_output[:120] + "..."))] ), final=False, ) await task_updater.add_artifact( parts=[Part(root=TextPart(text=full_output))], artifact_id=str(uuid4()), name="result", last_chunk=True, ) await task_updater.update_status( TaskState.completed, message=task_updater.new_agent_message([Part(root=TextPart(text=full_output))]), final=True, ) ``` For graphs without a `messages` state key, switch to `stream_mode="updates"` and iterate `for node_name, payload in chunk.items():` to detect completed nodes. ## Streaming Intermediate Status Messages Use intermediate status messages to keep users informed during long-running tasks. Emit an intent acknowledgment before the main agent loop: ```python theme={null} # Emit a brief acknowledgment before the agent starts processing. # This confirms to the user that the request was received. ack_message = updater.new_agent_message( parts=[Part(root=TextPart(text="Analysing your request..."))] ) await updater.update_status( TaskState.working, message=ack_message, final=False ) # Then run the main agent loop... ``` For long database queries, emit named step messages: ```python theme={null} for step_name, step_fn in pipeline_steps: await updater.update_status( TaskState.working, message=updater.new_agent_message( [Part(root=TextPart(text=f"Step: {step_name}"))] ), final=False, ) await step_fn() ``` ## Artifact Streaming Artifacts (charts, data files, analysis results) are emitted as `TaskArtifactUpdateEvent`. For multi-artifact agents (like a text-to-SQL agent that produces intermediate SQL plans followed by query results), artifacts are emitted in a specific order: 1. `sql_plan` — intermediate SQL planning artifact (TextPart, JSON) 2. `query_summary` — human-readable summary of results (TextPart) 3. `sql_query` — the generated SQL (DataPart with artifact URI) 4. `query_results` — Parquet results file (DataPart with artifact URI, `last_chunk=true`) ```python theme={null} # Emit artifacts in order using the task updater await updater.add_artifact( parts=[TextPart(text=summary_text)], artifact_id=str(uuid4()), name="query_summary", ) await updater.add_artifact( parts=[DataPart(data={ "type": "artifact", "uri": parquet_resource_uri, "resource_type": "parquet", "metadata": {"row_count": row_count, "columns": column_names}, })], artifact_id=str(uuid4()), name="query_results", last_chunk=True, ) ``` ## Cooperative Cancellation CRAFT supports cooperative cancellation: a client sends `tasks/cancel`, and the agent should stop its current work and emit a `canceled` terminal event. ### Pydantic AI — asyncio.CancelledError Implement cancellation via asyncio task cancellation. The `cancel()` method updates the task state in the database before the running asyncio task receives the cancellation signal: ```python theme={null} class MyAgentExecutor(AgentExecutor): async def cancel( self, context: RequestContext, event_queue: EventQueue ) -> None: task_id = context.task_id # Write canceled state to DB immediately so other replicas see it. # The active asyncio task will receive CancelledError on its next await. await self.database.task_repository.update_state(task_id, "CANCELED") updater = TaskUpdater(event_queue, task_id, context.context_id) cancel_message = updater.new_agent_message( parts=[Part(root=TextPart(text="Response stopped."))], ) await updater.update_status( TaskState.canceled, message=cancel_message, final=True, ) async def execute(self, context, event_queue): try: # ... main agent loop ... except asyncio.CancelledError: # CancelledError propagates from run_stream() when the task is cancelled. # Don't emit any more events — the cancel() method already did. logger.info("Agent execution canceled") ``` ### Cross-Instance Cancellation In Kubernetes with multiple agent replicas, the cancel request may hit a different pod than the one running the task. Use a background polling watcher to detect cross-instance cancellation: ```python theme={null} async def _cancellation_watcher( self, task_id: str, parent_task: asyncio.Task ) -> None: """Poll DB every 5s for cancellation from another instance.""" while True: await asyncio.sleep(5) state = await self.database.task_repository.get_state(task_id) if state == "CANCELED": parent_task.cancel() return ``` Start the watcher at the beginning of `execute()` and cancel it in the `finally` block: ```python theme={null} cancel_watcher = asyncio.create_task( self._cancellation_watcher(task_id, asyncio.current_task()) ) try: # ... agent loop ... finally: cancel_watcher.cancel() ``` ## Reconnection Handling If an SSE connection drops mid-stream, the client sends a `tasks/resubscribe` request. The `DefaultRequestHandler` from the `a2a` library handles this automatically when used with a persistent task store. ```python theme={null} from a2a.server.tasks import InMemoryTaskStore # for development # Use a persistent task store for production — survives pod restarts task_store = PersistentTaskStore(database) # your database-backed implementation handler = DefaultRequestHandler( agent_executor=executor, task_store=task_store, ) ``` `InMemoryTaskStore` does not survive pod restarts. In production, implement a database-backed `TaskStore` (e.g., backed by PostgreSQL). If the task store loses state, clients that attempt `tasks/resubscribe` will receive a `TaskNotFoundError`. ## LLM Error Retry Transient LLM errors (HTTP 5xx, 429) should be retried with exponential backoff. A well-behaved executor retries up to 2 times: ```python theme={null} MAX_LLM_RETRIES = 2 LLM_RETRY_BACKOFF_BASE = 2.0 # seconds: 2, 4 for attempt in range(MAX_LLM_RETRIES + 1): try: async with self.agent.run_stream(...) as run: async for text in run.stream_text(): ... break # success — exit retry loop except ModelHTTPError as exc: if exc.status_code == 429 or exc.status_code >= 500: if attempt < MAX_LLM_RETRIES: delay = LLM_RETRY_BACKOFF_BASE * (2 ** attempt) await asyncio.sleep(delay) else: await self._handle_agent_failure(exc, response_message, updater) raise else: # 4xx non-429: not retryable await self._handle_agent_failure(exc, response_message, updater) raise ``` ## First-Token Latency Track time-to-first-token to detect model cold starts and MCP connection overhead: ```python theme={null} stream_start = time.monotonic() first_token_recorded = False async for text_content in run.stream_text(): if not first_token_recorded: first_token_latency = time.monotonic() - stream_start metrics.record_stream_first_token(first_token_latency) first_token_recorded = True # ... stream the chunk ``` Typical first-token latencies on CRAFT: 500ms–2s for Gemini Flash, 1s–4s for Gemini Pro with cold MCP connections. ## Next Steps Inspect streaming traces and diagnose SSE issues. Set up Langfuse to evaluate streaming agent behaviour. # Tool Authoring Source: https://docs.emergence.ai/guides/agent-author/tool-authoring How to write function tools, connect MCP tools via FastMCPToolset, and apply schema discipline across frameworks. # Tool Authoring Tools are the primary way agents access external capabilities — databases, APIs, file storage, computation environments. Getting tools right is critical: poorly specified tools are the leading cause of agent errors, context bloat, and cost overruns. ## Function Tools Function tools are Python functions that the LLM can call directly. Every supported framework converts them to the appropriate wire format for the underlying model. ### Writing a Good Tool A well-authored function tool has four properties: 1. **A descriptive docstring** — the LLM reads this to decide when to call the tool 2. **Typed parameters** — every parameter has an explicit type annotation and docstring description 3. **A bounded return type** — string, dict, or a typed Pydantic model; never untyped or variable-shape 4. **Minimal side effects** — each call does one thing; avoid tools that "do everything" ```python theme={null} def get_schema(table_name: str, schema_fqn: str) -> dict: """Return the column definitions for a database table. Call this before generating SQL to understand the table structure. Returns an empty dict if the table does not exist. Args: table_name: The simple table name (e.g. "orders"). schema_fqn: The fully qualified schema name (e.g. "analytics-db.analytics_db.public"). Returns: A dict with keys: columns (list of {name, type, nullable}), row_count (int). """ # ... implementation return {"columns": [...], "row_count": 1234} ``` ### Schema Discipline The tool's parameter schema is sent to the LLM on every call. Verbose or inaccurate schemas degrade routing quality and inflate cost. Prefer `Literal["postgres", "redshift", "bigquery"]` over `str` when the set of valid values is known. The LLM will hallucinate less when the schema constrains choices. ```python theme={null} from typing import Literal def execute_query( sql: str, dialect: Literal["postgres", "redshift", "bigquery"], ) -> dict: """Execute a SQL query against the target database.""" ... ``` Use `Optional[T]` with a sensible default. Never use `Union[T, None]` without a default — the LLM will try to supply a value when it shouldn't. ```python theme={null} from typing import Optional def search_artifacts( query: str, max_results: int = 10, artifact_type: Optional[Literal["parquet", "chart", "code"]] = None, ) -> list[dict]: """Search for artifacts matching the query.""" ... ``` Raise only for truly unrecoverable situations. For expected failures (table not found, invalid SQL), return a structured error dict so the LLM can understand and recover. ```python theme={null} def execute_sql(sql: str) -> dict: """Execute SQL and return results or a structured error.""" try: rows = db.execute(sql) return {"success": True, "rows": rows, "error": None} except QueryError as e: # Return the error — don't raise. The LLM will retry with corrected SQL. return {"success": False, "rows": [], "error": str(e)} ``` ## MCP Tools via FastMCPToolset CRAFT exposes platform capabilities through `em-runtime-mcp` — an MCP server. Pydantic AI connects to it via `FastMCPToolset`; Google ADK can connect via a `MCPToolset` wrapper. ### Pydantic AI — FastMCPToolset A production-grade pattern for connecting a Pydantic AI agent to the CRAFT MCP server: ```python theme={null} from pydantic_ai.toolsets.fastmcp import FastMCPToolset from fastmcp import Client from fastmcp.client.transports import StreamableHttpTransport # Create a fresh client and toolset per agent request. # Never reuse a long-lived client — MCP sessions expire. mcp_client = Client( transport=StreamableHttpTransport( url="https://craft.emergence.ai/mcp", headers={ "Authorization": "Bearer ", "X-Project-ID": "", }, ) ) toolset = FastMCPToolset(mcp_client) # Inject into the agent run async with agent.run_stream( user_message, deps=agent_deps, toolsets=[toolset], ) as run: async for text in run.stream_text(): ... # Always close the client after the run await mcp_client.close() ``` ### Google ADK — MCP Tool Integration Google ADK connects to the CRAFT MCP server using `StreamableHTTPConnectionParams`: ```python theme={null} from google.adk.tools.mcp_tool import MCPToolset, StreamableHTTPConnectionParams from google.adk.agents import LlmAgent toolset = MCPToolset( connection_params=StreamableHTTPConnectionParams( url="https://craft.emergence.ai/mcp", headers={ "Authorization": "Bearer ", "X-Project-ID": "", }, ) ) root_agent = LlmAgent( model="gemini-3.5-flash", name="my_agent", description="Agent with CRAFT MCP tool access", instruction="Use available MCP tools to answer questions.", tools=[toolset], ) ``` The older import `from google.adk.toolsets.mcp import MCPToolset` is deprecated. Use `from google.adk.tools.mcp_tool import MCPToolset, StreamableHTTPConnectionParams` (ADK v1.0+). ### Claude Agent SDK — MCP Tool Integration The Anthropic API's `mcp_servers` connector only supports `authorization_token` — it cannot pass `X-Project-ID` as a separate header. Use the `mcp` Python SDK with a session-aware client to call CRAFT tools from a standard function-tool loop: ```python theme={null} import asyncio import anthropic from mcp import ClientSession from mcp.client.streamable_http import streamablehttp_client CRAFT_URL = "https://craft.emergence.ai/mcp" CRAFT_HEADERS = { "Authorization": "Bearer ", "X-Project-ID": "", } async def run_craft_agent(user_question: str) -> str: client = anthropic.Anthropic() async with streamablehttp_client(CRAFT_URL, headers=CRAFT_HEADERS) as (read, write, _): async with ClientSession(read, write) as session: await session.initialize() tools_result = await session.list_tools() tools = [ { "name": t.name, "description": t.description or "", "input_schema": t.inputSchema, } for t in tools_result.tools ] messages = [{"role": "user", "content": user_question}] while True: response = client.messages.create( model="claude-opus-4-8", max_tokens=1024, tools=tools, messages=messages, ) if response.stop_reason == "end_turn": return response.content[0].text tool_results = [] for block in response.content: if block.type == "tool_use": result = await session.call_tool(block.name, block.input) text = result.content[0].text if result.content else "" tool_results.append( {"type": "tool_result", "tool_use_id": block.id, "content": text} ) messages += [ {"role": "assistant", "content": response.content}, {"role": "user", "content": tool_results}, ] ``` ### LangGraph — MCP Tool Integration LangGraph connects to CRAFT via `MultiServerMCPClient` from `langchain-mcp-adapters` (install: `pip install langchain-mcp-adapters`): ```python theme={null} from langchain_mcp_adapters.client import MultiServerMCPClient from langgraph.prebuilt import create_react_agent async def build_craft_agent(): async with MultiServerMCPClient({ "craft": { "url": "https://craft.emergence.ai/mcp", "transport": "http", # streamable-HTTP (MCP 2025-11-25); "sse" is the legacy transport "headers": { "Authorization": "Bearer ", "X-Project-ID": "", }, } }) as mcp_client: tools = await mcp_client.get_tools() agent = create_react_agent("anthropic:claude-opus-4-8", tools) result = await agent.ainvoke( {"messages": [{"role": "user", "content": "List my data connections"}]} ) return result ``` `langchain-mcp-adapters` supports `headers` for static auth values. If the access token expires mid-session, reinitialise the client — dynamic per-request token injection is a [known limitation](https://github.com/langchain-ai/langchain-mcp-adapters/issues/194). ## Cross-Framework Tool Comparison | Feature | Google ADK | Claude Agent SDK | Pydantic AI | LangGraph | | --------------------- | ----------------------------------------------- | -------------------------------------------------- | ---------------------------------------------------------- | ------------------------------------------- | | Function tools | `tools=[my_fn]` | `tools=[{name, description, input_schema}]` | `@agent.tool` decorator or `tools=[my_fn]` | `bind_tools([my_fn])` | | CRAFT MCP connection | `MCPToolset` + `StreamableHTTPConnectionParams` | `mcp` SDK `streamablehttp_client` + function tools | `FastMCPToolset` + `StreamableHttpTransport` (first-class) | `MultiServerMCPClient` (`transport="http"`) | | Parallel tool calls | Yes (automatic) | Yes (with `betas=["interleaved-thinking"]`) | `parallel_tool_calls=True` in `ModelSettings` | Node-level parallelism | | Tool retry on failure | ADK handles automatically | Manual in tool loop | `ModelRetry` exception | Conditional edges | | Context injection | `ToolContext` | Manual in tool dispatch | `RunContext[Deps]` | `RunnableConfig` | ## Limiting Iteration and Cost Unbounded tool call loops are the leading cause of cost overruns. Implement iteration limits defensively. ```python theme={null} # Pydantic AI — track failures on deps if not hasattr(ctx.deps, "_code_failure_count"): ctx.deps._code_failure_count = 0 ctx.deps._code_failure_count += 1 if ctx.deps._code_failure_count >= settings.max_code_failures: # Return a structured error instead of raising ModelRetry. # Raising ModelRetry here exhausts the retry budget and raises # UnexpectedModelBehavior — losing the user-facing message. return { "success": False, "error": ( f"Execution has failed {ctx.deps._code_failure_count} times. " "Ask the user for clarification." ), } ``` Return structured error dicts (not `raise ModelRetry`) when you have hit an iteration limit. `ModelRetry` with the default `max_retries=1` already consumed will raise `UnexpectedModelBehavior` and produce a generic failure message instead of the clarifying ask you wanted the model to emit. ## Pre-Flight Validation Validate tool inputs before making expensive network calls. A `FastMCPToolset` subclass can run a lint pass before code execution: ```python theme={null} async def call_tool(self, name, tool_args, ctx, tool): if name == "run_code": code = tool_args.get("code", "") violations = lint_code(code) if violations: # Reject before the sandbox round-trip. # Saves latency and gives the model a sharper error message. ctx.deps._code_failure_count += 1 return { "success": False, "error": f"Forbidden pattern(s): {', '.join(v.pattern for v in violations)}", } # ... proceed to actual tool call ``` ## Next Steps Use tools and sub-agents together in orchestrated workflows. Inspect tool call traces and diagnose tool failures. # Your First CRAFT Agent Source: https://docs.emergence.ai/guides/agent-author/your-first-agent Build a working CRAFT-integrated agent in three incremental steps — from a bare hello-world to a registered, tool-calling platform agent — in under 20 minutes. This tutorial builds the same agent three times — once for each capability layer — so you can stop at whichever level you need. By the end you have a registered CRAFT agent that lists your project's data connections and is discoverable by other platform services. **What you'll build:** A data-assistant agent that: 1. Responds to prompts using CRAFT's managed LLM gateway 2. Calls the CRAFT Assets API to list real data connections 3. Is registered in the CRAFT agent registry **Time:** \~5 min per step. Steps are independent — come back later for the next one. **Prerequisites:** Obtain a CRAFT project access token via your deployment's OIDC token endpoint (client credentials grant) and export the connection settings: ```bash theme={null} export CRAFT_TOKEN="$(curl -s -X POST "${OIDC_TOKEN_URL}" \ -d "grant_type=client_credentials&client_id=${OIDC_CLIENT_ID}&client_secret=${OIDC_CLIENT_SECRET}" \ | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')" export CRAFT_GATEWAY_URL="" # LiteLLM sidecar or gateway export CRAFT_ASSETS_URL="" # CRAFT Assets API export CRAFT_PROJECT_ID="" ``` For local dev, run `docker-compose up` from the solution starter template — it pre-configures `OIDC_TOKEN_URL`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`, and the local URLs. *** ## Step 1 — Connect to CRAFT's LLM gateway The only difference between a standalone agent and a CRAFT-native agent is where it routes its LLM calls. CRAFT runs a [LiteLLM gateway](/platform/llm-gateway) that handles model selection, project billing, and rate limits. All four frameworks reach it through an OpenAI-compatible endpoint. Pick your framework: ```bash theme={null} pip install google-adk litellm ``` ```python agent.py theme={null} import os from google.adk.agents import Agent from google.adk.models.lite_llm import LiteLlm from google.adk.runners import Runner from google.adk.sessions import InMemorySessionService from google.genai import types # Point ADK at CRAFT's LiteLLM gateway via the LiteLlm wrapper. # The "openai/" prefix tells LiteLLM to use the OpenAI-compatible endpoint. model = LiteLlm( model="openai/claude-opus-4-8", api_base=os.environ["CRAFT_GATEWAY_URL"], api_key=os.environ["CRAFT_TOKEN"], ) agent = Agent( name="data_assistant", model=model, description="Helps users understand the data available in their CRAFT project.", instruction="You are a helpful data assistant. Answer questions about the user's data concisely.", ) async def main(): session_service = InMemorySessionService() session = await session_service.create_session( app_name="data_assistant", user_id="dev", session_id="s1" ) runner = Runner(agent=agent, app_name="data_assistant", session_service=session_service) async for event in runner.run_async( user_id="dev", session_id="s1", new_message=types.Content(role="user", parts=[types.Part(text="What can you help me with?")]), ): if event.is_final_response(): print(event.content.parts[0].text) if __name__ == "__main__": import asyncio asyncio.run(main()) ``` For production multi-agent patterns with A2A, see the [multi-agent patterns guide](/guides/agent-author/multi-agent-patterns) — it shows how an orchestrator uses ADK to chain a text-to-SQL agent and an insights agent over the A2A protocol. ```bash theme={null} pip install anthropic openai ``` ```python agent.py theme={null} import os from openai import OpenAI # Route through CRAFT's LiteLLM gateway using the OpenAI-compatible client. # LiteLLM serves claude-opus-4-8 (and other models) at the standard # /v1/chat/completions endpoint — no Anthropic account needed. client = OpenAI( base_url=os.environ["CRAFT_GATEWAY_URL"], api_key=os.environ["CRAFT_TOKEN"], ) def run(prompt: str) -> str: response = client.chat.completions.create( model="claude-opus-4-8", max_tokens=1024, messages=[ {"role": "system", "content": "You are a helpful data assistant. Answer questions about the user's data concisely."}, {"role": "user", "content": prompt}, ], ) return response.choices[0].message.content if __name__ == "__main__": print(run("What can you help me with?")) ``` The OpenAI client talks to CRAFT's LiteLLM gateway using Claude via the OpenAI-compatible endpoint. LiteLLM translates function-calling requests to the format each model expects, so the same pattern works for any model in the CRAFT allowlist. ```bash theme={null} pip install pydantic-ai ``` ```python agent.py theme={null} import os from pydantic_ai import Agent from pydantic_ai.models.openai import OpenAIChatModel from pydantic_ai.providers.openai import OpenAIProvider # Route through CRAFT's LiteLLM gateway using the OpenAI-compatible provider. model = OpenAIChatModel( "claude-opus-4-8", provider=OpenAIProvider( base_url=os.environ["CRAFT_GATEWAY_URL"], api_key=os.environ["CRAFT_TOKEN"], ), ) agent = Agent( model, system_prompt="You are a helpful data assistant. Answer questions about the user's data concisely.", ) result = agent.run_sync("What can you help me with?") print(result.output) ``` Pydantic AI's `FastMCPToolset` wires MCP servers as tools with first-class typing. See the [tool authoring guide](/guides/agent-author/tool-authoring) for that pattern. ```bash theme={null} pip install langgraph langchain-openai ``` ```python agent.py theme={null} import os from langchain_openai import ChatOpenAI from langgraph.prebuilt import create_react_agent from langchain_core.messages import HumanMessage # Route through CRAFT's LiteLLM gateway — it's OpenAI-compatible. llm = ChatOpenAI( model="claude-opus-4-8", base_url=os.environ["CRAFT_GATEWAY_URL"], api_key=os.environ["CRAFT_TOKEN"], ) agent = create_react_agent( model=llm, tools=[], # add tools in Step 2 prompt="You are a helpful data assistant. Answer questions about the user's data concisely.", ) result = agent.invoke({"messages": [HumanMessage(content="What can you help me with?")]}) print(result["messages"][-1].content) ``` That's it. Your agent is running on CRAFT's managed LLM infrastructure. ### Routing to specific backends The CRAFT gateway uses LiteLLM under the hood, so any model string it recognises works in the `model=` field — you don't need to change any client code. Ask your platform team which models are in the project's allowlist. ```python theme={null} # Model string format: vertex_ai/ # LiteLLM reads VERTEXAI_PROJECT + VERTEXAI_LOCATION (or the Google Cloud # standards GOOGLE_CLOUD_PROJECT + GOOGLE_CLOUD_LOCATION) and authenticates # via Workload Identity or GOOGLE_APPLICATION_CREDENTIALS automatically. # DEFAULT — fastest, highest quality on agentic + coding benchmarks model = "vertex_ai/gemini-3.5-flash" # SPECIALIZED — only if your agent depends on extensive custom tool calling # (the model prioritizes your registered functions over bash fallbacks) model = "vertex_ai/gemini-3.1-pro-preview-customtools" # SECONDARY — demonstrates model agnosticity model = "vertex_ai/claude-opus-4-8" ``` **Use `gemini-3.1-pro-preview-customtools` only when extensive custom tool calling is core to your agent.** Same intelligence as base `gemini-3.1-pro`, fine-tuned to prioritize registered custom functions over bash fallbacks. Google's own guidance: if >50% of requests don't involve tool calling, stay on `gemini-3.5-flash` — `3.5-flash` already beats `3.1-pro` on agentic benchmarks (MCP Atlas +5.4%) and coding (Terminal-Bench +6%) at 3.6× the speed ([source](https://wavespeed.ai/blog/posts/gemini-3-5-pro-flash/)), and the customtools variant degrades quality on non-tool workloads ([source](https://help.apiyi.com/en/gemini-3-1-pro-preview-customtools-agent-guide-en.html)). When you do use it: preview SLA, global endpoint only (`vertex_location` must be `global`), lower quota than GA — configure a fallback to `gemini-3.5-flash` or `claude-opus-4-8`. ```python theme={null} # Model string format: bedrock/ # LiteLLM uses AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY or instance profile. model = "bedrock/anthropic.claude-opus-4-8-20260301-v1:0" ``` ```python theme={null} # Model string format: azure/ # LiteLLM reads AZURE_API_KEY and AZURE_API_BASE from the gateway config. model = "azure/gpt-5.5" ``` ```python theme={null} # Nebius exposes an OpenAI-compatible endpoint; prefix with openai/. # LiteLLM reads NEBIUS_API_KEY from the gateway config. model = "openai/Qwen/Qwen3-30B-A3B" ``` *** ## Step 2 — Add a CRAFT tool Tools let your agent take action. The simplest CRAFT tool calls the Assets API to list the data connections registered in your project — real databases and warehouses the platform knows about. ```python agent.py theme={null} import os import httpx from google.adk.agents import Agent from google.adk.models.lite_llm import LiteLlm from google.adk.runners import Runner from google.adk.sessions import InMemorySessionService from google.genai import types # ── CRAFT tool ───────────────────────────────────────────────────────── def list_data_connections() -> str: """List the data connections available in the current CRAFT project.""" resp = httpx.get( f"{os.environ['CRAFT_ASSETS_URL']}/assets/data", headers={ "Authorization": f"Bearer {os.environ['CRAFT_TOKEN']}", "X-Project-ID": os.environ["CRAFT_PROJECT_ID"], }, ) resp.raise_for_status() items = resp.json().get("data", []) if not items: return "No data connections found in this project." return "\n".join(f"- {c['name']} ({c['connection_type']})" for c in items) # ── Agent ────────────────────────────────────────────────────────────── model = LiteLlm( model="openai/claude-opus-4-8", api_base=os.environ["CRAFT_GATEWAY_URL"], api_key=os.environ["CRAFT_TOKEN"], ) agent = Agent( name="data_assistant", model=model, description="Helps users understand the data available in their CRAFT project.", instruction="You are a helpful data assistant. Use list_data_connections to answer questions about available data.", tools=[list_data_connections], ) async def main(): session_service = InMemorySessionService() session = await session_service.create_session( app_name="data_assistant", user_id="dev", session_id="s1" ) runner = Runner(agent=agent, app_name="data_assistant", session_service=session_service) async for event in runner.run_async( user_id="dev", session_id="s1", new_message=types.Content(role="user", parts=[types.Part(text="What data connections do I have?")]), ): if event.is_final_response(): print(event.content.parts[0].text) if __name__ == "__main__": import asyncio asyncio.run(main()) ``` ```python agent.py theme={null} import os import json import httpx from openai import OpenAI # Same CRAFT gateway client as Step 1 — tool calling works over the # OpenAI-compatible endpoint via LiteLLM's function-calling support. client = OpenAI( base_url=os.environ["CRAFT_GATEWAY_URL"], api_key=os.environ["CRAFT_TOKEN"], ) # ── CRAFT tool definition (OpenAI function-calling format) ────────────── TOOLS = [ { "type": "function", "function": { "name": "list_data_connections", "description": ( "List all data connections (databases, data warehouses, APIs) registered " "in the current CRAFT project. Call this whenever the user asks what data " "is available." ), "parameters": {"type": "object", "properties": {}, "required": []}, }, } ] def list_data_connections() -> str: resp = httpx.get( f"{os.environ['CRAFT_ASSETS_URL']}/assets/data", headers={ "Authorization": f"Bearer {os.environ['CRAFT_TOKEN']}", "X-Project-ID": os.environ["CRAFT_PROJECT_ID"], }, ) resp.raise_for_status() items = resp.json().get("data", []) if not items: return "No data connections found in this project." return "\n".join(f"- {c['name']} ({c['connection_type']})" for c in items) TOOL_HANDLERS = {"list_data_connections": list_data_connections} # ── Agentic loop ──────────────────────────────────────────────────────── def run(prompt: str) -> str: messages = [ {"role": "system", "content": "You are a helpful data assistant. Use your tools to answer questions about available data."}, {"role": "user", "content": prompt}, ] while True: response = client.chat.completions.create( model="claude-opus-4-8", max_tokens=1024, tools=TOOLS, messages=messages, ) msg = response.choices[0].message messages.append(msg) if not msg.tool_calls: return msg.content # Execute tool calls and feed results back results = [] for tc in msg.tool_calls: output = TOOL_HANDLERS[tc.function.name](**json.loads(tc.function.arguments or "{}")) results.append({ "role": "tool", "tool_call_id": tc.id, "content": output, }) messages.extend(results) if __name__ == "__main__": print(run("What data connections do I have?")) ``` ```python agent.py theme={null} import os import httpx from pydantic_ai import Agent from pydantic_ai.models.openai import OpenAIChatModel from pydantic_ai.providers.openai import OpenAIProvider model = OpenAIChatModel( "claude-opus-4-8", provider=OpenAIProvider( base_url=os.environ["CRAFT_GATEWAY_URL"], api_key=os.environ["CRAFT_TOKEN"], ), ) agent = Agent( model, system_prompt="You are a helpful data assistant. Use your tools to answer questions about available data.", ) # ── CRAFT tool ───────────────────────────────────────────────────────── @agent.tool_plain def list_data_connections() -> str: """List all data connections registered in the current CRAFT project.""" resp = httpx.get( f"{os.environ['CRAFT_ASSETS_URL']}/assets/data", headers={ "Authorization": f"Bearer {os.environ['CRAFT_TOKEN']}", "X-Project-ID": os.environ["CRAFT_PROJECT_ID"], }, ) resp.raise_for_status() items = resp.json().get("data", []) if not items: return "No data connections found in this project." return "\n".join(f"- {c['name']} ({c['connection_type']})" for c in items) result = agent.run_sync("What data connections do I have?") print(result.output) ``` `@agent.tool_plain` registers the function as a tool with no dependency injection. Pydantic AI infers the JSON schema from the docstring and type hints automatically. ```python agent.py theme={null} import os import httpx from langchain_openai import ChatOpenAI from langchain_core.tools import tool from langchain_core.messages import HumanMessage from langgraph.prebuilt import create_react_agent llm = ChatOpenAI( model="claude-opus-4-8", base_url=os.environ["CRAFT_GATEWAY_URL"], api_key=os.environ["CRAFT_TOKEN"], ) # ── CRAFT tool ───────────────────────────────────────────────────────── @tool def list_data_connections() -> str: """List all data connections registered in the current CRAFT project. Call this whenever the user asks what data sources or databases are available. """ resp = httpx.get( f"{os.environ['CRAFT_ASSETS_URL']}/assets/data", headers={ "Authorization": f"Bearer {os.environ['CRAFT_TOKEN']}", "X-Project-ID": os.environ["CRAFT_PROJECT_ID"], }, ) resp.raise_for_status() items = resp.json().get("data", []) if not items: return "No data connections found in this project." return "\n".join(f"- {c['name']} ({c['connection_type']})" for c in items) agent = create_react_agent( model=llm, tools=[list_data_connections], prompt="You are a helpful data assistant. Use your tools to answer questions about available data.", ) result = agent.invoke({"messages": [HumanMessage(content="What data connections do I have?")]}) print(result["messages"][-1].content) ``` Your agent now answers questions using real platform data. Every other CRAFT capability — agents, files, models, artifacts — is one more `httpx.get()` away. *** ## Step 3 — Register your agent A registered agent is discoverable by other platform services, the UI, and other agents via the A2A protocol. Registration is one `POST` to the Assets API. ```python register.py theme={null} import os import httpx # POST /assets/agents expects the full Agent Card wrapped in "agent_card". # "name" and "version" are required; "url" points to your running service's # /.well-known/agent-card.json endpoint so other agents can fetch full details. registration = { "agent_card": { "name": "data-assistant", "version": "1.0.0", "description": "Answers questions about data connections in a CRAFT project.", "url": "http://your-service-host/.well-known/agent-card.json", "capabilities": { "streaming": False, "push_notifications": False, }, }, "tags": ["data", "assistant"], } resp = httpx.post( f"{os.environ['CRAFT_ASSETS_URL']}/assets/agents", json=registration, headers={ "Authorization": f"Bearer {os.environ['CRAFT_TOKEN']}", "X-Project-ID": os.environ["CRAFT_PROJECT_ID"], }, ) resp.raise_for_status() agent_id = resp.json()["resource_uri"] print(f"Registered: {agent_id}") ``` The `agent_card_url` points to a `/.well-known/agent-card.json` endpoint your service exposes — it describes your agent's skills so other agents can call it. See the [A2A protocol primer](/guides/agent-author/a2a-protocol-primer) for the Agent Card schema, and [multi-agent patterns](/guides/agent-author/multi-agent-patterns) for wiring A2A delegation. Re-POSTing the same `name` in a project returns **409 `RESOURCE_ALREADY_EXISTS`**. To update an existing registration, `PUT /assets/agents/{resource_uri}` and include the current ETag in `If-Match`: * `PUT` without `If-Match` returns **428 `Precondition Required`**. * `PUT` with a stale `If-Match` returns **412 `Precondition Failed`** (`"ETag mismatch — resource was modified"`). Read the current ETag from the `ETag` header on a prior `GET /assets/agents/{resource_uri}` (or use the `current_version` field from the JSON body — both are equivalent), and feed it back on the next `PUT`. Either the bare integer (`If-Match: 1`) or the quoted form (`If-Match: "1"`) is accepted; weak ETags (`W/"1"`) are rejected. Automate the `GET → PUT(If-Match)` cycle in your deploy pipeline so the registry stays in sync with your running service. *** ## What's next Function tools, MCP tools via FastMCPToolset, schema discipline. A2A delegation, parallel fan-out, supervisor agents. Agent Cards, JSON-RPC over SSE, task lifecycle. Golden traces, Langfuse evaluators, regression suites. # Backup & Restore Source: https://docs.emergence.ai/guides/backup-restore How to back up and restore CRAFT data including PostgreSQL databases, Keycloak realms, OpenFGA state, and Infisical secrets. # Backup and Restore Platform Data This guide covers backup and restore procedures for all stateful components of CRAFT. Each service owns its own database, and infrastructure services (Keycloak, OpenFGA, Infisical) maintain independent state that must be backed up separately. ## Stateful Components The platform has the following stateful components that require backup: | Component | Storage | Data | | ----------------- | --------------- | ------------------------------------------------- | | **Governance DB** | PostgreSQL | Organizations, projects, users, role assignments | | **Assets DB** | PostgreSQL | Artifacts, data connections, files, models | | **Utils DB** | PostgreSQL | Data catalog, scheduling, context packs, memories | | **Keycloak** | PostgreSQL | Realms, users, IdP configurations, sessions | | **OpenFGA** | PostgreSQL | Authorization model, relationship tuples | | **Infisical** | Internal store | Application secrets, data connection credentials | | **Redis** | In-memory + AOF | Cache, session state, event streams | | **Solution DBs** | PostgreSQL | Data Insights sessions, Data Governance profiles | Each service has a separate PostgreSQL database. There are no cross-service foreign keys, so databases can be backed up independently without coordination. ## PostgreSQL Backup ### Automated Backups (Cloud-Managed) For cloud-managed PostgreSQL (Cloud SQL, RDS, Azure Database): Cloud SQL provides automated daily backups with configurable retention. ```bash theme={null} # Enable automated backups gcloud sql instances patch \ --backup-start-time=02:00 \ --retained-backups-count=30 \ --retained-transaction-log-days=7 # Create an on-demand backup gcloud sql backups create --instance= # List available backups gcloud sql backups list --instance= ``` ```bash theme={null} # Enable automated backups with 30-day retention aws rds modify-db-instance \ --db-instance-identifier \ --backup-retention-period 30 # Create a manual snapshot aws rds create-db-snapshot \ --db-instance-identifier \ --db-snapshot-identifier emergence-backup-$(date +%Y%m%d) ``` ```bash theme={null} # Azure Database for PostgreSQL has automated backups enabled by default # Configure retention (7-35 days) az postgres flexible-server update \ --resource-group \ --name \ --backup-retention 30 ``` ### Manual Backups (On-Premises) For self-managed PostgreSQL deployments: ```bash theme={null} # Back up all platform databases DATABASES="keycloak openfga infisical governance assets utils prefect datareadiness talk2data" BACKUP_DIR="/backups/emergence/$(date +%Y%m%d_%H%M%S)" mkdir -p "$BACKUP_DIR" for db in $DATABASES; do pg_dump -h localhost -U postgres -Fc -f "$BACKUP_DIR/$db.dump" "$db" done # Verify backups for dump in "$BACKUP_DIR"/*.dump; do pg_restore --list "$dump" > /dev/null && echo "OK: $dump" || echo "FAILED: $dump" done ``` Use `pg_dump -Fc` (custom format) for backups. It supports parallel restore and selective table restoration, and compresses data automatically. ### Point-in-Time Recovery For production deployments, enable WAL archiving for point-in-time recovery: ```bash theme={null} # postgresql.conf archive_mode = on archive_command = 'cp %p /archive/wal/%f' wal_level = replica ``` ## Keycloak Backup Keycloak state is primarily stored in its PostgreSQL database, but realm configuration can also be exported as JSON for version control. ```bash theme={null} # Export realm configuration (includes users, IdPs, clients) # Run from within the Keycloak container or pod /opt/keycloak/bin/kc.sh export \ --dir /tmp/keycloak-export \ --realm \ --users realm_file # Copy the export out of the container kubectl cp :/tmp/keycloak-export ./keycloak-backup/ ``` Realm exports include user accounts but not user credentials (passwords). Users will need to reset passwords after a realm import. SSO users are unaffected since their credentials are managed by the external IdP. ## OpenFGA Backup OpenFGA stores its authorization model and relationship tuples in PostgreSQL. The database backup covers all OpenFGA state. For additional safety, export the authorization model: ```bash theme={null} # Export the current authorization model curl -s "http://:8080/stores//authorization-models" \ -H "Authorization: Bearer $TOKEN" | jq . > openfga-model-backup.json ``` ## Secrets Backend Backup The platform supports two secrets backends. Back up whichever you are using. Infisical manages application secrets including data connection credentials. Back up the Infisical database (PostgreSQL `infisical` database) and preserve the encryption keys. Infisical secrets are encrypted at rest. Backing up the database preserves the encrypted data, but you must also preserve the `ENCRYPTION_KEY` and `AUTH_SECRET` for restoration, without them, encrypted data cannot be decrypted. ```bash theme={null} pg_dump -h localhost -U postgres -Fc -f infisical-backup.dump infisical # Also export the encryption key from the infisical-bootstrap-secret: kubectl get secret infisical-bootstrap-secret -n em-runtime -o yaml > infisical-bootstrap-secret.yaml ``` Secrets stored in GCP Secret Manager have built-in version history. No separate backup action is required, GCP retains all secret versions. To export a snapshot of all current secret values: ```bash theme={null} # List all em-runtime secrets gcloud secrets list --project=your-project-id --filter="name:em-runtime" # Verify each secret has at least one active version gcloud secrets versions list em-runtime --project=your-project-id ``` For disaster recovery, ensure IAM bindings for the ESO service account are documented and reproducible via Terraform. ## Redis Backup Redis serves as a cache and event stream. While cache data is ephemeral, you may want to back up Redis for faster recovery: ```bash theme={null} # Trigger a manual RDB snapshot redis-cli -a $REDIS_PASSWORD BGSAVE # Copy the dump file kubectl cp :/data/dump.rdb ./redis-backup/dump.rdb ``` ## Restore Procedures ### Restore PostgreSQL Databases ```bash theme={null} # Restore a single database from a custom-format dump pg_restore -h localhost -U postgres -d governance -Fc --clean --if-exists governance.dump # Restore all platform databases for db in keycloak openfga infisical governance assets utils prefect datareadiness talk2data; do pg_restore -h localhost -U postgres -d "$db" -Fc --clean --if-exists "$db.dump" done ``` ### Restore Keycloak Realms ```bash theme={null} # Import realm configuration /opt/keycloak/bin/kc.sh import \ --dir /tmp/keycloak-import \ --override true ``` ### Restore from Cloud Managed Backups ```bash theme={null} # Restore from an automated backup gcloud sql backups restore \ --restore-instance= ``` ```bash theme={null} # Restore from a snapshot (creates a new instance) aws rds restore-db-instance-from-db-snapshot \ --db-instance-identifier \ --db-snapshot-identifier ``` ## Backup Schedule Recommendations | Component | Frequency | Retention | Method | | --------------------------- | ---------------------- | ---------- | ------------------------------------------------------------ | | PostgreSQL (all DBs) | Daily + continuous WAL | 30 days | Cloud-managed or pg\_dump | | Keycloak realm exports | Weekly | 90 days | JSON export | | OpenFGA model | On change | Indefinite | JSON export in version control | | Redis | Daily | 7 days | RDB snapshot | | Secrets backend (Infisical) | Daily | 30 days | Database backup with encryption keys | | Secrets backend (GCP SM) | N/A, managed | Indefinite | GCP retains all versions; document IAM bindings in Terraform | ## Disaster Recovery Checklist Restore all platform databases from the latest backup. Verify row counts and data integrity. Import realm configurations. SSO users will re-authenticate via their IdP. Local users may need password resets. Confirm the authorization model is loaded. The Governance service re-applies the schema on startup. **Infisical**: Restore the Infisical database and encryption keys. Verify data connection credentials are accessible. **ESO + GCP Secret Manager**: Verify GCP Secret Manager secrets are intact and ESO ClusterSecretStore can authenticate via Workload Identity. Follow the startup order: PostgreSQL -> Redis -> Keycloak -> OpenFGA -> Governance -> Assets/Utils -> Solutions. Run health checks on all services. Test authentication, permission checks, and data connection queries. ## Next Steps Review the full deployment architecture and infrastructure requirements. Configure Helm values for backup-related settings. Understand data retention requirements for GDPR compliance. Secure backup data in transit and at rest. # Audit Log Source: https://docs.emergence.ai/guides/customer-admin/audit-log View and export the audit log for your CRAFT organisation — what events are captured and how to use them for compliance. The audit log is a tamper-evident record of significant actions taken by users and the platform within your organisation. It is available to organisation owners and admins and is scoped strictly to your organisation — you cannot see events from other organisations. ## Where to find the audit log In the CRAFT platform, navigate to **Settings** and select **Audit Log** from the sidebar. ## What the audit log captures The audit log records actions across the following categories: | Category | Examples of events captured | | ---------------------- | -------------------------------------------------------------------------------------- | | **User management** | User invited, role changed, account deactivated, account reactivated | | **Project management** | Project created, project deleted, member added to project, member removed from project | | **Data connections** | Data connection created, updated, or deleted | | **Secrets** | Secret created, secret deleted (content is never logged) | | **Authentication** | Login, logout, failed login attempt | | **Agent activity** | Agent tool calls (when agent audit is enabled for a project) | | **Audit settings** | Retention period changed | Each event record includes: | Field | Description | | ------------- | ------------------------------------------------------------ | | **Timestamp** | When the event occurred (UTC) | | **Actor** | The user or service that performed the action | | **Action** | What was done (for example, `user.invite`, `project.create`) | | **Resource** | The specific resource that was affected | | **Project** | The project context, if applicable | | **Details** | Additional context specific to the event type | ## Viewing and filtering the audit log Navigate to **Settings** and select **Audit Log**. Use the filter options to narrow down the events you want to review: * **Date range**: select a start and end date * **Actor**: filter by a specific user * **Action**: filter by action type (for example, show only login events) * **Resource type**: filter by the type of resource affected * **Project**: filter to events within a specific project Filters can be combined. For example, you can see all actions taken by a specific user within a date range. The list shows matching events in reverse chronological order (newest first). Select any event to expand it and see the full details. ## Exporting the audit log You can export audit events to a CSV file for use in compliance reports, external monitoring tools, or archiving. Set the date range and any other filters for the events you want to export. Select **Export to CSV** in the top-right area of the Audit Log page. The file will download to your browser. Exports are paginated at up to 50,000 rows per file. If your date range contains more than 50,000 events, export in smaller date ranges and combine the files. ## Audit log retention By default, audit events are retained for 90 days. Organisation owners and admins can adjust the retention period under **Settings → Audit Log → Retention Settings**. Reducing the retention period will immediately delete events outside the new window. This action cannot be undone. Set a retention period that meets your compliance requirements before reducing it. ## Compliance use cases The audit log is designed to support common compliance and security review scenarios: **Access reviews:** Filter by date range to see which users had access to which resources during a specific period. Look for role change events to identify when permissions were granted or removed. **Incident investigation:** Filter by actor and date range to reconstruct the sequence of actions taken by a specific user during an incident window. **Data access auditing:** Filter by resource type `data_connection` and action `execute` to see which users ran queries against a specific data source. **Offboarding verification:** After deactivating a user, search for events by their user ID to confirm their access was removed and identify any resources that may need a new owner. # Configure SSO Source: https://docs.emergence.ai/guides/customer-admin/configure-sso Set up single sign-on for your organisation in CRAFT using your existing identity provider. Single sign-on (SSO) lets your team log in to CRAFT using their existing company credentials. Once SSO is configured, users are redirected to your identity provider when they access CRAFT — they do not need to set or remember a separate CRAFT password. SSO configuration requires organisation owner permissions. If you do not see the SSO settings described below, contact your organisation owner. ## Supported identity providers CRAFT supports SSO via two standard protocols: | Protocol | Compatible providers | | ------------------------- | ----------------------------------------------------------------------------------------------- | | **OIDC** (OpenID Connect) | Microsoft Entra ID (formerly Azure AD), Google Workspace, Okta, and any OIDC-compliant provider | | **SAML 2.0** | Microsoft Entra ID, Okta, Ping Identity, and any SAML 2.0-compliant provider | If your provider is not listed, contact Emergence support — CRAFT can be configured to work with any standards-compliant identity provider. ## Before you begin You will need the following information from your identity provider administrator: **For OIDC:** * Discovery URL (also called the "well-known" endpoint), in the format `https://your-provider.example.com/.well-known/openid-configuration` * Client ID and client secret (you will create an application in your identity provider and receive these) * Redirect URL to enter in your identity provider: provided in the CRAFT SSO settings screen **For SAML:** * Identity provider metadata URL or XML file * Redirect URL and SP entity ID to enter in your identity provider: provided in the CRAFT SSO settings screen ## Configure SSO in CRAFT In CRAFT, navigate to **Settings** and select **Single Sign-On** from the sidebar. Choose **OIDC** or **SAML** depending on what your identity provider supports. If both are available, OIDC is recommended for simplicity. Fill in the connection details for your provider: * **For OIDC**: enter the discovery URL, client ID, and client secret. * **For SAML**: upload the metadata XML file or enter the metadata URL. CRAFT will show you the redirect URL and any other values you need to enter in your identity provider's application configuration. In your identity provider's administration console, create an application or enterprise app for CRAFT and enter the values shown in the CRAFT SSO settings screen (redirect URL, entity ID, etc.). Refer to your identity provider's documentation for the specific steps: * [Microsoft Entra ID — OIDC app registration](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) * [Google Workspace — OIDC setup](https://developers.google.com/identity/openid-connect/openid-connect) * [Okta — OIDC app setup](https://developer.okta.com/docs/guides/implement-grant-type/authcode/main/) For the engineer-side configuration details and Helm chart settings, see [SSO Integration Guide](/guides/sso-integration). After saving your configuration, select **Test Connection**. CRAFT will attempt to authenticate with your identity provider. If the test succeeds, SSO is ready to use. Test with your own account first before asking other users to switch to SSO login. This way you can confirm the configuration is correct without affecting your team. Once the connection test passes, toggle **SSO Enabled** to on. From this point, users will be redirected to your identity provider when they log in to CRAFT. ## What happens to existing users When you enable SSO, existing users who log in with a CRAFT password are asked to re-authenticate via your identity provider. Their email address in CRAFT must match the email address in your identity provider — otherwise they will be treated as a new user. If a user's email address in CRAFT does not match their identity provider email, contact Emergence support to merge the accounts. ## Troubleshooting The user's email address may not be in the group or application you configured in your identity provider. Check that the user has been assigned to the CRAFT application in your identity provider's admin console. Check that the redirect URL in your identity provider exactly matches the value shown in the CRAFT SSO settings screen. A trailing slash or incorrect protocol (http vs https) will cause redirect failures. Double-check the client ID and client secret. If you recently rotated the client secret in your identity provider, you will need to update it in the CRAFT SSO settings. # Manage Users Source: https://docs.emergence.ai/guides/customer-admin/manage-users Invite users to your organisation, assign roles, and deactivate accounts in CRAFT. User management is available to organisation owners and admins. You can invite new users, assign or change their roles, and deactivate accounts that should no longer have access. ## Where to find user management In the CRAFT platform, go to **Settings** in the main navigation, then select **Members** or **Users** from the settings sidebar. ## Invite a new user Navigate to **Settings** and select **Members**. You will see a list of all current members in your organisation. Select the **Invite Member** button in the top-right area of the Members page. Type the email address of the person you want to invite. Make sure you use the same email address they use with your company's identity provider if SSO is configured — otherwise they may have trouble logging in. Choose the role that matches what this person needs to do. See [Roles and Permissions](/guides/customer-admin/roles-and-permissions) for a description of each role. You can change the role at any time after the user accepts their invitation. Select **Send Invite**. The user will receive an email with a link to accept the invitation and set up their account. If the user does not receive the invitation email within a few minutes, ask them to check their spam folder. If SSO is configured, they may be directed to log in with their company credentials instead of setting a password. ## Change a user's role On the **Members** page, locate the user whose role you want to change. You can use the search field to find them by name or email address. Select the user's name or the options menu (three dots) next to their row. Select **Edit Role** or the current role name and choose a new role from the dropdown. The change takes effect immediately — the user does not need to log out and back in. ## Deactivate an account When someone leaves your organisation or no longer needs access, deactivate their account rather than deleting it. Deactivation preserves any work they created while removing their ability to log in. On the **Members** page, locate the user you want to deactivate. Select the options menu (three dots) next to the user's row and choose **Deactivate**. Confirm the action when prompted. The user will appear as **Inactive** on the Members page. They will not be able to log in, and their API keys (if any) will be revoked immediately. Deactivating a user does not delete their work. Projects, data connections, and other resources they created remain in place. If you need to transfer ownership of a resource, reassign it to another user before deactivating the account. ## Reactivate a deactivated account If someone needs access again after being deactivated, select the options menu next to their row on the Members page and choose **Reactivate**. You will be prompted to confirm and re-select their role. # Customer Admin Guide Source: https://docs.emergence.ai/guides/customer-admin/overview What CRAFT org admins can do — and what they cannot — with quick links to each admin task. This guide is for people who manage a CRAFT organisation on behalf of their company. You might be an IT administrator, a Customer Success lead, or a compliance officer. You do not need to be a developer to use the features described here. ## What an org admin can do As an org admin, you manage your organisation's presence on CRAFT. Your responsibilities include: * **Inviting and deactivating users** — control who has access to your organisation * **Assigning roles** — determine what each person can see and do * **Configuring single sign-on (SSO)** — connect CRAFT to your company's identity provider * **Registering data connections** — add the data sources your teams will use * **Reviewing audit logs** — track actions taken by users in your organisation for compliance purposes ## What an org admin cannot do Org admins work within the application layer. You do not have access to the underlying infrastructure: * You cannot access the Kubernetes cluster or server configuration * You cannot modify the CRAFT platform itself — only your organisation's settings * You cannot see data or projects belonging to other organisations * You cannot grant yourself access to a project you have not been given access to — project-level access must be granted by a project owner or admin If you need infrastructure-level changes (for example, adding more storage or adjusting resource limits), contact your platform operator or Emergence support. ## Navigating the admin area All admin tasks are available under **Settings** in the CRAFT platform. After you log in, select **Settings** from the main navigation to access: Invite new users, assign roles, and deactivate accounts. Understand what each role can do and how to assign roles to users. Connect your identity provider so your team can log in with their existing credentials. Add and manage the data sources available to your team. Review and export a record of actions taken in your organisation. ## Who else to contact | Need | Contact | | ----------------------------- | ----------------------------------------------------------------------------- | | Account or billing questions | Your Emergence account manager | | Platform outages or bugs | Emergence support | | Infrastructure changes | Your platform operator | | Onboarding a new project team | Assign a project owner in [Manage Users](/guides/customer-admin/manage-users) | # Register Data Connections Source: https://docs.emergence.ai/guides/customer-admin/register-data-connections Add and manage data sources in CRAFT so your team can ask questions about their data. Data connections let your team point CRAFT at the databases and data sources they want to query and analyse. Once a data connection is registered, it appears as an option when your users start a data conversation in Talk2Data or run a data workflow. Registering data connections requires organisation owner or admin permissions, or project admin permissions for project-scoped connections. ## Where to find data connections In the CRAFT platform, navigate to **Data Connectors** in the main sidebar. This page lists all data connections registered in your organisation. ## Supported connection types CRAFT supports connecting to a range of data source types. The exact list depends on which data connectors have been enabled for your deployment — contact your platform operator if you need a connector that is not shown. Common connection types include: | Type | Examples | | ----------------------- | ------------------------------------------------------------------------------------ | | **Relational database** | PostgreSQL, MySQL, Microsoft SQL Server, Amazon Redshift, Google BigQuery, Snowflake | | **Data warehouse** | Databricks, Azure Synapse | | **File storage** | CSV or Excel files uploaded directly to the platform | ## Register a new data connection Navigate to **Data Connectors** in the main sidebar and select **Add Connection**. Choose the type of data source you want to connect. The form will update to show the fields required for that connection type. Fill in the connection settings for your data source: * **Name**: a descriptive name your team will recognise (for example, "Production Analytics DB" or "Customer Data Warehouse") * **Host / endpoint**: the address of your database server * **Database name**: the specific database to connect to * **Port**: the connection port (CRAFT pre-fills the default for most database types) Use a descriptive name — this is what your users will see when they select a data source to query. "postgres-db-01" is harder to find than "Customer Analytics — Snowflake". Enter the credentials CRAFT should use to connect to your data source. Use a dedicated read-only service account wherever possible — CRAFT only needs read access to run queries. Credentials are stored securely and are not visible after you save the connection. Do not use a personal account or an admin account for the data connection. Use a service account with the minimum permissions required (read-only access to the relevant schemas or tables). Select **Test Connection** before saving. CRAFT will attempt to connect to your data source using the credentials you entered. If the test fails, check: * That the host address and port are correct * That the service account has been granted access to the database from the CRAFT platform's network * That firewall rules or IP allowlists include the CRAFT platform's egress IP addresses (ask your platform operator for these) After a successful connection test, save the connection. You can then assign it to one or more projects so that project members can use it in their data conversations. ## Edit or remove a data connection To update the credentials or settings for an existing connection, select the connection on the **Data Connectors** page and choose **Edit**. To remove a connection, choose **Delete** — this will make the connection unavailable to all projects that use it. Deleting a data connection will break any saved conversations or workflows that reference it. Make sure users are aware before removing a connection that is in active use. ## Assign a connection to a project Data connections can be scoped to specific projects so that only users in that project can access them. To assign a connection, open the connection's detail page and select **Assign to Project**, then choose the project from the list. For more information on how data connections work within the CRAFT platform, see [Data Connections](/platform/data-connections). # Roles and Permissions Source: https://docs.emergence.ai/guides/customer-admin/roles-and-permissions Available roles in CRAFT, what each role can do, and how to assign or remove roles via the UI. CRAFT uses a role-based access system to control what each person can see and do within your organisation and its projects. Roles apply at two levels: the **organisation level** and the **project level**. ## Organisation-level roles Organisation-level roles control access to organisation-wide settings and the ability to manage members. | Role | What they can do | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Owner** | Full access to everything — invite and deactivate members, manage all projects, configure SSO, view audit logs, and manage organisation settings. There must always be at least one owner. | | **Admin** | Invite and deactivate members, create and manage projects, view audit logs. Cannot change organisation billing or ownership. | | **Member** | Access to projects they have been explicitly added to. Cannot manage members or organisation settings. | ## Project-level roles Within a project, users can be assigned more granular roles by a project owner or admin. | Role | What they can do | | ------------- | ----------------------------------------------------------------------------------------- | | **Owner** | Full access to the project — manage members, create and delete resources, manage secrets. | | **Admin** | Manage project members and resources. Cannot delete the project. | | **Developer** | Create, read, and modify resources. Can manage secrets within the project. | | **Operator** | Read resources and trigger workflows. Cannot create or delete resources. | | **Viewer** | Read-only access to resources and metadata within the project. | Organisation owners and admins automatically have admin-level access to all projects in the organisation, even if they are not explicitly added to a project. ## How to assign a role You assign organisation-level roles when inviting a user or by editing their profile on the **Members** page. See [Manage Users](/guides/customer-admin/manage-users) for step-by-step instructions. For project-level roles, a project owner or admin assigns roles from within the project settings. Organisation admins can also assign project roles. ## How to remove a role To remove a user from a project, navigate to the project's **Members** settings and remove them. To change an organisation role, edit the user's profile on the **Members** page. Removing a user from a project does not deactivate their account — they retain access to other projects they belong to and to the organisation. # Data Source Setup Source: https://docs.emergence.ai/guides/data-source-setup How to connect a PostgreSQL database to CRAFT as a data source for querying and governance. # Connect a PostgreSQL Database This guide walks you through registering a PostgreSQL database as a data connection in CRAFT. Once connected, the database is available to solutions like Data Insights (for NL-to-SQL queries) and Data Governance (for profiling and enrichment). ## Prerequisites Before you begin, ensure you have: * A running CRAFT instance * A valid JWT token with the `developer` or `admin` role * Network connectivity between the platform and your PostgreSQL instance * A PostgreSQL user with read access to the target schemas ## How Data Connections Work Data connections are managed by the **Assets** service (reachable at `/api/assets` through the platform gateway). When you register a connection: 1. The connection metadata (host, port, database name) is stored in the Assets database 2. Credentials (username, password) are stored securely via the platform Secrets API (Infisical or ESO + GCP Secret Manager) -- never in the Assets database 3. Solutions retrieve the connection at runtime via the Assets SDK and use the credentials to establish a live database session 4. All connections are scoped to an **organization** and **project** for multi-tenant isolation Data connection credentials are never exposed in API responses. They are injected at runtime only when a solution needs to establish a connection. ## Step 1: Prepare Your Database Ensure your PostgreSQL instance is configured to accept connections from the platform. ```sql theme={null} -- Connect to your PostgreSQL instance as a superuser CREATE USER emergence_reader WITH PASSWORD 'your-secure-password'; -- Grant read access to the target schema GRANT USAGE ON SCHEMA public TO emergence_reader; GRANT SELECT ON ALL TABLES IN SCHEMA public TO emergence_reader; -- Ensure future tables are also accessible ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO emergence_reader; ``` Update `pg_hba.conf` to allow connections from the platform's network: ```text theme={null} # Allow connections from the Kubernetes cluster CIDR host your_database emergence_reader 192.0.2.0/24 scram-sha-256 ``` Reload the configuration: ```bash theme={null} sudo systemctl reload postgresql ``` From a pod in the platform's Kubernetes cluster, test the connection: ```bash theme={null} psql -h -U emergence_reader -d your_database -c "SELECT 1;" ``` ## Step 2: Register the Data Connection Use the Assets API to register the connection in the platform. ```bash theme={null} curl -X POST "https:///api/assets/data" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " \ -H "Content-Type: application/json" \ -d '{ "name": "production-analytics", "description": "Production analytics database (read-only)", "connection_type": "postgres", "database_config": { "host": "db.example.com", "port": 5432, "database": "analytics", "ssl_mode": "require" }, "credentials": { "username": "emergence_reader", "password": "your-secure-password" } }' ``` A successful registration returns `201 Created` with the resource: ```json theme={null} { "resource_uri": "data:::production-analytics", "name": "production-analytics", "connection_type": "postgres", "status": "PENDING", "project_id": "", "owner_id": "", "tags": [], "created_at": "2026-05-31T23:40:43.698045Z" } ``` * The platform constructs `resource_uri` as `data:::` (or a kebab-case `resource_uri` override if you supply one in the request). * The `credentials` field is encrypted and stored via the platform Secrets API upon registration. It will **not** appear on subsequent `GET` responses (use `GET /assets/data/{resource_uri}/secret` if a solution needs to read the credentials at runtime). * For object-storage connections (`s3`, `gcs`, `minio`), use `storage_config` instead of `database_config`. For filesystem connections (`nfs`, `smb`), use `filesystem_config`. The complete list of supported `connection_type` values is `GET /api/assets/data/types`. ## Step 3: Test the Connection Verify the platform can reach your database. There are two flavors of verify: ```bash theme={null} # Verify an already-registered connection (idempotent, no body): curl -X POST \ "https:///api/assets/data//verify" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " ``` To verify a candidate connection **before** registration (e.g. as part of a UI wizard), `POST /api/assets/data/verify` with the same body shape as the create request — no `name`/`description` needed: ```bash theme={null} curl -X POST "https:///api/assets/data/verify" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " \ -H "Content-Type: application/json" \ -d '{ "connection_type": "postgres", "database_config": { "host": "db.example.com", "port": 5432, "database": "analytics" }, "credentials": { "username": "emergence_reader", "password": "..." } }' ``` Both flavors return the same shape: ```json theme={null} { "success": true, "message": "Connection verified", "latency_ms": 15.2, "details": { "server_version": "17.2", "schemas": ["public", "analytics"] } } ``` A failed verify returns `success: false` and a `message` describing the failure mode (`details` and `latency_ms` may be `null`). ## Step 4: Use the Connection with Data Insights Once registered, the connection's `resource_uri` (returned in Step 2) is what you pass to other platform services. The Data Insights solution consumes it from its chat UI / SDK; the underlying HTTP contract lives at `POST /api/talk2data/chat/messages` (paired with `/api/talk2data/chat/sessions` for session management — see the [Data Insights › Text-to-SQL guide](/data-insights/text-to-sql) for the full message-and-session model). The Data Insights UI handles session creation implicitly when a user opens a new chat against this connection. ## Connection Configuration Reference | Parameter | Type | Required | Description | | ---------- | ------- | -------- | --------------------------------------------------------------------------------- | | `host` | string | Yes | Database hostname or IP | | `port` | integer | No | Port number (default: 5432) | | `database` | string | Yes | Database name | | `schema` | string | No | Default schema (default: `public`) | | `ssl_mode` | string | No | SSL mode: `disable` (no TLS) or `require` (TLS, encrypt-only). Default `disable`. | | Parameter | Type | Required | Description | | ---------- | ------ | -------- | ----------------- | | `username` | string | Yes | Database username | | `password` | string | Yes | Database password | ## Security Considerations Credentials are stored via the platform Secrets API (backed by Infisical or ESO + GCP Secret Manager). They are encrypted at rest and organization-scoped. Credentials are injected at runtime only when a solution needs to establish a connection. Configure your database firewall to accept connections only from the platform's Kubernetes cluster CIDR. Use `ssl_mode: require` for production deployments so traffic is TLS-encrypted between the platform and your database. Certificate-pinning modes (`verify-ca`, `verify-full`) are not currently accepted by the Assets API. Always create a dedicated read-only user for the platform. Never use a superuser or an account with write privileges unless the solution explicitly requires write access. ## Troubleshooting Verify network connectivity between the Kubernetes cluster and your database. Check firewall rules, security groups, and `pg_hba.conf` configuration. Ensure the database host is resolvable from within the cluster. Confirm the username and password are correct. Check that the user exists in the target database and has the `CONNECT` privilege. Verify the `pg_hba.conf` entry allows the authentication method being used. If your database requires SSL, set `ssl_mode: require`. Certificate-pinning modes (`verify-ca`, `verify-full`) are not currently accepted by the Assets API — track this if your database requires server-cert verification at the client. ## Next Steps Learn how to ask natural-language questions against your connected database. Profile your connected database to understand data quality and structure. Learn how to back up your platform data including connection configurations. Review network security policies for data connections. # Register Your First Agent Source: https://docs.emergence.ai/guides/first-agent Step-by-step tutorial to register, configure, and test your first A2A agent on CRAFT. # Register Your First Agent CRAFT is an agent platform built on the A2A protocol. Every capability — from natural-language data queries to semiconductor yield analysis — runs as a discrete, independently deployable agent. This guide explains what you can build on CRAFT, who it is for, and where to find the detailed authoring documentation. ## What This Guide Covers This guide walks you through the final step of agent deployment: registering your agent in the CRAFT Assets registry. Once registered, your agent becomes discoverable to orchestrators and other platform consumers via the A2A protocol. Registration involves three steps: authoring an A2A-compliant Agent Card, validating it with the platform's dry-run endpoint, and posting it to the Assets API. You will also learn how to list, fetch, update, and delete registered agents. This tutorial is suitable for developers and operators who have already built and deployed an agent service that exposes an Agent Card at `/.well-known/agent-card.json`. If you have not built your agent yet, start with the framework-specific authoring guides below. ## Who This Guide Is For This guide is written for two audiences: * **Developers** deploying a new agent service to CRAFT for the first time, who need to register it so the orchestrator can discover and invoke it * **Operators** managing the agent registry — listing active agents, promoting lifecycle stages, or cleaning up retired registrations If you are building a new agent from scratch, you need prerequisites beyond what this guide covers: a framework choice, tool authoring, streaming implementation, and evaluation setup. ## Register Your Agent Once your agent service is running and its Agent Card is accessible, use the Assets API to register it on CRAFT. Obtain a JWT token from the platform identity provider to authenticate with the platform APIs. ```bash theme={null} TOKEN=$(curl -s -X POST \ "https:///realms//protocol/openid-connect/token" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=" \ -d "client_secret=" \ | jq -r '.access_token') ``` For local development, use `make get-token` in the em-runtime repository to obtain a token for the org admin user. Use the validation endpoint to check your Agent Card before registering it. This is a dry-run with no database write. ```bash theme={null} curl -X POST "https:///api/assets/agents:validate" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " \ -H "Content-Type: application/json" \ -d @agent-card.json ``` A valid card returns `{"valid": true, "errors": [], "warnings": []}`. ```bash theme={null} curl -i -X POST "https:///api/assets/agents" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " \ -H "Content-Type: application/json" \ -d '{ "agent_card": '"$(curl -s https://my-agent.example.com/.well-known/agent-card.json)"', "tags": ["demo"], "visibility": "TENANT_ONLY" }' ``` A successful registration returns `201 Created` with a `Location` header and `ETag`. The platform generates a `resource_uri` from the agent name, organization, and project: `agent:{org_id}:{project_id}:{name}`. ```bash theme={null} curl "https:///api/assets/agents?page=1&limit=20" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " ``` Your agent should appear in the listing with `status: ACTIVE` and `lifecycle_stage: EXPERIMENTAL`. ## Lifecycle Stages New registrations default to `EXPERIMENTAL`. The lifecycle stage tracks agent maturity. | Stage | Description | | -------------- | --------------------------------------- | | `EXPERIMENTAL` | Initial registration, under development | | `STABLE` | Production-ready, fully tested | | `DEPRECATED` | Scheduled for removal, still functional | | `RETIRED` | No longer available | Stage transitions are managed by the platform. `UpdateAgentRequest` accepts `agent_card`, `tags`, and `visibility` — not `lifecycle_stage` directly. ## Troubleshooting Verify your token has `can_create_resources` for the target project. Check that the `X-Project-ID` header matches a project you have access to. An agent with the same `name` already exists in the project. Choose a unique name or use `PUT` to update the existing registration. The Agent Card failed validation. Common causes: missing both `url` and `supportedInterfaces`, missing required `name`/`version` fields, or field length limits exceeded (255 chars for `name`, 2000 for `description`). Use `:validate` to surface specific errors. The `ETag` you sent no longer matches — another update happened in between. Fetch the agent again with `GET` to retrieve the latest `ETag`, then retry. ## Next Steps Learn about the full agent registry supporting A2A v0.3 and v1.0 protocols. Connect a database so your agents can query real data. Configure fine-grained permissions for your agents and projects. Explore the full API documentation for the Assets service. # Langfuse Setup Source: https://docs.emergence.ai/guides/langfuse-setup Step-by-step guide to deploying Langfuse and connecting it to all CRAFT solutions for LLM observability. # Langfuse Setup Guide This guide walks through deploying Langfuse and connecting it to your CRAFT solutions. Langfuse provides LLM call tracing, evaluation, and prompt management for Data Insights, Data Governance, and Semiconductor solutions. ## Prerequisites * A running CRAFT deployment (any version) * Docker and Docker Compose (for Docker deployment) or a Kubernetes cluster * Domain name for the Langfuse instance (recommended for production) ## Step 1: Deploy Langfuse ```bash theme={null} # Clone the Langfuse repository (provided in your deployment kit) git clone cd langfuse cp .env .env.local # Edit .env.local with your configuration ``` Edit `.env.local` with your configuration: ```env theme={null} # Required NEXTAUTH_URL=https://langfuse.your-domain.com NEXTAUTH_SECRET= SALT= # Database (auto-provisioned by compose) DATABASE_URL=postgresql://postgres:postgres@langfuse-db:5432/langfuse # Encryption ENCRYPTION_KEY= ``` Start the services: ```bash theme={null} docker compose up -d docker compose ps # verify all services are healthy ``` Langfuse is now available at `http://localhost:3000` (or your configured domain). ```bash theme={null} helm repo add langfuse https://langfuse.github.io/langfuse-k8s helm repo update # Create secrets kubectl create secret generic langfuse-secrets \ --namespace langfuse \ --from-literal=nextauth-secret=$(openssl rand -hex 32) \ --from-literal=salt=$(openssl rand -hex 32) \ --from-literal=encryption-key=$(openssl rand -hex 32) \ --from-literal=pg-password= # Install helm install langfuse langfuse/langfuse \ --namespace langfuse \ --create-namespace \ --set langfuse.nextauth.url="https://langfuse.example.com" \ --set langfuse.nextauth.existingSecret=langfuse-secrets \ --set langfuse.nextauth.existingSecretKey=nextauth-secret \ --set langfuse.salt.existingSecret=langfuse-secrets \ --set langfuse.salt.existingSecretKey=salt ``` ## Step 2: Create a Langfuse Project 1. Navigate to your Langfuse instance 2. Sign up for an account (or log in with the admin account) 3. Create a new **Project** for each solution you want to trace: * `data-insights`, for Data Insights traces * `data-governance`, for Data Governance traces * `semiconductor`, for Semiconductor traces 4. In each project, go to **Settings → API Keys** and copy the **Public Key** and **Secret Key** ## Step 3: Store Credentials Store Langfuse credentials as Kubernetes Secrets (or in your Secrets backend): ```bash theme={null} # Create one secret per solution (use separate Langfuse projects for isolation): kubectl create secret generic langfuse-api-keys \ --namespace em-runtime \ --from-literal=public-key= \ --from-literal=secret-key= kubectl create secret generic langfuse-api-keys-dr \ --namespace em-runtime \ --from-literal=public-key= \ --from-literal=secret-key= kubectl create secret generic langfuse-api-keys-semi \ --namespace em-runtime \ --from-literal=public-key= \ --from-literal=secret-key= ``` ## Step 4: Configure Solutions Set the Langfuse environment variables in each solution's Helm values: ### Data Insights ```yaml theme={null} # Helm values key for em-talk2data (Data Insights solution) em-talk2data: extraEnvVars: - name: LANGFUSE_HOST value: "https://langfuse.example.com" - name: LANGFUSE_PUBLIC_KEY valueFrom: secretKeyRef: name: langfuse-api-keys key: public-key - name: LANGFUSE_SECRET_KEY valueFrom: secretKeyRef: name: langfuse-api-keys key: secret-key ``` ### Data Governance Same configuration, referencing the data-governance project keys: ```yaml theme={null} # Helm values key for em-data-readiness (Data Governance solution) em-data-readiness: extraEnvVars: - name: LANGFUSE_HOST value: "https://langfuse.example.com" - name: LANGFUSE_PUBLIC_KEY valueFrom: secretKeyRef: name: langfuse-api-keys-dr key: public-key - name: LANGFUSE_SECRET_KEY valueFrom: secretKeyRef: name: langfuse-api-keys-dr key: secret-key ``` ### Semiconductor Apply to both the Backend API and AI Agent services: ```yaml theme={null} # Helm values key for em-semi (Semiconductor solution) em-semi: extraEnvVars: - name: LANGFUSE_HOST value: "https://langfuse.example.com" - name: LANGFUSE_PUBLIC_KEY valueFrom: secretKeyRef: name: langfuse-api-keys-semi key: public-key - name: LANGFUSE_SECRET_KEY valueFrom: secretKeyRef: name: langfuse-api-keys-semi key: secret-key ``` ## Step 5: Verify Traces After deploying the configuration, trigger an LLM call in any solution: 1. In Data Insights: ask a natural language question 2. In Data Governance: run a profiling job 3. In Semiconductor: send a chat message to the AI agent Then check the **Traces** tab in Langfuse, you should see the call appear within seconds. ## Troubleshooting 1. Verify `LANGFUSE_HOST`, `LANGFUSE_PUBLIC_KEY`, and `LANGFUSE_SECRET_KEY` are set correctly in the pod 2. Check that the Langfuse host is reachable from the pod: `kubectl exec -it -- curl -I https://langfuse.example.com` 3. Check solution logs for Langfuse callback errors 4. Verify the public key corresponds to the correct Langfuse project 1. Check pod status: `kubectl get pods -n langfuse` 2. Check worker logs: `kubectl logs -n langfuse -l app=langfuse-worker` 3. Verify the `NEXTAUTH_URL` matches the URL you're accessing # AWS Marketplace Source: https://docs.emergence.ai/guides/marketplace/aws Deploy CRAFT from the AWS Marketplace using private offers, container products, and EDP committed spend drawdown. # AWS Marketplace CRAFT is available on the **AWS Marketplace** as a Container Product. Customers can deploy the platform into their own EKS clusters and apply purchases against their AWS Enterprise Discount Program (EDP) commitments. ## Why AWS Marketplace Customers apply CRAFT purchases against their AWS Enterprise Discount Program (EDP) commitments, removing budget approval friction. Private offers enable AWS field teams to co-sell, earning quota credit and aligning sales incentives. Marketplace-transacted deals are 112% larger on average than direct sales, according to industry research. 3% fee for contract-based pricing, 1.5-3% for private offers. Contract-based pricing is recommended to avoid higher hourly/monthly fees. ## Architecture CRAFT's cloud-agnostic Helm chart deploys the same platform services on EKS as on GKE. The AWS-specific work is scoped to infrastructure modules and marketplace packaging: | Component | Source | | ------------------------- | ------------------------------------------------------------------ | | **Application chart** | `em-runtime` Helm chart (cloud-agnostic) | | **Infrastructure module** | EKS Terraform module (to be built as part of marketplace strategy) | | **Container images** | Published to Amazon ECR via CI/CD multi-registry pipeline | | **Marketplace packaging** | Container product with Helm charts and container images | ## Deployment Model Private offers are managed through the AWS Marketplace Management Portal (AMMP). The sales team configures pricing using contract-based pricing (recommended) with custom dimensions and generates a private offer for the target customer. The customer receives the private offer link, reviews pricing and EULA terms, and subscribes through their AWS Console. Charges apply against their EDP commitment. After subscription, deploy the platform into the customer's EKS cluster using the Terraform module and Helm chart provided with the listing. In the self-service model, the listing includes a CloudFormation template that provisions: * An EKS cluster (or targets an existing one) * Amazon RDS for PostgreSQL (9 databases) * Amazon ElastiCache for Redis * IAM roles and policies for IRSA (IAM Roles for Service Accounts) * Helm chart installation with marketplace-specific values ## Billing Integration AWS billing uses the **MeterUsage** and **RegisterUsage** APIs for consumption tracking. ### Pricing Model AWS Marketplace requires **contract-based pricing** for the most favorable fee structure: | Model | Fee | Mechanism | | -------------------------- | --------------- | ----------------------------------------------------- | | **Contract** | 3% | Customer commits to a term (monthly/annual) | | **Contract + Consumption** | 3% base + usage | Contract base plus metered overage via MeterUsage API | | **Private Offer** | 1.5-3% | Custom pricing per customer | ### Metered Dimensions | Dimension | Unit | API | | ---------------------- | ----- | ------------ | | `agents_registered` | count | `MeterUsage` | | `api_requests` | count | `MeterUsage` | | `data_transfer_out_gb` | GB | `MeterUsage` | | `health_checks` | count | `MeterUsage` | AWS enforces stricter dimension naming conventions than GCP or Azure. Dimension names must use snake\_case and match exactly between the listing definition and the metering API calls. ## Container Image Publishing The CI/CD pipeline publishes images to Amazon ECR as part of the multi-registry strategy: ```yaml theme={null} # ci-main.yml excerpt - name: Push to ECR run: | aws ecr get-login-password --region us-east-1 | \ docker login --username AWS --password-stdin $ECR_REGISTRY docker tag emergence/governance:$VERSION $ECR_REGISTRY/emergence/governance:$VERSION docker push $ECR_REGISTRY/emergence/governance:$VERSION ``` The following images are published: * 3 production services (Governance, Assets, Utils) * 5 subchart images (Keycloak, OpenFGA, PostgreSQL, Redis, Infisical (or ESO)) * 4 init container images (schema migration, bootstrap, health-check, seed data) ## Prerequisites for Deployment Before deploying from AWS Marketplace: 1. **EKS cluster** -- An existing EKS cluster (1.28+) with managed node groups or Fargate profiles 2. **Amazon RDS** -- PostgreSQL 17+ instance with 9 databases 3. **ElastiCache** -- Redis 7 cluster for caching and event streaming 4. **IAM** -- IRSA (IAM Roles for Service Accounts) configured for service-to-AWS authentication 5. **Networking** -- VPC with private subnets and security groups allowing cluster-to-database connectivity ## Comparison with GCP and Azure | Aspect | AWS | GCP | Azure | | ------------------- | ---------------------------- | ----------------------- | ------------------------- | | **Offer Type** | Container Product | Terraform K8s App | Container Offer (K8s App) | | **Packaging** | Container images + Helm | Helm + Terraform module | CNAB bundle | | **Commit Drawdown** | EDP | MCPO | MACC | | **Fee Structure** | 3% contract / 1.5-3% private | 3% / 1.5% renewals | \~3% flat | | **Complexity** | Moderate | Lowest | Highest | ## Troubleshooting Verify the EKS node role has the `ecr:GetDownloadUrlForLayer`, `ecr:BatchGetImage`, and `ecr:GetAuthorizationToken` permissions. Ensure the ECR repository policy allows cross-account access if the customer's EKS cluster is in a different AWS account. Confirm the product code matches the listing ID in AMMP. Ensure the metering calls originate from within the customer's AWS account using IRSA credentials. Verify the customer's EDP agreement covers AWS Marketplace purchases. Some older EDP agreements exclude marketplace spend -- the customer should contact their AWS account team. ## Next Steps Learn about the GCP Marketplace listing and GKE deployment. Learn about the Azure Marketplace listing and AKS deployment. # Azure Marketplace Source: https://docs.emergence.ai/guides/marketplace/azure Deploy CRAFT from the Azure Marketplace using private offers, CNAB bundles, and MACC committed spend drawdown. # Azure Marketplace CRAFT is available on the **Azure Marketplace** as a Container Offer (Kubernetes Application). Customers can deploy the platform into their own AKS clusters and apply purchases against their Microsoft Azure Consumption Commitment (MACC). ## Why Azure Marketplace Customers apply CRAFT purchases against their Microsoft Azure Consumption Commitment (MACC), eliminating budget approval friction for Azure-committed enterprises. Private offers through Partner Center enable Microsoft field teams to co-sell, providing priority access to enterprise accounts. Purchases appear on the customer's existing Azure invoice with no separate vendor billing. Azure Marketplace listings undergo Microsoft certification, providing additional trust signals for enterprise procurement. ## Architecture The Azure listing packages the same cloud-agnostic `em-runtime` Helm chart into a CNAB (Cloud Native Application Bundle) format required by Azure Marketplace: | Component | Source | | ------------------------- | ------------------------------------------------------------------ | | **Application chart** | `em-runtime` Helm chart (cloud-agnostic) | | **Packaging format** | CNAB bundle (Helm + ARM template + createUIDef) | | **Infrastructure module** | AKS Terraform module (to be built as part of marketplace strategy) | | **Container images** | Published to Azure Container Registry (ACR) via CI/CD | | **Partner management** | Azure Partner Center | Azure Marketplace has the highest packaging complexity of the three cloud marketplaces due to the CNAB bundle requirement. The bundle must include an ARM template and a `createUIDef.json` for the deployment UI. ## Deployment Model Private offers are managed through Azure Partner Center. The sales team configures pricing, selects target Azure tenants or subscription IDs, and generates a private offer with custom terms and pricing. The customer receives the private offer notification in the Azure Portal, reviews terms, and subscribes. Charges apply against their MACC commitment. After subscription, the platform is deployed into the customer's AKS cluster using the CNAB bundle, which installs the Helm chart with marketplace-specific values. The self-service deployment uses the CNAB bundle with a `createUIDef.json` that provides: * A guided deployment wizard in the Azure Portal * AKS cluster selection or provisioning * Azure Database for PostgreSQL Flexible Server configuration * Azure Cache for Redis provisioning * Managed Identity configuration for pod-to-Azure-service authentication ## CNAB Bundle Structure The Azure Container Offer requires a CNAB bundle with these components: ```text theme={null} cnab/ bundle.json # CNAB manifest referencing Helm chart and images app/ Chart.yaml # em-runtime Helm chart metadata values.yaml # Marketplace-specific defaults templates/ # Helm templates arm/ mainTemplate.json # ARM template for Azure resource provisioning createUIDef.json # Deployment UI definition for Azure Portal ``` ### createUIDef.json The `createUIDef.json` defines the deployment wizard that customers see in the Azure Portal: * **Basics**: Subscription, resource group, region selection * **AKS Configuration**: Cluster selection or creation parameters * **Database**: Azure Database for PostgreSQL connection settings * **Cache**: Azure Cache for Redis connection settings * **Identity**: Managed Identity configuration for Keycloak and secrets backend (Infisical or ESO) ## Billing Integration Azure billing uses the **Metering Service API** for consumption tracking. ### Pricing Model | Model | Fee | Mechanism | | ----------------- | ---------- | -------------------------------------------- | | **Flat rate** | \~3% | Monthly or annual subscription | | **Custom meters** | \~3% | Per-dimension usage via Metering Service API | | **Private Offer** | Negotiable | Custom pricing per customer | ### Metered Dimensions | Dimension | Unit | Description | | ------------------- | ----- | ------------------------------ | | `agentsRegistered` | count | Active A2A agent registrations | | `apiRequests` | count | Total API requests | | `dataTransferOutGB` | GB | Data egress from the platform | | `healthChecks` | count | Health check probes performed | ## Container Image Publishing The CI/CD pipeline publishes images to Azure Container Registry (ACR): ```bash theme={null} # Authenticate to ACR az acr login --name emergenceai # Tag and push images docker tag emergence/governance:$VERSION emergenceai.azurecr.io/emergence/governance:$VERSION docker push emergenceai.azurecr.io/emergence/governance:$VERSION ``` All 12+ container images (3 production + 5 subchart + 4 init) are published to ACR as part of the multi-registry CI/CD pipeline. ## Prerequisites for Deployment Before deploying from Azure Marketplace: 1. **AKS cluster** -- An existing AKS cluster (1.28+) with system and user node pools 2. **Azure Database for PostgreSQL** -- Flexible Server with PostgreSQL 17+ and 9 databases 3. **Azure Cache for Redis** -- Redis 7 instance for caching and event streaming 4. **Managed Identity** -- User-assigned managed identity for pod-to-service authentication 5. **Networking** -- VNet with AKS integration and private endpoints for database and cache ## Partner Center Certification Azure Marketplace requires additional certification steps: | Requirement | Description | | ------------------------ | ------------------------------------------------------------ | | **Technical validation** | Microsoft validates the CNAB bundle deploys correctly to AKS | | **Security review** | Container images are scanned for vulnerabilities | | **Documentation** | Deployment guide and support documentation required | | **Support plan** | SLA and support contact information must be provided | Partner Center certification can take 2-4 weeks. Plan for this timeline when targeting Azure Marketplace availability. ## Troubleshooting Verify the ARM template in `mainTemplate.json` is valid using `az deployment group validate`. Ensure the `createUIDef.json` schema matches the ARM template parameters. Verify the AKS cluster has the `AcrPull` role assignment on the ACR instance. For cross-subscription deployments, ensure the managed identity has appropriate permissions. Confirm the offer is MACC-eligible in Partner Center. The customer's MACC agreement must include Azure Marketplace purchases -- older agreements may not include this benefit. ## Next Steps Learn about the GCP Marketplace listing and GKE deployment. Learn about the AWS Marketplace listing and EKS deployment. # GCP Marketplace Source: https://docs.emergence.ai/guides/marketplace/gcp Deploy CRAFT from the Google Cloud Marketplace using private offers, Terraform Kubernetes apps, and integrated billing. # GCP Marketplace CRAFT is available on the **Google Cloud Marketplace** as a Terraform Kubernetes Application. Customers can deploy the platform into their own GKE clusters and apply purchases against existing GCP committed spend. ## Why GCP Marketplace Apply CRAFT purchases against existing GCP committed spend via the Marketplace Commerce Platform for Operators (MCPO), with a 25% cap for channel deals. Pre-approved vendor status on GCP Marketplace eliminates lengthy procurement cycles -- up to 4 weeks faster than traditional vendor onboarding. Customers receive one consolidated GCP bill rather than a separate vendor invoice. Standard marketplace fee of 3%, with renewal rates as low as 1.5%. ## Architecture Advantage CRAFT is natively deployed on GKE Autopilot in production. The GCP Marketplace listing leverages the same Helm chart (`em-runtime`) and Terraform modules used for direct deployments, ensuring consistency: | Component | Source | | ------------------------- | ------------------------------------------------------------------- | | **Application chart** | `em-runtime` Helm chart (cloud-agnostic) | | **Infrastructure module** | GKE Terraform module (provided in deployment kit) | | **Container images** | Published to Google Artifact Registry via CI/CD | | **Configuration** | Environment-specific Helm values files (provided in deployment kit) | ## Deployment Model Private offers allow the sales team to send a pre-configured pricing link directly to target customers. No public storefront is required. Using the GCP Producer Portal, the sales team configures pricing (subscription, usage-based, or hybrid) and generates a private offer link. The customer clicks the private offer link, reviews pricing, and accepts within their GCP console. The purchase is applied against their GCP committed spend. After acceptance, the customer or the Emergence team deploys the platform into the customer's GKE cluster using the provided Terraform module. In the self-service model, customers click "Deploy" in the GCP Marketplace listing and the platform deploys automatically into their GKE cluster. The deployment uses: * A `schema.yaml` defining customer-configurable parameters * Terraform modules for GKE cluster, Cloud SQL, Memorystore, and GCS provisioning * Helm chart installation with marketplace-specific values * Workload Identity for secure service-to-service authentication ## Billing Integration GCP billing uses a metering agent sidecar that reports usage metrics to the GCP Marketplace Billing API. ### Metered Dimensions Metering dimensions are not yet finalized. The dimensions below are candidates under review by the product team. Final dimensions will be aligned with customer value metrics before the billing integration goes live. | Candidate Dimension | Unit | Description | Status | | --------------------- | ----- | -------------------------------------------------- | ------------ | | Active users | count | Active platform users per billing period | Under review | | API calls | count | Total API requests (aggregate or by service) | Under review | | Data volume processed | bytes | Total data volume processed by the platform | Under review | | Managed nodes | count | Number of managed infrastructure nodes or clusters | Under review | ### Pricing Model The GCP listing supports a hybrid pricing model: * **Base subscription**: Monthly or annual fee for platform access * **Usage-based metering**: Additional charges based on consumption of metered dimensions * **Private offer pricing**: Custom pricing negotiated per customer via the Producer Portal ## Container Image Publishing The CI/CD pipeline publishes container images to Google Artifact Registry for the following components: | Category | Images | | ------------------------- | -------------------------------------------------------- | | **Production services** | Governance, Assets, Utils | | **Subchart dependencies** | Keycloak, OpenFGA, PostgreSQL, Redis, Infisical (or ESO) | | **Init containers** | Schema migration, bootstrap, health-check | The `ci-main.yml` pipeline builds and pushes images to Artifact Registry on every merge to main. Pre-release images are built from feature branches via `ci-branches.yml`. ## Prerequisites for Deployment Before deploying from GCP Marketplace: 1. **GKE cluster** -- An existing GKE Autopilot or Standard cluster (1.28+) 2. **Cloud SQL** -- PostgreSQL 17+ instance with 9 databases (one per service) 3. **Memorystore** -- Redis 7 instance for caching and event streaming 4. **IAM** -- Workload Identity configured for service-to-GCP-service authentication 5. **Networking** -- VPC with private service access for Cloud SQL and Memorystore ## Troubleshooting Verify the deploying user has the `roles/container.admin` and `roles/iam.serviceAccountUser` roles. Workload Identity requires the service account to be bound to the Kubernetes service account. Check that the metering agent sidecar is running in the Governance pod. Verify the `MARKETPLACE_METERING_ENABLED` environment variable is set to `true` and the service account has the `roles/servicemanagement.reporter` role. Ensure the private offer was created in the correct GCP Producer Portal project and the customer's billing account ID matches the offer target. ## Next Steps Learn about the AWS Marketplace listing and EKS deployment. Learn about the Azure Marketplace listing and AKS deployment. Configure Helm values for GKE deployment. # Memory Integration Source: https://docs.emergence.ai/guides/memory-integration How-to guide for integrating the Memory Service into your agents, creating and managing memories via the Platform Utils API. # Memory Integration This guide shows how to integrate memory into your agents and workflows. Memory lets agents remember information across sessions, accumulate domain knowledge, and personalize responses. Memory is built into the **Utils service** (`em-runtime-utils`) — the same service that handles schedules, metadata, and catalog tags. No separate service is required. ## Prerequisites * A valid JWT token from the platform (user or service account) * The JWT must contain an `org_id` claim for organization scoping * An `X-Project-ID` header for project scoping on all memory requests ## Basic Integration Pattern The most common pattern for an agent: 1. **Before responding**, list memories from the relevant Context Pack for context 2. **After responding**, extract and store new memories from the conversation ```python theme={null} import httpx UTILS_BASE = "https://api.example.com/utils" async def agent_respond(user_message: str, token: str, pack_name: str, project_id: str) -> str: headers = { "Authorization": f"Bearer {token}", "X-Project-ID": project_id, } async with httpx.AsyncClient() as client: # Step 1: List memories from the relevant Context Pack mem_resp = await client.get( f"{UTILS_BASE}/context-packs/{pack_name}/memories", headers=headers, ) memories = mem_resp.json().get("data", []) # Step 2: Build context from memories context = "\n".join(f"- {m['content']}" for m in memories) # Step 3: Call LLM with memory context prompt = f"Relevant context:\n{context}\n\nUser: {user_message}" response = await call_llm(prompt) # Step 4: Store new memories from this interaction if should_memorize(user_message, response): await client.post( f"{UTILS_BASE}/context-packs/{pack_name}/memories", headers=headers, json={ "name": "session_fact", "content": { "type": "text", "text": extract_memorable_fact(user_message, response), }, "memory_type": "fact", "source": {"type": "agent"}, }, ) return response ``` ## Storing Memories The following snippets show individual API calls. In an async agent, wrap them in `async with httpx.AsyncClient() as client:` and use `await client.post(...)` / `await client.get(...)` as shown in the integration pattern above. ### Basic Memory Creation Memories are created within a named Context Pack: ```python theme={null} # Store a user preference in a context pack await client.post( f"{UTILS_BASE}/context-packs/{pack_name}/memories", headers=headers, json={ "name": "metric_explanation_style", "content": { "type": "text", "text": "User prefers metric definitions before SQL queries", }, "memory_type": "preference", "source": {"type": "user"}, }, ) ``` ### Memory Types Choose the type based on the nature of the information: | Type | When to use | Example | | ----------------- | ----------------------- | -------------------------------------------------------- | | `fact` | Factual knowledge | "The `orders` table has 50M rows" | | `experience` | Learned behaviors | "Query optimization worked better with CTEs" | | `observation` | Observed patterns | "Revenue spikes every Q4" | | `instruction` | Directives / guidelines | "Always include date filters for large tables" | | `preference` | User/system preferences | "Prefers concise output without verbose explanations" | | `summary` | Condensed summaries | "Session covered revenue analysis across 3 regions" | | `glossary` | Business terms | "ARR = Annual Recurring Revenue" | | `join` | Table relationships | "Table `orders` has FK to `customers`" | | `ontology` | Domain classifications | "Column `status` uses enum: active, inactive, churned" | | `policy` | Governance rules | "PII columns must not appear in SQL results" | | `textual_pattern` | Text patterns in data | "Email column matches a standard email regex pattern" | | `kpi` | KPI definitions | "Churn Rate = lost customers / total customers x 100" | | `exemplar` | Representative examples | "Typical order: id=1234, amount=49.99, status=completed" | | `numeric_pattern` | Numeric distributions | "Revenue column: mean=1250, median=980, stddev=450" | ### Named Memories Use `name` for memories you want to retrieve directly by name within a pack: ```python theme={null} # Store with a name for direct retrieval await client.post( f"{UTILS_BASE}/context-packs/{pack_name}/memories", headers=headers, json={ "name": "user_data_source_primary", "content": { "type": "text", "text": "User's primary data source is Snowflake, database: analytics_prod", }, "memory_type": "fact", "source": {"type": "user"}, }, ) # Retrieve by name within a pack response = await client.get( f"{UTILS_BASE}/context-packs/{pack_name}/memories/user_data_source_primary", headers=headers, ) # Retrieve by memory ID response = await client.get( f"{UTILS_BASE}/memories/{memory_id}", headers=headers, ) ``` ## Retrieving Memories ### List All Memories in a Context Pack ```python theme={null} response = await client.get( f"{UTILS_BASE}/context-packs/{pack_name}/memories", headers=headers, ) memories = response.json().get("data", []) for memory in memories: print(f"[{memory['memory_type']}] {memory['content']}") ``` ### List All Memories in the Project ```python theme={null} response = await client.get( f"{UTILS_BASE}/memories", headers=headers, ) all_memories = response.json() # {"data": [...], "pagination": {...}} ``` ## Context Packs For structured memory management, use Context Packs to group related memories by domain: ```python theme={null} # Create a domain knowledge pack pack_resp = await client.post( f"{UTILS_BASE}/context-packs", headers=headers, json={ "name": "user_data_domain", "pack_type": "domain", "description": "User's data domain knowledge: schema, metrics, business rules", }, ) pack_name = pack_resp.json()["name"] # Add a memory to the pack await client.post( f"{UTILS_BASE}/context-packs/{pack_name}/memories", headers=headers, json={ "name": "revenue_metric_definition", "content": { "type": "text", "text": "Revenue metric = sum of completed order amounts, excluding refunds", }, "memory_type": "fact", "source": {"type": "agent"}, }, ) # Retrieve all memories from the pack response = await client.get( f"{UTILS_BASE}/context-packs/{pack_name}/memories", headers=headers, ) memories = response.json().get("data", []) ``` ## Updating and Deleting ### Update a Memory ```python theme={null} # When a fact changes, update rather than delete await client.patch( f"{UTILS_BASE}/memories/{memory_id}", headers=headers, json={ "content": { "type": "text", "text": "User's primary data source is BigQuery, project: analytics_prod (migrated from Snowflake)", }, }, ) ``` ### Delete a Memory ```python theme={null} # Soft delete (recoverable) await client.delete( f"{UTILS_BASE}/memories/{memory_id}", headers=headers, ) # Permanent delete (irreversible) await client.post( f"{UTILS_BASE}/memories/{memory_id}/permanent-delete", headers=headers, ) ``` ### Pin a Memory Pinned memories stay in the Active tier and bypass inactivity decay: ```python theme={null} await client.post( f"{UTILS_BASE}/memories/{memory_id}/pin", headers=headers, ) ``` ## Best Practices Each memory should make sense without knowing the conversation that produced it. Instead of "Yes, that's right" store "User confirmed that the 'revenue' column uses gross revenue, not net." Typed memories retrieve more accurately. If you know a memory is a preference, mark it `preference`, use specific types for better retrieval. Before creating a new memory, list the pack's memories and check whether an existing one covers the same information. Update existing memories rather than creating duplicates. For domain-specific knowledge bases (data dictionaries, business rules, entity relationships), group memories into typed Context Packs. This improves retrieval precision and enables pack-level operations. Memories are scoped by the `org_id` claim in the JWT token and the `X-Project-ID` header. Ensure each agent uses a token scoped to the correct organization and passes the correct project ID to prevent information leakage. ## Related Concepts overview and API reference. Detailed Context Pack architecture and management. # RBAC Configuration Source: https://docs.emergence.ai/guides/rbac-configuration How to set up OpenFGA roles and permissions for fine-grained access control across organizations, projects, and resources. # Configure Roles and Permissions CRAFT uses **OpenFGA** for Relationship-Based Access Control (ReBAC). This guide covers setting up roles, assigning permissions, and configuring the permission inheritance model across organizations, projects, and resources. ## Prerequisites Before you begin, ensure you have: * A running CRAFT with the Governance service bootstrapped * Administrative access (the `admin` or `owner` role on the target organization) * A valid JWT token for the admin user * The OpenFGA schema loaded during Governance bootstrap ## How Permissions Work The platform uses a **relationship-based** model rather than traditional role-based access control. Permissions are computed at query time from relationships between users, roles, and resources. ```text theme={null} User --[role]--> Organization --[parent]--> Project --[parent]--> Resource ``` ### Role Hierarchy | Role | Scope | Capabilities | | ----------- | ---------------------- | ------------------------------------------------------------------------ | | `owner` | Organization | Full control, can transfer ownership, manage all settings | | `admin` | Organization / Project | Manage users, projects, settings. Cannot transfer organization ownership | | `member` | Project | Standard access to project resources | | `developer` | Project | Create and modify agents, data connections, workflows | | `operator` | Project | Deploy, schedule, and monitor resources | | `viewer` | Project | Read-only access to all resources | ### Computed Permissions Permissions are derived from roles via the OpenFGA schema: | Permission | owner | admin | member | developer | operator | viewer | | -------------------- | ----- | ----- | ------ | --------- | -------- | ------ | | `can_read` | Yes | Yes | Yes | Yes | Yes | Yes | | `can_write` | Yes | Yes | Yes | Yes | No | No | | `can_delete` | Yes | Yes | No | No | No | No | | `can_execute` | Yes | Yes | Yes | Yes | Yes | No | | `can_manage_secrets` | Yes | Yes | No | No | No | No | ## Step 1: Understand the OpenFGA Schema The permission model is defined in the [OpenFGA Schema DSL](https://openfga.dev/docs/configuration-language). Key type definitions used by the platform: ```text theme={null} type organization relations define owner: [user] define admin: [user] or owner define member: [user] or admin type project relations define parent: [organization] define admin: [user] or admin from parent define developer: [user] or admin define operator: [user] or admin define viewer: [user] or developer or operator define can_read: viewer or member from parent define can_write: developer or admin define can_delete: admin define can_execute: operator or developer or admin ``` ## Step 2: Grant a role on a resource The Governance API exposes a single grant endpoint that is used for every role-on-resource combination — there is no separate "add organization member" or "add project member" endpoint. A "role assignment" is a tuple `(user_or_group, relation, resource_type:resource_id)` written to the OpenFGA store via `POST /governance/permissions/grant`. ```bash theme={null} # Grant the 'admin' relation to a user on an organization curl -X POST "https:///api/governance/permissions/grant" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " \ -H "Content-Type: application/json" \ -d '{ "user_or_group": "", "relation": "admin", "resource_type": "organization", "resource_id": "" }' ``` Response (`200 OK`): ```json theme={null} { "message": "Granted admin permission to user '' on organization ''" } ``` The same endpoint also assigns project-scoped roles — change `resource_type` to `project` and `resource_id` to the project UUID: ```bash theme={null} curl -X POST "https:///api/governance/permissions/grant" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " \ -H "Content-Type: application/json" \ -d '{ "user_or_group": "", "relation": "developer", "resource_type": "project", "resource_id": "" }' ``` Same shape for any resource type: `artifact`, `data_connection`, `secret`, etc. The `relation` must be one of the role names defined on that resource type in the OpenFGA schema (Step 1). The auto-generated client is an [openapi-python-client](https://github.com/openapi-generators/openapi-python-client) package — module-level functions per endpoint, called against an `AuthenticatedClient` instance. There is no nested `client.permissions.grant(...)` method. ```python theme={null} from em_runtime_governance_sdk import AuthenticatedClient from em_runtime_governance_sdk.api.permissions import grant_permission from em_runtime_governance_sdk.models import GrantPermissionRequest client = AuthenticatedClient( base_url="https:///api/governance", token=admin_token, ) # Grant on an organization await grant_permission.asyncio( client=client, body=GrantPermissionRequest( user_or_group=user_id, relation="admin", resource_type="organization", resource_id=org_id, ), ) # Grant on a project — same call, different resource_type/resource_id await grant_permission.asyncio( client=client, body=GrantPermissionRequest( user_or_group=user_id, relation="developer", resource_type="project", resource_id=project_id, ), ) ``` The same pattern applies to the other permission endpoints — import `revoke_permission`, `check_permission`, `list_accessible_objects`, `set_resource_parent`, or `delete_resource_permissions` from the same `em_runtime_governance_sdk.api.permissions` module. ### Permission Inheritance Roles inherit downward through the hierarchy because the OpenFGA schema expresses each role on a child as derivable from its parent: Automatically has admin-level permissions on all projects and resources within the organization. Automatically has admin-level permissions on all projects. Can create and delete projects. Has `can_read`, `can_write`, and `can_execute` on all resources within the specific project. Cannot delete resources or manage secrets. Has `can_read` only. Cannot modify, execute, or delete any resource. When you create a child resource (e.g. an artifact within a project), register the parent relationship so inheritance applies: ```bash theme={null} curl -X POST "https:///api/governance/permissions/set-parent" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " \ -H "Content-Type: application/json" \ -d '{ "resource_type": "artifact", "resource_id": "", "parent_type": "project", "parent_id": "" }' ``` ## Step 3: Check or list permissions Two read endpoints answer the common questions: ```bash theme={null} # 1) Can perform on ? # Note: this is a GET with query params — NOT a POST with a body. curl -G "https:///api/governance/permissions/check" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " \ --data-urlencode "action=can_write" \ --data-urlencode "resource_type=agent" \ --data-urlencode "resource_id=" ``` Response (`200 OK`): the response body for an allowed call is `null` (absence of a `403` body means the action is permitted). A denied check returns `403 Forbidden`. ```bash theme={null} # 2) Which objects can read across every resource type? curl "https:///api/governance/permissions/accessible-objects" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " ``` Response (`200 OK`): ```json theme={null} { "object_ids": [ "project:", "data_connection:data:::", "agent:agent:::" ] } ``` Each entry is `:`, with `resource_id` echoing the canonical `resource_uri` for `agent` / `data_connection` / `artifact` resources. ## Step 4: Revoke a role `POST /governance/permissions/revoke` mirrors `grant`: ```bash theme={null} curl -X POST "https:///api/governance/permissions/revoke" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " \ -H "Content-Type: application/json" \ -d '{ "user_or_group": "", "relation": "developer", "resource_type": "project", "resource_id": "" }' ``` Response (`200 OK`): ```json theme={null} { "message": "Revoked developer permission from user '' on project ''" } ``` To delete every permission tuple for a resource (e.g. before deleting the resource itself), use `POST /governance/permissions/delete-all`: ```bash theme={null} curl -X POST "https:///api/governance/permissions/delete-all" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: " \ -H "Content-Type: application/json" \ -d '{ "resource_type": "project", "resource_id": "" }' ``` Response includes `deleted_count` so you can confirm the sweep. ## Common Permission Patterns Assign `viewer` on the project containing data connections and dashboards. Users can run queries via Data Insights but cannot modify data connections or agent configurations. Assign `operator` on the target project. Operators can deploy agents, trigger schedules, and monitor health, but cannot modify agent code or data connection credentials. Create Keycloak client credentials and assign `developer` or `operator` on the target project. Service accounts authenticate via client credentials grant and have the same permission model as human users. A user can have different roles on different projects within the same organization. Assign roles per project to implement least-privilege access. ## Integration with SSO Groups When SSO is configured, IdP groups map through Keycloak to OpenFGA: ```text theme={null} IdP Group -> Keycloak Group -> OpenFGA Relation (role assignment) ``` This pipeline automates role assignment when users are provisioned via SSO. See the [SSO Integration guide](/guides/sso-integration) for configuring the IdP side. ## Troubleshooting Verify the OpenFGA schema is loaded by checking Governance bootstrap logs. Ensure the user's JWT contains the correct groups claim. Confirm the user has been assigned a role on the target project. Check that the project has a `parent` relationship to the organization in OpenFGA. This is set automatically when projects are created via the Governance API. OpenFGA evaluates permissions at query time -- there is no caching delay. If changes are not reflected, verify the role assignment was successful by listing members on the project. ## Next Steps Configure SSO to automate group-to-role mapping from your IdP. Learn about the full OpenFGA schema and computed permissions model. Understand how permissions integrate with the multi-tenant architecture. Review the complete security model including authentication and data protection. # Access LLMs Source: https://docs.emergence.ai/guides/solution-dev/access-llms Call LLMs from your solution through the platform's LiteLLM gateway — provider-agnostic, observable via Langfuse, with project-scoped cost attribution. # Access LLMs Solutions never call OpenAI, Anthropic, or Gemini directly. Instead, point an OpenAI-compatible SDK at the platform's **LiteLLM gateway**. You get provider-agnostic access, central API-key management, per-project cost attribution, rate-limit enforcement, and Langfuse traces — for free. The canonical reference implementation is in [Data Insights › Text-to-SQL](/data-insights/text-to-sql); this page extracts the patterns every solution needs. ## Why a gateway | Without a gateway | With LiteLLM gateway | | ------------------------------------------------------- | ------------------------------------ | | Each solution holds its own API keys | One key per provider, held centrally | | Switching from OpenAI → Anthropic = code change | Switching is an env var change | | Per-solution cost attribution requires custom telemetry | Free via gateway metadata + Langfuse | | Rate limits enforced per-key (poorly) | Enforced centrally, per project | | Adding observability requires per-call wrapping | Auto-instrumented via Langfuse hook | ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR SOL["Your solution
litellm.acompletion(...)"] META["metadata=
{project_id, solution}"] SOL -.includes.-> META GW["LiteLLM gateway
(provider-agnostic)"] SOL -->|"OpenAI-compatible
HTTPS"| GW GW --> OPENAI[("OpenAI")] GW --> ANTH[("Anthropic")] GW --> GEMINI[("Google Gemini")] GW --> OTHER[("...other
providers")] LANGFUSE[("Langfuse
(traces + cost)")] GW -.auto-traces.-> LANGFUSE META -.attributes spend by.-> LANGFUSE classDef sol fill:#56B4E9,stroke:#555555,color:#000 classDef infra fill:#0072B2,stroke:#555555,color:#fff classDef ext fill:#CC79A7,stroke:#555555,color:#000 classDef act fill:#E69F00,stroke:#555555,color:#000 classDef store fill:#009E73,stroke:#555555,color:#000 class SOL sol class GW infra class OPENAI,ANTH,GEMINI,OTHER ext class META act class LANGFUSE store ``` **Read the diagram**: your solution calls one OpenAI-compatible endpoint; the gateway routes by `provider/model` to the chosen upstream and auto-traces every call into Langfuse. The `metadata={"project_id": ..., "solution": ...}` you pass on every call is what makes per-project cost attribution possible. ## Configure your service Three env vars (plus Langfuse credentials when you want tracing): ```yaml charts//values.yaml theme={null} api: env: LLM_GATEWAY_URL: "https://litellm.example.com/v1" # platform operator provides this LLM_DEFAULT_MODEL: "openai/gemini-3.5-flash" envVars: - name: LLM_GATEWAY_API_KEY valueFrom: { secretKeyRef: { name: -secrets, key: llm-gateway-api-key } } # Observability (Langfuse) — see /guides/langfuse-setup - name: LANGFUSE_HOST valueFrom: { configMapKeyRef: { name: -langfuse, key: host } } - name: LANGFUSE_PUBLIC_KEY valueFrom: { secretKeyRef: { name: -secrets, key: langfuse-public-key } } - name: LANGFUSE_SECRET_KEY valueFrom: { secretKeyRef: { name: -secrets, key: langfuse-secret-key } } ``` The gateway URL and API key live in your secrets pipeline (see [Manage Secrets](/guides/solution-dev/manage-secrets)). ## Code patterns ### Minimal call (litellm) `litellm` speaks the OpenAI completions wire format and works against any compatible endpoint. Install with `uv add litellm`. ```python packages/api/src/api/llm.py theme={null} import os import litellm litellm.api_base = os.environ["LLM_GATEWAY_URL"] litellm.api_key = os.environ["LLM_GATEWAY_API_KEY"] DEFAULT_MODEL = os.environ.get("LLM_DEFAULT_MODEL", "openai/gemini-3.5-flash") async def complete(prompt: str, *, model: str | None = None, project_id: str, solution: str) -> str: response = await litellm.acompletion( model=model or DEFAULT_MODEL, messages=[{"role": "user", "content": prompt}], max_tokens=1024, # set generously — Gemini 3.5-flash uses thinking tokens before text # metadata flows to Langfuse + the gateway's cost-attribution layer metadata={ "project_id": project_id, "solution": solution, "trace_id": "auto", }, ) return response.choices[0].message.content ``` **Use the `openai/` prefix** when pointing litellm at a custom proxy (the CRAFT gateway). Without it, litellm infers the provider from the model name and routes directly to Anthropic/Vertex instead of through the proxy. `openai/gemini-3.5-flash` tells litellm "use the OpenAI wire format, route to whatever `api_base` says" — the gateway then forwards to the correct upstream. **Set `max_tokens ≥ 200`** for Gemini 3.5-flash at the `global` Vertex endpoint. The model uses adaptive thinking by default, consuming thinking tokens before text tokens. With `max_tokens=30`, all tokens go to thinking and the response is empty. The token budget needs room for both. ### Per-call model override The default model is configurable at deploy time, but any call site can override it: ```python theme={null} # Default for most calls — fastest + highest quality on agentic / coding benchmarks fast_answer = await complete("…", model="openai/gemini-3.5-flash", project_id=p, solution=s) # Only when extensive custom tool calling is core to this code path # (e.g. the model otherwise calls `cat`/`grep`/`sed` instead of your registered tools) tool_heavy = await complete("…", model="openai/gemini-3.1-pro-preview-customtools", project_id=p, solution=s) # Secondary provider — demonstrates model agnosticity secondary = await complete("…", model="openai/claude-opus-4-8", project_id=p, solution=s) # Local dev with no external dependency local_dev = await complete("…", model="ollama/llama3.1", project_id=p, solution=s) ``` The gateway routes by the model name you pass. Use the `openai/` prefix for models served over the OpenAI-compatible wire format (the common case for all gateway-hosted models); provider-native prefixes such as `ollama/` apply only for local sidecars not routed through the gateway. Ask the platform operator for the current model list. **Use `gemini-3.1-pro-preview-customtools` only when extensive custom tool calling is a core requirement of your agent.** It's a fine-tuned variant of `gemini-3.1-pro` that prioritizes your registered custom functions over bash fallbacks — same intelligence as base 3.1 Pro, narrowed for one job. **Google's own guidance**: if >50% of requests don't involve tool calling, stay on `gemini-3.5-flash`; quality on non-tool workloads is measurably lower on the customtools variant ([Apiyi: When to Switch to Customtools](https://help.apiyi.com/en/gemini-3-1-pro-preview-customtools-agent-guide-en.html)). `gemini-3.5-flash` already beats `gemini-3.1-pro` on agentic benchmarks (MCP Atlas +5.4%) and coding (Terminal-Bench +6%) at 3.6× the speed ([Gemini 3.5 launch data](https://wavespeed.ai/blog/posts/gemini-3-5-pro-flash/)) — so reach for the customtools variant only when you've **measured** that bash fallbacks are overriding your registered tools (Apiyi suggests: when bash usage exceeds 30% of actions that could have been handled by registered tools). Operational notes when you do use it: preview SLA (may be deprecated or renamed without the GA notice period), lower quota than GA models (will 429 sooner under sustained load — wrap in a fallback chain via [Failure modes](#failure-modes) below), global-endpoint only (`vertex_location` must be `global`), and **Provisioned Throughput (PT) is not supported**. ### Streaming (SSE) ```python theme={null} async def stream_completion(prompt: str, *, project_id: str, solution: str): response = await litellm.acompletion( model=DEFAULT_MODEL, messages=[{"role": "user", "content": prompt}], stream=True, metadata={"project_id": project_id, "solution": solution}, ) async for chunk in response: delta = chunk.choices[0].delta.content if delta: yield delta ``` Wrap in a FastAPI `StreamingResponse` to expose to your UI as Server-Sent Events. ## Observability — Langfuse When `LANGFUSE_HOST`, `LANGFUSE_PUBLIC_KEY`, and `LANGFUSE_SECRET_KEY` are set, `litellm` auto-emits traces. Each call shows up as a trace with the `metadata` you passed (project, solution, latency, token counts, model, prompt+completion). See [Guides › Langfuse Setup](/guides/langfuse-setup) and [Deployment › Observability › Langfuse](/deployment/observability/langfuse). Add `litellm.success_callback = ["langfuse"]` once at startup (the gateway's reference implementation does this in `commons/llm/__init__.py` of `em-talk2data`). For LLM-specific observability patterns, see [Deployment › Observability › LLM Observability](/deployment/observability/llm-observability). ## Cost attribution The `metadata={"project_id": ..., "solution": ...}` you pass on each call is the *only* thing that gives the platform per-project cost attribution. **Always pass it.** Without it, the call rolls up to a generic bucket and the cost dashboard cannot tell you which project caused the spike. If you wrap `litellm.acompletion` in a service helper (recommended), make `project_id` a required parameter so it's impossible to call without it. ## Failure modes The gateway enforces per-project rate limits. Back off exponentially (`tenacity` with `wait_random_exponential`) and surface a friendly message to the user. Do not retry forever — let the user see the rate limit. Either the gateway is down (rare) or the upstream provider rejected the request. Retry once with backoff, then fall back to a different model — cross-provider first for resilience (`fallbacks=["openai/claude-opus-4-8", "openai/gemini-3.1-pro-preview"]` is a `litellm` parameter — both route through the gateway's `model_list` to Vertex AI). **Don't list `gemini-3.1-pro-preview-customtools` as a general fallback** — it's specialized for tool-heavy agents and may degrade quality for non-tool workloads. `LLM_GATEWAY_API_KEY` rotated and your pod hasn't restarted. If you've configured Stakater Reloader on the secret (default for `em-service`), it should already be rolling. If not, `kubectl rollout restart` the deployment manually. The model you requested isn't routed by the gateway. Check the gateway's model list with `curl $LLM_GATEWAY_URL/models -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" | jq '.data[].id'`. A spike with no project attribution means a code path is calling without `metadata`. Grep for `litellm.acompletion(` and confirm every call passes `metadata={"project_id": ..., "solution": ...}`. See [Troubleshooting › LLM](/guides/solution-dev/troubleshooting#llm) for more. ## Verification ```bash theme={null} # Confirm the gateway is reachable from the pod kubectl -n em- exec deployment/-api -- \ sh -c 'curl -s -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" $LLM_GATEWAY_URL/models | head -c 200; echo' # Confirm a call attributes correctly in Langfuse curl -s -H "Authorization: Bearer $TOKEN" -H "X-Project-ID: $PROJECT_ID" \ http://localhost:8000/llm/test # your route that issues a complete() call # Then look in Langfuse for the trace tagged project_id=$PROJECT_ID ``` For gateway internals from an operator perspective (model allowlist, rate limits, provider routing), see [LLM Gateway](/platform/llm-gateway). ## Next steps Stand up Langfuse and wire it to your service. The platform-side observability stack. Trace inspection, cost dashboards, model comparison. Reference implementation that uses this pattern in production. # Architecture for Solution Developers Source: https://docs.emergence.ai/guides/solution-dev/architecture-for-solution-developers The integration surface a solution sees — what the platform provides, what your solution owns, and the contracts between them. # Architecture for Solution Developers This page explains the platform from the solution developer's perspective: what you can rely on, what you own, and what contracts cross the boundary. It is *not* a complete platform architecture deep-dive — for that, see [Architecture Overview](/getting-started/architecture). ## Three layers, your job is the top one CRAFT has three architectural layers. Solutions live at the top: | Layer | What lives here | Who owns it | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | **Solutions** | Domain apps (Data Insights, Data Governance, your new solution) | Solution teams | | **Platform** | Identity (Keycloak), authorization (OpenFGA), Governance/Assets/Utils services, secrets, storage abstraction, LLM gateway, microfrontend shell | Platform team | | **Infrastructure** | Cloud provisioning, GitOps, networking, cluster lifecycle | SRE / Infra team | Your solution **inherits** platform capabilities — never reimplements them. You do not run your own Keycloak. You do not write your own OpenFGA schema. You delegate via SDK calls and let the platform do the work. ## What the platform provides ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TB subgraph Solution["Your solution (em-<solution> namespace)"] UI["UI
(microfrontend)"] API["API service
(FastAPI)"] WORKER["Worker / agent
(optional)"] DB[("Your DB
(Postgres)")] UI --> API API --> WORKER API --> DB end subgraph Platform["Platform (em-runtime namespace)"] KC["Keycloak
(OIDC/JWT)"] GOV["Governance
(8000)"] ASSETS["Assets
(8000)"] UTILS["Utils
(8000)"] SECRETS["Secrets
(Infisical/ESO)"] OBSTORE["Object storage
(obstore)"] LITELLM["LiteLLM gateway"] end subgraph External["External"] MODELS["Model providers
(OpenAI, Anthropic, Gemini, ...)"] end API -->|JWT| KC API -->|"register agents,
connections"| ASSETS API -->|permission checks| GOV API -->|secrets at runtime| SECRETS API -->|read/write blobs| OBSTORE API -->|inference| LITELLM LITELLM --> MODELS classDef solution fill:#56B4E9,stroke:#555555,color:#000 classDef platform fill:#0072B2,stroke:#555555,color:#fff classDef store fill:#009E73,stroke:#555555,color:#000 classDef external fill:#CC79A7,stroke:#555555,color:#000 class UI,API,WORKER solution class KC,GOV,ASSETS,UTILS,SECRETS,OBSTORE,LITELLM platform class DB store class MODELS external ``` | Capability | Where | What you do | | ----------------- | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | Authentication | Keycloak (multi-realm OIDC/PKCE) | Validate JWTs in your service. See [Authenticate Users](/guides/solution-dev/authenticate-users). | | Authorization | OpenFGA via Governance SDK | Forward the user JWT + resource ID; receive allow/deny. | | Multi-tenancy | `org_id` (from JWT) + `X-Project-ID` header | Read both from request context; scope all queries by them. See [Concepts](/getting-started/concepts). | | Resource registry | Assets API (port 8000) | Register agents, data connections, files, models. See [Register a Solution](/guides/solution-dev/register-a-solution). | | Secrets | Infisical (self-hosted) or ESO + GCP Secret Manager (cloud) | Declare in your chart's `envVars`; read from `os.environ` or `/mnt/secrets/`. See [Manage Secrets](/guides/solution-dev/manage-secrets). | | Shared storage | `obstore` abstraction over S3, GCS, Azure Blob, MinIO | Initialize a client from a Connection; `put`/`get`/`list`. See [Use Shared Storage](/guides/solution-dev/use-shared-storage). | | LLM access | LiteLLM gateway (provider-agnostic) | Set `LLM_GATEWAY_URL`; call with the OpenAI SDK. See [Access LLMs](/guides/solution-dev/access-llms). | | Observability | Langfuse for LLM traces; OpenTelemetry for everything else | Set `LANGFUSE_HOST`; instrument with OTel SDK. See [Langfuse Setup](/guides/langfuse-setup). | | UI shell | `em-runtime-ui` (React Module Federation 2.0) | Expose your UI as a remote module. (Out of scope for v1 of this guide.) | ## What your solution owns You own everything that is specific to your domain: * **The UI** (typically a React microfrontend remote module loaded into `em-runtime-ui`) * **The backend services** (FastAPI + uvicorn is the platform default; nothing stops you choosing differently) * **Your own PostgreSQL database** — *no cross-service foreign keys with platform DBs*. Each service owns its data; consistency across services is by API contract, not joins. * **Your Helm chart** that wraps `em-service` (one alias per component) * **Optional Prefect workflows** for orchestration-heavy work * **Optional A2A or Pydantic AI agents** for agentic capabilities * **Your CI/CD pipeline** (per-repo, calling shared modules) ## Integration contracts Five contracts cross the platform-solution boundary. Get these right and the rest is implementation detail. ### 1. JWT (authentication) Every request to your service carries `Authorization: Bearer `. The token is issued by Keycloak in the user's organization realm. Validate the signature against Keycloak's JWKS, extract `org_id` from the issuer URL, and reject requests with no token. See [Authenticate Users](/guides/solution-dev/authenticate-users) for code. ### 2. `X-Project-ID` header (project scoping) JWT alone gives you organization context. The active project is communicated via `X-Project-ID`. Every database query, storage path, and downstream call must be scoped by both. See [Concepts](/getting-started/concepts) for the full multi-tenancy model. ### 3. Connections API (data sources) When your solution needs to talk to a customer database or object store, you do *not* ask the customer for credentials directly. They register a [Data Connection](/platform/data-connections) via Assets, and you fetch the configured connection at runtime. Credentials never live in your code or config. ### 4. obstore (shared storage) For artifacts, files, and intermediate data, use the platform's `obstore` abstraction. Same code works against S3, GCS, Azure Blob, and local MinIO — pick the backend with the configured connection. See [Use Shared Storage](/guides/solution-dev/use-shared-storage). ### 5. LiteLLM gateway (LLM access) Solutions never call OpenAI or Anthropic directly. Instead, point your OpenAI-compatible SDK at `LLM_GATEWAY_URL`. The gateway provides central key management, per-project cost attribution, rate-limit enforcement, and observability via Langfuse. See [Access LLMs](/guides/solution-dev/access-llms). ## What does NOT cross the boundary | You should never… | Because… | | ----------------------------------------------- | -------------------------------------------------------------------------- | | Run your own Keycloak | Multi-realm setup is platform infrastructure; rolling your own breaks SSO | | Write your own RBAC | OpenFGA is the platform's source of truth; permission decisions stay there | | Bake credentials into images or git | Use the secrets pipeline; rotation depends on it | | Add a foreign key from your DB to a platform DB | Breaks independent restore/backup; use API contracts instead | | Call OpenAI/Anthropic from your service code | LiteLLM gateway is the only attribution surface | | Access another solution's database directly | Cross-service joins are forbidden; use the other solution's API | ## Next steps Build a working hello-solution in 30 minutes. Naming, namespaces, image registries, registration steps. Iterate quickly with docker-compose, hot reload, and mocked platform services. Read the platform-side concepts (orgs, projects, X-Project-ID). # Authenticate Users Source: https://docs.emergence.ai/guides/solution-dev/authenticate-users Validate JWTs, use service accounts, and propagate X-Project-ID from your solution's services. # Authenticate Users This page shows how a solution validates user identity and project context. The platform provides Keycloak-issued JWTs (multi-realm OIDC/PKCE); your service is responsible for validating the signature, extracting claims, and propagating context to downstream calls. For the full platform-side authentication architecture (realm-per-org, JWKS rotation, SSO providers), see [Platform › Authentication](/platform/authentication) and [Security › Authentication](/security/authentication). ## Request flow ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% sequenceDiagram autonumber participant U as User / Client participant S as Your service participant KC as Keycloak (JWKS) participant G as Governance (OpenFGA) participant DB as Your DB U->>S: Request + Authorization: Bearer JWT + X-Project-ID S->>KC: GET /realms/<org>/protocol/openid-connect/certs (cached) KC-->>S: JWKS keys S->>S: Verify signature, exp, aud, iss S->>G: require_permission(user, action, resource) G-->>S: allow / deny alt Allowed S->>DB: Query scoped by org_id + project_id DB-->>S: Result S-->>U: 200 OK else Denied S-->>U: 403 Forbidden end ``` ## Steps Use `python-jose` to verify the token against the realm's JWKS. Cache the JWKS for \~10 minutes to avoid hammering Keycloak. ```python packages/api/src/api/auth.py theme={null} import os import time from typing import Annotated import httpx from fastapi import Depends, Header, HTTPException, status from jose import jwt, JWTError KEYCLOAK_ISSUER_URL = os.environ["KEYCLOAK_ISSUER_URL"] # e.g. https://kc.example.com/realms/ KEYCLOAK_AUDIENCE = os.environ["KEYCLOAK_AUDIENCE"] # your service's client_id # JWKS TTL — Keycloak rotates signing keys (default ~12h). Lifetime cache # (e.g. lru_cache) WILL produce 401s for valid tokens after rotation. _JWKS_TTL_SECONDS = 600 # 10 minutes _jwks_cache: dict | None = None _jwks_cached_at: float = 0.0 def _jwks() -> dict: global _jwks_cache, _jwks_cached_at now = time.monotonic() if _jwks_cache is None or now - _jwks_cached_at > _JWKS_TTL_SECONDS: url = f"{KEYCLOAK_ISSUER_URL.rstrip('/')}/protocol/openid-connect/certs" _jwks_cache = httpx.get(url, timeout=5.0).raise_for_status().json() _jwks_cached_at = now return _jwks_cache def current_user(authorization: Annotated[str | None, Header()] = None) -> dict: if not authorization or not authorization.lower().startswith("bearer "): raise HTTPException(status.HTTP_401_UNAUTHORIZED, "Missing bearer token") token = authorization.split(" ", 1)[1] try: claims = jwt.decode( token, _jwks(), algorithms=["RS256"], audience=KEYCLOAK_AUDIENCE, issuer=KEYCLOAK_ISSUER_URL, ) except JWTError as e: raise HTTPException(status.HTTP_401_UNAUTHORIZED, f"Invalid token: {e}") return claims def project_id(x_project_id: Annotated[str | None, Header()] = None) -> str: if not x_project_id: raise HTTPException(status.HTTP_400_BAD_REQUEST, "Missing X-Project-ID header") return x_project_id ``` ```python packages/api/src/api/main.py theme={null} from fastapi import Depends, FastAPI from typing import Annotated from .auth import current_user, project_id app = FastAPI() @app.get("/healthz") async def healthz() -> dict[str, str]: return {"status": "ok"} @app.get("/echo") async def echo( msg: str, user: Annotated[dict, Depends(current_user)], proj: Annotated[str, Depends(project_id)], ) -> dict[str, str]: return { "echo": msg, "user_sub": user["sub"], "org_id": user.get("org_id", ""), "project_id": proj, } ``` Routes that need auth declare `current_user` and (when project context matters) `project_id` as dependencies. Health probes stay unauthenticated. Standard claims you'll use: | Claim | Meaning | Notes | | -------- | ----------------- | ----------------------------------------------- | | `sub` | User UUID | Stable across sessions | | `iss` | Issuer URL | Realm path encodes `org_id` (last path segment) | | `aud` | Audience | Must equal your service's client\_id | | `exp` | Expiry | UNIX seconds | | `org_id` | Organization | Custom claim — also extractable from `iss` | | `groups` | Group memberships | Used by Governance for role mapping | Helper to extract `org_id` from the issuer when the custom claim isn't set: ```python theme={null} def org_id_from_issuer(iss: str) -> str: # iss like "https://kc.example.com/realms/acme-corp" return iss.rstrip("/").rsplit("/", 1)[-1] ``` Solutions never implement RBAC themselves — they delegate to OpenFGA via Governance. Forward the user's JWT and the resource ID; Governance returns allow/deny. ```python theme={null} import httpx, os GOVERNANCE_URL = os.environ.get("GOVERNANCE_URL", "http://em-runtime-governance.em-runtime:8000") async def require_permission(token: str, project_id: str, action: str, resource_uri: str) -> None: async with httpx.AsyncClient() as c: r = await c.post( f"{GOVERNANCE_URL}/governance/permissions/check", headers={"Authorization": f"Bearer {token}", "X-Project-ID": project_id}, json={"action": action, "resource_uri": resource_uri}, timeout=5.0, ) r.raise_for_status() if not r.json()["allowed"]: raise HTTPException(status.HTTP_403_FORBIDDEN, "Not authorized") ``` Use the auto-generated [Python SDK](/sdks/python) instead of raw HTTP in real services — it handles error mapping, retries, and OpenAPI-validated request shapes. Local tests should not call real Keycloak. Generate a signed token with a fixture private key and load the matching public key as a JWKS override. ```python tests/conftest.py theme={null} from datetime import datetime, timedelta, timezone from jose import jwt import pytest PRIV_KEY = open("tests/fixtures/dev-private.pem").read() PUB_KEY = open("tests/fixtures/dev-public.pem").read() @pytest.fixture def dev_token(): now = datetime.now(timezone.utc) return jwt.encode( { "sub": "user-123", "iss": "http://localhost:8080/realms/dev", "aud": "hello-solution-api", "org_id": "dev", "groups": ["developers"], "iat": int(now.timestamp()), "exp": int((now + timedelta(hours=1)).timestamp()), }, PRIV_KEY, algorithm="RS256", ) ``` Generate the keypair once with `openssl`: ```bash theme={null} openssl genrsa -out tests/fixtures/dev-private.pem 2048 openssl rsa -in tests/fixtures/dev-private.pem -pubout -out tests/fixtures/dev-public.pem ``` Configure your `current_user` dependency to load `tests/fixtures/dev-public.pem` as JWKS in test mode (e.g., when `PYTEST_CURRENT_TEST` is set). ## Service-to-service auth When your solution's worker calls another service (or back into Governance/Assets), it can't use a user's JWT. Use a **service account**: a Keycloak client with `client_credentials` grant. See [Platform › Service Accounts](/platform/service-accounts) for the management API. ```python theme={null} async def service_token() -> str: async with httpx.AsyncClient() as c: r = await c.post( f"{KEYCLOAK_ISSUER_URL}/protocol/openid-connect/token", data={ "grant_type": "client_credentials", "client_id": os.environ["SVC_ACCOUNT_CLIENT_ID"], "client_secret": os.environ["SVC_ACCOUNT_SECRET"], }, timeout=5.0, ) return r.json()["access_token"] ``` The service-account client must be granted the same OpenFGA roles you'd grant a user. Cache tokens for `expires_in - 60` seconds to avoid thundering herds at expiry. ## Common errors See the auth section of [Troubleshooting](/guides/solution-dev/troubleshooting#auth). ## Next steps Move `KEYCLOAK_AUDIENCE` and DB creds out of `.env`. Configure OpenFGA roles for your solution's resources. Reference for all platform-API auth headers. Use the auto-generated SDK instead of raw HTTP. # Local Development Source: https://docs.emergence.ai/guides/solution-dev/local-development Iterate on your solution locally with hot reload, mocked platform services, and fast feedback before deploying to the cluster. # Local Development This page covers three local-dev modes — choose based on what you're iterating on. The [Quickstart](/guides/solution-dev/quickstart) uses Kind; this page also covers the docker-compose loop (faster) and port-forward to a shared dev cluster (slowest, but real platform). ## Pick a mode ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TD Q{"What are you
iterating on?"} Q -->|"Service code only"| Q2{"Need real K8s
semantics?"} Q -->|"Platform integration
(identity + authz services)"| C["Port-forward to
shared dev cluster
(~30s per redeploy)"] Q2 -->|"No (probes mocked OK)"| A["docker-compose
+ uvicorn --reload
(<5s hot reload)"] Q2 -->|"Yes (probes, ingress,
configmaps)"| B["Kind cluster
+ kind-load.sh
(~30s per redeploy)"] classDef q fill:#F0E442,stroke:#555555,color:#000 classDef mode fill:#0072B2,stroke:#555555,color:#fff class Q,Q2 q class A,B,C mode ``` | Mode | Use when | Setup time | Iteration time | | -------------------------------------- | ---------------------------------------------------------------------------------------- | ---------- | ------------------ | | **docker-compose** | Iterating on your service code only; platform services can be mocked or stubbed | \~2 min | \<5s (hot reload) | | **Kind** | You need a real Kubernetes context (probes, configmaps, ingress) | \~5 min | \~30s per redeploy | | **Port-forward to shared dev cluster** | You need real platform services (real identity/authorization services, real Connections) | \<1 min | \~30s per redeploy | Most loops are docker-compose. Move to Kind for the integration-test pass. Use the shared dev cluster only when you need cross-service coordination. ## Mode 1: docker-compose with hot reload This is the daily-driver loop. Your service runs on the host with `uvicorn --reload`; supporting services (Postgres for your DB, MinIO for shared storage, a local identity provider) run in containers. ```yaml docker-compose.yaml theme={null} services: postgres: image: postgres:17-alpine environment: POSTGRES_USER: dev POSTGRES_PASSWORD: dev POSTGRES_DB: hello_solution ports: ["5432:5432"] healthcheck: test: ["CMD-SHELL", "pg_isready -U dev"] interval: 2s retries: 20 minio: image: minio/minio:latest command: server /data --console-address ":9001" environment: MINIO_ROOT_USER: minioadmin MINIO_ROOT_PASSWORD: minioadmin ports: ["9000:9000", "9001:9001"] healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/ready"] interval: 2s retries: 20 # Local identity provider for JWT validation. Uses the platform's standard # dev image so JWKS endpoints behave realistically. keycloak: image: quay.io/keycloak/keycloak:26.0 command: ["start-dev"] environment: KEYCLOAK_ADMIN: admin KEYCLOAK_ADMIN_PASSWORD: admin ports: ["8080:8080"] ``` ```bash .env.example theme={null} # Database (matches docker-compose service) DATABASE_URL=postgresql+asyncpg://dev:dev@localhost:5432/hello_solution # Object storage (MinIO with path-style addressing) OBSTORE_ENDPOINT=http://localhost:9000 OBSTORE_ACCESS_KEY=minioadmin OBSTORE_SECRET_KEY=minioadmin OBSTORE_BUCKET=hello-solution OBSTORE_PATH_STYLE=true # Auth (point at the local identity provider realm you create with the bootstrap below) KEYCLOAK_ISSUER_URL=http://localhost:8080/realms/dev KEYCLOAK_AUDIENCE=hello-solution-api # LLM gateway (set to a real LiteLLM gateway URL in dev/stg/prod; # for local-only loops you can stub the LLM client) LLM_GATEWAY_URL=http://localhost:4000 LLM_GATEWAY_API_KEY=local-dev-key LLM_DEFAULT_MODEL=gemini-3.5-flash ``` Copy to `.env` (which should be in `.gitignore`) before running. ```bash theme={null} docker compose up -d docker compose ps # confirm all healthy # Pre-create the MinIO bucket docker run --rm --network host minio/mc \ sh -c "mc alias set local http://localhost:9000 minioadmin minioadmin && mc mb -p local/hello-solution" # Run your service with hot reload set -a; source .env; set +a uv run uvicorn api.main:app --host 0.0.0.0 --port 8000 --reload ``` Editing `packages/api/src/api/*.py` reloads the server inside \~1 second. Hit `http://localhost:8000/echo?msg=hi` to confirm. ```bash theme={null} docker compose down -v # -v also removes the named volumes ``` ## Mode 2: Kind cluster Use Kind when you need real Kubernetes semantics (probes firing, ConfigMaps mounting, Ingress routing). The [Quickstart](/guides/solution-dev/quickstart) walks through this end-to-end. Make iteration faster with this rebuild-and-load script: ```bash kind-load.sh theme={null} #!/usr/bin/env bash set -euo pipefail CLUSTER="${1:-hello-solution}" TAG="${2:-dev}" docker build -t hello-solution-api:$TAG . kind load docker-image hello-solution-api:$TAG --name $CLUSTER # Trigger a rolling restart (image pull policy IfNotPresent + same tag won't redeploy) kubectl -n em-hello-solution rollout restart deployment/hello-solution-api kubectl -n em-hello-solution rollout status deployment/hello-solution-api ``` Run `./kind-load.sh` after each code change. \~30 seconds per cycle. ## Mode 3: Port-forward to shared dev cluster Use this when you need real platform behavior — real identity provider realms, real authorization data, real Connections. Run your code locally and forward platform services from a shared `dev` cluster. ```bash theme={null} # Connect to your shared dev cluster (context provisioned out of band by your platform team) kubectl config use-context # Port-forward platform services to localhost. # Each em-runtime service exposes Service port 8000; the local port is chosen for distinctness. kubectl -n em-runtime port-forward svc/em-runtime-governance 8001:8000 & kubectl -n em-runtime port-forward svc/em-runtime-assets 8002:8000 & kubectl -n em-runtime port-forward svc/em-runtime-utils 8003:8000 & kubectl -n em-runtime port-forward svc/keycloak 8080:8080 & # Run your service against them set -a; source .env.dev; set +a uv run uvicorn api.main:app --reload ``` Useful pitfall to know about: when `port-forward` drops (any error in the cluster), curl from your service will fail until you restart the forward. Wrap forwards in a tiny supervisor script (`while true; do kubectl port-forward ... ; done`) for long-running loops. ## Mocking platform services Three things are commonly mocked in unit tests: 1. **JWT verification** — generate a signed test token with a fixture private key; load the matching public key as `KEYCLOAK_JWKS_OVERRIDE` (a path to a JSON file your auth code prefers over the JWKS endpoint when set). 2. **Permission checks** — patch the Governance SDK's permission method to return `True` (or the value under test). 3. **LiteLLM gateway** — point `LLM_GATEWAY_URL` at a stub HTTP server that returns canned completions; or use `litellm`'s built-in mock provider (`model="mock-openai/..."`). See [SDKs › Python](/sdks/python) for SDK extension points and [Authenticate Users](/guides/solution-dev/authenticate-users) for the JWT fixture pattern. ## Common pitfalls Containers in Kind cannot use `localhost` to reach your host machine. Use `host.docker.internal` (macOS/Windows) or the host's LAN IP (Linux). Or — better — run dependencies inside Kind with another `helm install`. MinIO requires path-style addressing. Set `OBSTORE_PATH_STYLE=true` (or the equivalent in your S3 client config). Virtual-hosted style works on real S3 but not on MinIO without DNS magic. `uvicorn --reload` watches the working directory by default. If your source is in a non-default location (e.g., `packages/api/src/api/`), pass `--reload-dir packages/api/src`. For container-based reload, mount the source directory as a volume and run uvicorn inside the container. Every call to Governance, Assets, or Utils requires `X-Project-ID`. In local dev, hardcode a fixture project ID (e.g., `00000000-0000-0000-0000-000000000001`) until you wire user-driven project switching. ## CI smoke test The [Quickstart](/guides/solution-dev/quickstart) runs end-to-end in CI weekly via `.github/workflows/quickstart-smoke.yml`. The workflow provisions a Kind cluster, builds the example image, installs the chart, port-forwards, and asserts both endpoints respond. If it fails, an issue is opened automatically with the label `solution-dev-guide` — that is your signal to update the guide. This is the strongest possible self-sustainability signal: the quickstart cannot rot silently. ## Next steps Wire JWT validation into your local app. Move secrets out of `.env` into the platform pipeline. Replace MinIO test code with the obstore client. Stuck? Check known causes by error symptom. # Manage Secrets Source: https://docs.emergence.ai/guides/solution-dev/manage-secrets Retrieve secrets at runtime — environment injection via em-service, file mounts at /mnt/secrets/, and adding new secrets to your chart. # Manage Secrets This page is about how a *solution* gets secrets at runtime. The platform handles sourcing, rotation, and delivery — your job is to declare what you need and read it correctly. For the platform-side architecture (Infisical vs ESO + GCP Secret Manager, rotation mechanics, Workload Identity), see [Security › Secrets Management](/security/secrets-management) and [Deployment › Infrastructure › Secrets](/deployment/infrastructure/secrets). ## Secret pipeline ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR subgraph Upstream["Secret backend (platform-managed)"] BACKEND[("Infisical
or
ESO + GCP SM")] end subgraph Cluster["Your namespace (em-<solution>)"] K8S[("K8s Secret
<solution>-secrets")] POD["Your pod"] ENV["env vars
(via envFrom)"] FILE["files at
/mnt/secrets/<key>"] end RELOAD["Stakater Reloader"] BACKEND -->|"sync (~1h)"| K8S K8S --> ENV K8S --> FILE ENV --> POD FILE --> POD K8S -.->|"on change"| RELOAD RELOAD -.->|"rolling restart"| POD classDef store fill:#009E73,stroke:#555555,color:#000 classDef infra fill:#0072B2,stroke:#555555,color:#fff classDef act fill:#E69F00,stroke:#555555,color:#000 classDef sol fill:#56B4E9,stroke:#555555,color:#000 class BACKEND,K8S store class RELOAD infra class ENV,FILE act class POD sol ``` **Read the diagram left-to-right**: the upstream backend is the source of truth; ESO or Infisical syncs values into a K8s Secret in your namespace; `em-service` injects them into your pod as env vars (via `envFrom`) and/or files (via volume mount). On rotation, Reloader detects the K8s Secret change and rolls your Deployment. ## Two delivery shapes The `em-service` chart delivers secrets to your pod in two ways. Pick whichever fits the consumer: | Shape | Format | Best for | | ------------------------ | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | **Environment variable** | Loaded via `envFrom` from a synthesized `-secrets` Kubernetes Secret | Most use cases (DB URLs, API keys, OIDC client secrets) | | **File mount** | Files at `/mnt/secrets/` | Multi-line secrets (PEM keys, certs); SDKs that expect a file path (gcloud, kubeconfig) | Both are populated from the *same* upstream secret backend; the difference is purely how your code reads the value. ## Steps Reference the secret by `secretKeyRef` in your `values.yaml`. The Kubernetes Secret itself (`-secrets`) is created by the platform's secret pipeline — you do not check secret values into the chart. ```yaml charts//values.yaml theme={null} api: env: # Plain values: use the simple form LOG_LEVEL: "INFO" KEYCLOAK_ISSUER_URL: "https://keycloak.example.com/realms/acme" KEYCLOAK_AUDIENCE: "-api" envVars: # Secret values: use envVars with valueFrom.secretKeyRef - name: DATABASE_URL valueFrom: secretKeyRef: name: -secrets key: database-url - name: LLM_GATEWAY_API_KEY valueFrom: secretKeyRef: name: -secrets key: llm-gateway-api-key ``` For file-mount secrets, declare a volume and mount it (the platform's `em-service` chart supports custom mounts via `volumeMounts` and `volumes`): ```yaml theme={null} api: volumeMounts: - name: tls mountPath: /mnt/secrets/tls readOnly: true volumes: - name: tls secret: secretName: -tls ``` Environment variables: standard `os.environ`. File mounts: read from disk lazily. ```python theme={null} import os from pathlib import Path # Env-injected secret db_url = os.environ["DATABASE_URL"] # File-mounted secret tls_cert = Path("/mnt/secrets/tls/tls.crt").read_text() tls_key = Path("/mnt/secrets/tls/tls.key").read_text() ``` Two rules: 1. **Never log a secret value.** Wrap reads in repr-safe types or use `pydantic.SecretStr`. 2. **Don't re-read on every request.** Read once at startup; re-read only when a rotation signal fires (see Stakater Reloader below). Adding a secret requires two changes: 1. **Add it to your chart** (the `envVars` snippet above) referencing the key name. 2. **Add the value to the upstream secret backend** in each environment. The upstream backend depends on where you're deploying: | Deployment | Where to add the secret | | ---------------------------------- | -------------------------------------------------------------------------------------------------- | | Local dev (docker-compose / Kind) | `.env` (gitignored) or `kubectl create secret generic -secrets --from-literal=key=value` | | Self-hosted (Infisical) | Infisical UI for the project; ESO syncs to the cluster | | Cloud (GCP Secret Manager via ESO) | `gcloud secrets create` + grant Workload Identity SA `secretmanager.secretAccessor` | See [Security › Secrets Management](/security/secrets-management) and [Deployment › Infrastructure › Secrets](/deployment/infrastructure/secrets) for backend setup. For local-dev secrets: * **Use `.env`** (in `.gitignore`) for env vars; `set -a; source .env; set +a` before `uv run`. * **Mount fixture files** for file-style secrets via docker-compose `volumes`. * **Never commit secrets** — even fixture/dummy ones if they look real. Use values like `dev-only-not-a-real-key`. * **Generate fresh keys** for tests (e.g., the `dev-private.pem` from [Authenticate Users](/guides/solution-dev/authenticate-users)). ## Where the values come from Solutions don't talk to the secret backend directly — the platform's secret pipeline ([ESO](/deployment/infrastructure/secrets) for cloud, [Infisical](/security/secrets-management) for self-hosted) materializes upstream secrets into Kubernetes Secrets in your namespace. Your chart's `secretKeyRef` references those K8s Secrets. The platform handles: * Authentication to the upstream backend (Workload Identity in GCP, machine identities in Infisical) * Sync intervals (ESO defaults to 1h; configurable per ExternalSecret) * Resync on upstream change You do not see the upstream backend from your service code. Just declare the K8s Secret and key. ## Rotation The platform pairs secrets with [Stakater Reloader](https://github.com/stakater/Reloader). When the underlying K8s Secret changes (because ESO synced an update from upstream), Reloader rolls your Deployment automatically — *if* you opt in by annotating the Deployment. `em-service` adds the annotation by default for the secrets it injects via `envFrom`. For custom volume mounts (like `-tls`), add the annotation manually: ```yaml charts//values.yaml theme={null} api: podAnnotations: secret.reloader.stakater.com/reload: "-tls" ``` Verify the rollout fires: ```bash theme={null} # Update the upstream secret, then watch kubectl -n em- rollout status deployment/-api ``` ## Verification ```bash theme={null} # Confirm the K8s Secret exists with the expected keys kubectl -n em- get secret -secrets -o jsonpath='{.data}' | jq 'keys' # Confirm the env var reaches the pod kubectl -n em- exec deployment/-api -- printenv DATABASE_URL | head -c 60; echo # Confirm the file mount reaches the pod (if applicable) kubectl -n em- exec deployment/-api -- ls -la /mnt/secrets/tls ``` If the secret is declared but the pod fails with `secret not found`: ESO/Infisical hasn't synced yet, or the upstream secret doesn't exist. See [Troubleshooting › secrets](/guides/solution-dev/troubleshooting#secrets). ## Next steps Reference for `env` vs `envVars` precedence and dedup rules. Storage credentials follow the same pattern as DB creds. Platform-side architecture (ESO vs Infisical). Pod CrashLoopBackOff with "secret not found" and similar. # Solution Developer Guide Source: https://docs.emergence.ai/guides/solution-dev/overview Build, package, and deploy a new solution on the CRAFT platform — from first FastAPI service to production deployment. **For AI agents and LLMs**: this site is Mintlify-hosted, so every page is also available as Markdown by appending `.md` to the URL (or via content negotiation with `Accept: text/markdown`). Prefer those variants — HTML wastes context tokens. The site-wide index lives at `/llms.txt` and the full corpus at `/llms-full.txt`. **Cursor / Claude Code / Windsurf users** — connect your IDE to `https://docs.emergence.ai/mcp` for live doc queries during coding tasks (MCP search tool enabled). # Solution Developer Guide This guide walks you through building a new **solution** on the CRAFT platform: registering it, wiring it to platform services (auth, secrets, storage, LLMs), running it locally, and deploying it. It is the single starting point for engineers building on top of CRAFT. ## Who this is for This guide is for **engineers building a new solution on CRAFT** — Emergence platform engineers, embedded teams, and co-design partners. It is *not* for end users of an already-built solution — those have their own docs: [Data Insights](/data-insights/overview), [Data Governance](/data-governance/overview), and [Semiconductor](/semiconductor/overview). ### Reader audiences and where you read There are two reader sites: | Site | URL | You see | | -------- | ------------------------------------------------- | ------------------------------------------------------------ | | Internal | `docs-internal.emergence.ai` (password-protected) | Everything, including internal-only pages and gated sections | | External | `docs.emergence.ai` (public) | Only the partner-safe content | If you are reading on the public site (`docs.emergence.ai`), some implementation-specific sections are hidden by design — the partner-friendly capability descriptions remain. ## Journey map ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR A["Read overview"] --> B["Quickstart
(30 min)"] B --> C["Understand
architecture"] C --> D["Local dev"] D --> E["Register your
solution"] E --> F["Wire integrations"] F --> G["Package &
deploy"] G --> H["Operate"] F -.-> F1["Auth"] F -.-> F2["Secrets"] F -.-> F3["Storage"] F -.-> F4["LLMs"] classDef entry fill:#56B4E9,stroke:#555555,color:#000 classDef step fill:#0072B2,stroke:#555555,color:#fff classDef leaf fill:#E69F00,stroke:#555555,color:#000 class A entry class B,C,D,E,F,G,H step class F1,F2,F3,F4 leaf ``` ## Use this when… You want a working hello-solution in 30 minutes. You want to understand what the platform provides and what your solution owns. You're iterating on your service locally and want hot reload + mocked platform services. You're starting a new solution and need names, namespaces, and registration steps. You need to validate JWTs and propagate `X-Project-ID` from your service. You need to inject secrets into pods and rotate them safely. You need to read and write objects (artifacts, files) to S3/GCS/Azure/MinIO. Your solution calls LLMs and you need provider-agnostic, observable, cost-attributed access. You're ready to build a multi-component Helm chart and ship it to a cluster. You want copy-pasteable scaffolds for three common solution shapes. Something is broken and you want a probable cause + fix in seconds. ## Issue-to-page traceability This guide addresses the [seven onboarding requirements](#issue-to-page-traceability) tracked internally as follows: | Requirement | Primary page(s) | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | Register a new solution | [Register a solution](/guides/solution-dev/register-a-solution) | | Container & namespace conventions | [Register a solution](/guides/solution-dev/register-a-solution); [Package and deploy](/guides/solution-dev/package-and-deploy) | | Connect to Keycloak (auth) | [Authenticate users](/guides/solution-dev/authenticate-users) | | Connect to Infisical (secrets) | [Manage secrets](/guides/solution-dev/manage-secrets) | | Connect to shared storage | [Use shared storage](/guides/solution-dev/use-shared-storage) | | LLM integration | [Access LLMs](/guides/solution-dev/access-llms) | | Local development setup | [Local development](/guides/solution-dev/local-development) | | Minimal working example | [Quickstart](/guides/solution-dev/quickstart); [Starter templates](/guides/solution-dev/starter-templates) Tier 1 | | Starter templates for common scenarios | [Starter templates](/guides/solution-dev/starter-templates) | ## Reference solutions Three production solutions live on the platform today; treat them as canonical examples when you need a working pattern. | Solution | Code repo | Pattern | | -------------------------------------------- | ------------------- | -------------------------------------------------------------------------------------- | | [Data Insights](/data-insights/overview) | `em-talk2data` | Multi-agent A2A protocol — gateway service + insights agent + text2sql agent + LiteLLM | | [Data Governance](/data-governance/overview) | `em-data-readiness` | API service + Prefect workflows — profiling, enrichment, DQ rules | | [Semiconductor](/semiconductor/overview) | `em-semi` | Pydantic AI agents + air-gapped deployment — fab analytics | The starter templates page maps each tier to one of these reference solutions: Tier 1 ≈ minimal API service, Tier 2 ≈ `em-data-readiness` shape, Tier 3 ≈ `em-talk2data` shape. ## Success criteria — your feedback closes the loop This guide has one explicit performance target: > **The [Quickstart](/guides/solution-dev/quickstart) must take you 30 minutes or less, end to end.** If it takes longer, that is a guide bug — please report it via the 👎 thumbs at the bottom of the affected page (Mintlify captures the comment and routes it to the docs team). Use the 👍 / 👎 thumbs at the bottom of every page. Thumbs-down comments roll up into the quarterly docs governance review for this group. The quickstart also runs in CI weekly (see [Local development](/guides/solution-dev/local-development#ci-smoke)) so we know within a week if it stops working. ## Conventions used in this guide * **Placeholders**: `` is your solution's name (DNS-safe, lowercase, hyphens). `` is one of `dev`, `stg`, `prod`. `` and `` come from your JWT and `X-Project-ID` header — see [Concepts](/getting-started/concepts). * **Pinned versions**: Code samples are pinned to **em-service v0.0.15** (the version in [em-service Chart](/deployment/helm/em-service) as of 2026-05-14). Major em-service version bumps refresh the Tier 1 fixtures under `scripts/tests/fixtures/solution-dev/tier1/`; `npm run check-snippets` (CI) fails if inline snippets drift. * **Service catalog**: There is no Backstage-style service catalog yet; the closest current surface is the em-runtime Assets API where you register agents, data connections, and artifacts — see [Register Your First Agent](/guides/first-agent). A first-class portal is on the platform roadmap. * **Cross-refs**: Use absolute paths (`/guides/...`, `/platform/...`). * **Diagrams**: Mermaid only, with the project init block and the Okabe-Ito palette (entry `#56B4E9`, infrastructure `#0072B2`, data store `#009E73`, action `#E69F00`, decision `#F0E442`, external `#CC79A7`). ## What's next If you are reading this for the first time, go straight to the [Quickstart](/guides/solution-dev/quickstart). If you have a running hello-solution and want to add real integrations, jump to the relevant how-to from the journey map above. # Package and Deploy Source: https://docs.emergence.ai/guides/solution-dev/package-and-deploy Wrap a multi-component solution in a Helm chart, manage per-environment overlays, and ship the image to your registry. # Package and Deploy The [Quickstart](/guides/solution-dev/quickstart) deploys a single-component solution. This page extends to multi-component (api + worker + scheduler + agent…), per-environment overlays, image-publish workflows, and where ArgoCD picks up from there. ## Single vs multi-component `em-service` (v0.0.15) is the base chart for *one* deployable workload — one Deployment, one Service, one set of probes. Solutions that need more than one workload include `em-service` multiple times as subchart dependencies, each with a unique `alias`. ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TB subgraph Parent["Your chart: charts/<solution>"] CHART["Chart.yaml
(declares subcharts)"] VAL["values.yaml
+ values.<env>.yaml"] end subgraph EM["em-service v0.0.15 (per alias)"] API["alias: api
Deployment + Service + probes"] WORKER["alias: worker
Deployment + probes"] SCHED["alias: scheduler
Deployment (singleton)"] end CHART --> API CHART --> WORKER CHART --> SCHED VAL -->|"merge: defaults +
values.<env>.yaml"| API VAL --> WORKER VAL --> SCHED classDef parent fill:#56B4E9,stroke:#555555,color:#000 classDef act fill:#E69F00,stroke:#555555,color:#000 classDef infra fill:#0072B2,stroke:#555555,color:#fff class CHART,VAL parent class API,WORKER,SCHED infra ``` **Read top-down**: one umbrella chart per solution; one `em-service` instance per workload (api / worker / scheduler / agent); per-environment values overlay-merge with the defaults. Add a fourth alias and you've added a fourth deployable component — no other moving parts change. See [em-service Chart › Using as a subchart](/deployment/helm/em-service#using-as-a-subchart) for the canonical reference. ## Multi-component layout A typical "API + worker" solution: ```yaml charts//Chart.yaml theme={null} apiVersion: v2 name: description: on CRAFT type: application version: 0.1.0 appVersion: "0.1.0" dependencies: - name: em-service alias: api version: 0.0.15 repository: oci://ghcr.io/emergenceai/em-charts - name: em-service alias: worker version: 0.0.15 repository: oci://ghcr.io/emergenceai/em-charts - name: em-service alias: scheduler # optional Prefect / cron component version: 0.0.15 repository: oci://ghcr.io/emergenceai/em-charts ``` Each alias has its own values block in `values.yaml`: ```yaml charts//values.yaml theme={null} api: image: repository: ghcr.io//-api tag: 0.1.0 service: type: ClusterIP port: 8000 env: LOG_LEVEL: "INFO" envVars: - name: DATABASE_URL valueFrom: { secretKeyRef: { name: -secrets, key: database-url } } readinessProbe: { httpGet: { path: /healthz, port: 8000 } } livenessProbe: { httpGet: { path: /healthz, port: 8000 } } autoscaling: enabled: true minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 70 worker: image: repository: ghcr.io//-worker tag: 0.1.0 service: enabled: false # workers don't expose HTTP env: LOG_LEVEL: "INFO" QUEUE_NAME: "-jobs" envVars: - name: DATABASE_URL valueFrom: { secretKeyRef: { name: -secrets, key: database-url } } scheduler: image: repository: ghcr.io//-scheduler tag: 0.1.0 service: enabled: false replicaCount: 1 # singleton — never scale schedulers horizontally ``` ## Per-environment overlays Use one `values.yaml` for shared defaults, plus `values..yaml` overlays for things that change between environments (image tags, replica counts, host names, log levels). Apply with `helm -f`: ```bash theme={null} helm upgrade --install ./charts/ \ --namespace em- --create-namespace \ --values ./charts//values.yaml \ --values ./charts//values.dev.yaml \ --wait --timeout 5m ``` Example dev overlay: ```yaml charts//values.dev.yaml theme={null} api: image: { tag: dev } replicaCount: 1 autoscaling: { enabled: false } env: { LOG_LEVEL: "DEBUG" } worker: image: { tag: dev } replicaCount: 1 ``` Production overlay (just the deltas): ```yaml charts//values.prod.yaml theme={null} api: image: { tag: 1.4.7 } autoscaling: { minReplicas: 3, maxReplicas: 20 } ingress: enabled: true className: nginx hosts: - host: .example.com paths: [{ path: /, pathType: Prefix }] worker: image: { tag: 1.4.7 } replicaCount: 4 ``` ## Build and publish images Per-component, multi-stage `Dockerfile` (one image per workload). Use the same Dockerfile parametrized via `--target` if your components share a build base; or one Dockerfile per component if they diverge significantly. ```bash theme={null} # Build docker build -t ghcr.io//-api:1.4.7 --target api . docker build -t ghcr.io//-worker:1.4.7 --target worker . # Login (whatever registry you use) docker login ghcr.io -u -p $GITHUB_TOKEN # Publish docker push ghcr.io//-api:1.4.7 docker push ghcr.io//-worker:1.4.7 ``` For private registries, configure `imagePullSecrets` in `values.yaml`: ```yaml theme={null} api: imagePullSecrets: [{ name: ghcr-pull }] worker: imagePullSecrets: [{ name: ghcr-pull }] ``` ## Versioning Keep the Helm chart `version` and the image `tag` independent: | What | Bumped when | | --------------------------------- | --------------------------------------------------------------------- | | Chart `version` (in `Chart.yaml`) | Chart structure changes (added/removed/renamed values, alias changes) | | `appVersion` | Default app version this chart targets — informational | | Image `tag` (in `values.yaml`) | Application code changes | Use [semver](https://semver.org/) for both. See [Helm › Upgrades](/deployment/helm/upgrades) for the upgrade policy (which chart-version bumps are breaking, how to roll back, etc.). ## Deploy ### Local ```bash theme={null} helm dependency update ./charts/ helm upgrade --install ./charts/ \ --namespace em- --create-namespace \ -f ./charts//values.yaml \ -f ./charts//values.dev.yaml \ --wait --timeout 5m kubectl -n em- get pods -w ``` Verify all components are running and probes are passing. ### CI/CD For automated deploys, use whatever CI/CD your team prefers; the chart itself is environment-agnostic. ## Air-gapped deployments If the target cluster has no internet access (some customer environments), pre-stage: 1. The chart tarball (`helm pull oci://ghcr.io/emergenceai/em-charts/em-service --version 0.0.15`) 2. All component images (`docker save | docker load` on the air-gapped side) 3. The pull-secret pointing at the customer's internal registry ## Verification ```bash theme={null} # Chart renders cleanly with the production overlay helm template ./charts/ \ -f ./charts//values.yaml \ -f ./charts//values.prod.yaml | head -50 # All declared components have running pods kubectl -n em- get deployments kubectl -n em- get pods # Image tags match what you intended to deploy kubectl -n em- get pods -o jsonpath='{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u ``` ## Next steps Full values reference for the base chart. Platform-wide chart configuration. Upgrade and rollback policy. Pre-built scaffolds for common solution shapes. # Quickstart: Hello Solution Source: https://docs.emergence.ai/guides/solution-dev/quickstart Stand up a minimal FastAPI service, package it as a Helm chart wrapping em-service, and deploy it to a local Kind cluster — under 30 minutes. # Quickstart: Hello Solution By the end of this tutorial you will have a FastAPI service running in a local Kubernetes cluster, packaged as a Helm chart that wraps the platform's `em-service` base chart, and reachable by `curl`. Total time: **30 minutes or less**. If it takes longer, that's a guide bug — please report via the 👎 thumbs at the bottom of this page. This tutorial keeps every command and config snippet inline. Its one external dependency is the platform's `em-service` base chart, pulled from Emergence's **private** container registry during install — registry access is granted to the Emergence team and Solution/Agent design partners (you authenticate with `helm registry login` in the install step). ## Prerequisites Install these tools. Versions are floors, not pins — newer is fine. | Tool | Version | Purpose | | ------- | ------- | ------------------------------ | | Docker | 24+ | Container runtime for Kind | | Kind | 0.20+ | Local Kubernetes cluster | | kubectl | 1.28+ | Kubernetes CLI | | Helm | 3.13+ | Package manager for Kubernetes | | uv | 0.4+ | Python project manager | ```bash theme={null} brew install docker kind kubectl helm uv ``` ```bash theme={null} # Docker — follow distro-specific instructions at https://docs.docker.com/engine/install/ # Kind curl -Lo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64 chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind # kubectl curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" chmod +x kubectl && sudo mv kubectl /usr/local/bin/ # Helm curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash # uv curl -LsSf https://astral.sh/uv/install.sh | sh ``` ## What you'll build A solution called **`hello-solution`** with one component (`api`) that exposes: * `GET /healthz` → `{"status":"ok"}` for k8s probes * `GET /echo?msg=...` → echoes the message back The image is built locally and loaded into Kind (no registry needed). The Helm chart wraps `em-service` v0.0.15 with `alias: api`, so you'll see how multi-component solutions extend (just add another alias). ## Steps Pick a working directory and scaffold these files: ```text theme={null} hello-solution/ ├── packages/api/ │ ├── pyproject.toml │ └── src/api/ │ └── main.py ├── Dockerfile └── charts/hello-solution/ ├── Chart.yaml └── values.yaml ``` ```bash theme={null} mkdir -p hello-solution/packages/api/src/api hello-solution/charts/hello-solution cd hello-solution ``` ```python packages/api/src/api/main.py theme={null} from fastapi import FastAPI app = FastAPI(title="hello-solution") @app.get("/healthz") async def healthz() -> dict[str, str]: return {"status": "ok"} @app.get("/echo") async def echo(msg: str = "hi") -> dict[str, str]: return {"echo": msg} ``` ```toml packages/api/pyproject.toml theme={null} [project] name = "hello-solution-api" version = "0.1.0" requires-python = ">=3.12" dependencies = [ "fastapi>=0.110", "uvicorn[standard]>=0.30", ] [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["src/api"] ``` A multi-stage build keeps the runtime image small. ```dockerfile Dockerfile theme={null} FROM python:3.12-slim AS build WORKDIR /app RUN pip install --no-cache-dir uv COPY packages/api/pyproject.toml packages/api/pyproject.toml COPY packages/api/src packages/api/src RUN uv pip install --system --no-cache-dir ./packages/api FROM python:3.12-slim COPY --from=build /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages COPY --from=build /usr/local/bin/uvicorn /usr/local/bin/uvicorn EXPOSE 8000 CMD ["uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "8000"] ``` The chart declares one `em-service` subchart, aliased as `api`. The platform's `em-service` (v0.0.15) handles Deployment, Service, probes, env vars — you only configure your image and ports. To add a worker later, you'd add another `em-service` entry with `alias: worker` and configure it under that alias key in `values.yaml`. See [em-service Chart](/deployment/helm/em-service) for the full values reference. ```yaml charts/hello-solution/Chart.yaml theme={null} apiVersion: v2 name: hello-solution description: A minimal CRAFT solution type: application version: 0.1.0 appVersion: "0.1.0" dependencies: - name: em-service alias: api version: 0.0.15 # latest tag published to the OCI registry repository: oci://ghcr.io/emergenceai/em-charts ``` **Why `0.0.15` and not the latest source version?** The em-charts source `Chart.yaml` is on `0.0.16` (and an unrelated `0.1.0` ships for `em-charts-internal/em-service-internal`), but the OCI tag `ghcr.io/emergenceai/em-charts/em-service:0.0.15` is the most recent version actually published to the registry — `0.0.16` returns "not found" on `helm pull`. em-talk2data's own `Chart.yaml` pins `0.0.15` for the same reason. When em-charts publishes a newer tag, bump here. ```yaml charts/hello-solution/values.yaml theme={null} api: # em-service's fullname template defaults to the release name, which would # name every component "hello-solution" — collide if you add another alias. # Set fullnameOverride per component so resources are named "-" # (this is what em-runtime itself does for governance/assets/utils). fullnameOverride: hello-solution-api image: repository: hello-solution-api tag: dev pullPolicy: IfNotPresent service: type: ClusterIP port: 8000 env: LOG_LEVEL: "INFO" readinessProbe: httpGet: path: /healthz port: 8000 initialDelaySeconds: 2 livenessProbe: httpGet: path: /healthz port: 8000 initialDelaySeconds: 5 ``` ```bash theme={null} # Create cluster kind create cluster --name hello-solution # Build and load image docker build -t hello-solution-api:dev . kind load docker-image hello-solution-api:dev --name hello-solution ``` `kind load docker-image` makes your local image available inside the cluster without a registry. ```bash theme={null} # Authenticate to the private em-charts registry first (team / design-partner access) helm registry login ghcr.io helm dependency update ./charts/hello-solution helm upgrade --install hello-solution ./charts/hello-solution \ --namespace em-hello-solution --create-namespace \ --wait --timeout 5m ``` First install pulls `em-service` from `ghcr.io/emergenceai/em-charts`, a **private** registry — `helm registry login ghcr.io` authenticates you (access is granted to the Emergence team and Solution/Agent design partners). ```bash theme={null} kubectl port-forward -n em-hello-solution svc/hello-solution-api 8000:8000 & curl -s "http://localhost:8000/healthz" | jq . # {"status":"ok"} curl -s "http://localhost:8000/echo?msg=ship%20it" | jq . # {"echo":"ship it"} ``` If both calls succeed, **you have shipped your first CRAFT solution**. ```bash theme={null} kill %1 2>/dev/null # stop port-forward helm uninstall hello-solution -n em-hello-solution kubectl delete namespace em-hello-solution kind delete cluster --name hello-solution ``` ## What you just did ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR SRC["Source
(packages/api/)"] DOCK["docker build
hello-solution-api:dev"] LOAD["kind load
docker-image"] HELM["helm install
(em-service v0.0.15)"] NS[("Namespace
em-hello-solution")] POD["Pod: api
(port 8000)"] PF["kubectl
port-forward"] CURL["curl /echo"] SRC --> DOCK --> LOAD --> HELM HELM --> NS NS --> POD POD --> PF --> CURL classDef sol fill:#56B4E9,stroke:#555555,color:#000 classDef act fill:#E69F00,stroke:#555555,color:#000 classDef store fill:#009E73,stroke:#555555,color:#000 class SRC,POD sol class DOCK,LOAD,HELM,PF,CURL act class NS store ``` You wrapped a 25-line FastAPI app in a Helm chart that depends on the platform's `em-service` base chart, deployed it to a namespace named `em-` (the platform convention), and reached it via `port-forward`. You did not configure auth, secrets, storage, or LLMs yet — those are the next how-tos: Make this naming + namespace pattern systematic. Protect `/echo` with JWT validation. Wire a secret-backed env var into your chart. Loop faster with docker-compose + hot reload. # Register a New Solution Source: https://docs.emergence.ai/guides/solution-dev/register-a-solution End-to-end checklist for registering a new solution on the platform — name, namespace, image registry, Helm wiring, database, and platform integration. # Register a New Solution This is the canonical checklist for starting a new solution on CRAFT. Follow these steps when you're past the [Quickstart](/guides/solution-dev/quickstart) prototype and ready to make the solution real. Most decisions here are conventions — the platform doesn't *force* you, but solutions that follow the conventions get free deployment, observability, and CI integration. ## Registration sequence ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR NAME["1. Pick a name
<solution>"] NS[("2. Create namespace
em-<solution>")] REG["3. Choose image
registry"] HELM["4. Wire Helm chart
(em-service alias)"] DB[("5. Provision DB
+ Alembic")] AG{"6. Has agents?"} AG_Y["Register via
Assets API"] CONN{"7. Has data
connections?"} CONN_Y["Register via
Assets API"] INT["8. Wire integrations
(auth/secrets/storage/LLMs)"] DONE([Solution live]) NAME --> NS --> REG --> HELM --> DB --> AG AG -->|yes| AG_Y --> CONN AG -->|no| CONN CONN -->|yes| CONN_Y --> INT CONN -->|no| INT INT --> DONE classDef act fill:#E69F00,stroke:#555555,color:#000 classDef store fill:#009E73,stroke:#555555,color:#000 classDef q fill:#F0E442,stroke:#555555,color:#000 classDef sol fill:#56B4E9,stroke:#555555,color:#000 class NAME,REG,HELM,AG_Y,CONN_Y,INT act class NS,DB store class AG,CONN q class DONE sol ``` The dashed sequence is the **order**, not the schedule — most solutions land steps 1–4 in one PR, 5 in the next, and 6/7/8 incrementally. Steps 6 and 7 are conditional; only Tier 3 agentic solutions need them up front. ## Pre-flight checklist Lowercase, alphanumeric + hyphens only, ≤ 30 chars. Used as the Helm release name, the namespace suffix (`em-`), and the package prefix in your monorepo. One repo per solution. Monorepo layout from day one — even with a single component — so adding a worker later is a 3-step diff. See [Architecture for Solution Developers](/guides/solution-dev/architecture-for-solution-developers#what-your-solution-owns). Any OCI registry your target cluster can pull from. Internal solutions use `ghcr.io/emergenceai/`. Partners typically use their own registry. Your own Postgres database. No cross-service foreign keys. Migrations versioned with Alembic from day one. Agents, data connections, files, and models registered through the platform's Assets API are discoverable to other solutions and the UI. Decide upfront what your solution exposes. ## Steps Match the solution name across: * Repo name (e.g. `em-talk2data`) * Helm release name (e.g. `talk2data`) * Namespace (e.g. `em-talk2data`) * Database (e.g. `talk2data`) Existing examples: `em-talk2data` (Data Insights), `em-data-readiness` (Data Governance), `em-semi` (Semiconductor). Stick to this rhythm — it makes ops searches and grep-based audits trivial. | Namespace | Used by | | ---------------------- | ------------------------------------------------------------------ | | `em-runtime` | Platform services (Governance, Assets, Utils, Keycloak, OpenFGA) | | `em-` | Your solution's pods, services, and config | | `em--system` | Cluster-scoped sidecars or operators owned by your solution (rare) | Create the namespace explicitly with a label so the platform's NetworkPolicy / Reloader / cost-attribution pick it up: ```bash theme={null} kubectl create namespace em- kubectl label namespace em- emergence.ai/solution= ``` Configure `image.repository` in your chart's `values.yaml`. For private registries, also configure `imagePullSecrets`: ```yaml theme={null} api: image: repository: /-api tag: 0.1.0 imagePullSecrets: - name: ``` Single component? See [Quickstart Step 4](/guides/solution-dev/quickstart). Multi-component (api + worker + scheduler + agent…)? See [Package and Deploy](/guides/solution-dev/package-and-deploy) for the alias pattern. The minimum is one `em-service` subchart per component, with each component aliased so its values live under that key: ```yaml charts//Chart.yaml theme={null} dependencies: - { name: em-service, alias: api, version: 0.0.15, repository: oci://ghcr.io/emergenceai/em-charts } - { name: em-service, alias: worker, version: 0.0.15, repository: oci://ghcr.io/emergenceai/em-charts } ``` Your solution owns its database. Two non-negotiable rules: 1. **No cross-service foreign keys.** Each service must be backup-restorable independently. Cross-service consistency is by API contract, not joins. 2. **Migrations from day one** — Alembic is the platform standard. Migrations are immutable once committed; never edit a published migration, write a new one. Provision via your chart (the Postgres subchart pattern) or via Crossplane claims for managed databases. Connection string injected as a secret-backed env var (see [Manage Secrets](/guides/solution-dev/manage-secrets)). If your solution exposes A2A agents (the Data Insights pattern), register each with the Assets API so other services and the UI can discover them. Follow [Register Your First Agent](/guides/first-agent) — that walkthrough covers Agent Card authoring (`/.well-known/agent-card.json`), validation, registration, listing, and update with optimistic concurrency. If your solution reads customer data (databases, object stores, file shares), the customer registers a [Data Connection](/platform/data-connections) and grants your solution permission. Your code fetches the connection at runtime — credentials never live in your config. Step through these in order: Validate JWTs and propagate `X-Project-ID`. Inject runtime secrets via the chart. Use obstore for cross-cloud blob access. Call the LiteLLM gateway (never providers directly). ## What you get when you follow the conventions | If you… | You get… | | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | Follow `em-` namespace | Auto-applied NetworkPolicy, Reloader watching your Secrets, cost attribution by solution | | Wrap `em-service` with one alias per component | Standard probes, env var deduplication, Gateway API ingress, HPA — all configurable but with sane defaults | | Register agents in Assets | Discoverability from the UI shell + other solutions; permission integration with OpenFGA | | Use Connections for data sources | Credential rotation handled by the platform; never in your code | | Call LLMs via LiteLLM gateway | Per-project cost attribution, rate limits enforced centrally, traces in Langfuse | ## Verification ```bash theme={null} # Namespace label set kubectl get namespace em- -o jsonpath='{.metadata.labels}' # Helm release lives in the right place helm list -n em- # Image pulls succeed (no ImagePullBackOff) kubectl -n em- get pods # DB migration ran (if applicable) kubectl -n em- logs job/-migrate ``` If anything is unhealthy, see [Troubleshooting](/guides/solution-dev/troubleshooting). ## Next steps Multi-component charts, env overlays, image publish. Test changes locally before pushing. The first integration most solutions need. Common errors at registration time. # Starter Templates Source: https://docs.emergence.ai/guides/solution-dev/starter-templates Copy-paste scaffolds for three common solution shapes — minimal API service, multi-component (api + worker + DB), and agentic A2A solution. # Starter Templates Three tiers, increasing in complexity. Each tier is a starting point — copy, then mutate to fit. The Tier 1 fixtures are mirrored in `scripts/tests/fixtures/solution-dev/tier1/` and verified by `npm run check-snippets` so this page does not silently rot. **Future**: when the `em-service-template` repo lands, these tiers collapse to `cookiecutter gh:emergenceai/em-service-template --no-input solution_name= tier=<1|2|3>`. The inline scaffolds here are the bridge until that ships. Every scaffold pins `em-service` to **v0.0.15** (the version in [em-service Chart](/deployment/helm/em-service) as of 2026-05-14). Major em-service version bumps trigger a fixture refresh; `npm run check-snippets` (CI) fails if the inline snippets drift from the canonical fixtures. The simplest viable solution: one HTTP service, one image, JWT validation, env-injected secrets. Maps to the [Quickstart](/guides/solution-dev/quickstart) shape with auth and secrets wired in. ### File tree ```text theme={null} / ├── packages/api/ │ ├── pyproject.toml │ └── src/api/ │ ├── __init__.py │ ├── main.py │ └── auth.py ├── Dockerfile └── charts// ├── Chart.yaml ├── values.yaml └── values.dev.yaml ``` ### Files ```toml packages/api/pyproject.toml theme={null} [project] name = "-api" version = "0.1.0" requires-python = ">=3.12" dependencies = [ "fastapi>=0.110", "uvicorn[standard]>=0.30", "python-jose[cryptography]>=3.3", "httpx>=0.27", ] [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["src/api"] ``` ```python packages/api/src/api/main.py theme={null} from fastapi import Depends, FastAPI from typing import Annotated from .auth import current_user, project_id app = FastAPI(title="") @app.get("/healthz") async def healthz() -> dict[str, str]: return {"status": "ok"} @app.get("/echo") async def echo( msg: str, user: Annotated[dict, Depends(current_user)], proj: Annotated[str, Depends(project_id)], ) -> dict[str, str]: return {"echo": msg, "user_sub": user["sub"], "project_id": proj} ``` ```python packages/api/src/api/auth.py theme={null} import os import time from typing import Annotated import httpx from fastapi import Header, HTTPException, status from jose import jwt, JWTError KEYCLOAK_ISSUER_URL = os.environ["KEYCLOAK_ISSUER_URL"] KEYCLOAK_AUDIENCE = os.environ["KEYCLOAK_AUDIENCE"] # JWKS TTL — Keycloak rotates signing keys (default ~12h) _JWKS_TTL_SECONDS = 600 _jwks_cache: dict | None = None _jwks_cached_at: float = 0.0 def _jwks() -> dict: global _jwks_cache, _jwks_cached_at now = time.monotonic() if _jwks_cache is None or now - _jwks_cached_at > _JWKS_TTL_SECONDS: url = f"{KEYCLOAK_ISSUER_URL.rstrip('/')}/protocol/openid-connect/certs" _jwks_cache = httpx.get(url, timeout=5.0).raise_for_status().json() _jwks_cached_at = now return _jwks_cache def current_user(authorization: Annotated[str | None, Header()] = None) -> dict: if not authorization or not authorization.lower().startswith("bearer "): raise HTTPException(status.HTTP_401_UNAUTHORIZED, "Missing bearer token") try: return jwt.decode( authorization.split(" ", 1)[1], _jwks(), algorithms=["RS256"], audience=KEYCLOAK_AUDIENCE, issuer=KEYCLOAK_ISSUER_URL, ) except JWTError as e: raise HTTPException(status.HTTP_401_UNAUTHORIZED, f"Invalid token: {e}") def project_id(x_project_id: Annotated[str | None, Header()] = None) -> str: if not x_project_id: raise HTTPException(status.HTTP_400_BAD_REQUEST, "Missing X-Project-ID header") return x_project_id ``` ```dockerfile Dockerfile theme={null} FROM python:3.12-slim AS build WORKDIR /app RUN pip install --no-cache-dir uv COPY packages/api/pyproject.toml packages/api/pyproject.toml COPY packages/api/src packages/api/src RUN uv pip install --system --no-cache-dir ./packages/api FROM python:3.12-slim COPY --from=build /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages COPY --from=build /usr/local/bin/uvicorn /usr/local/bin/uvicorn EXPOSE 8000 CMD ["uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "8000"] ``` ```yaml charts//Chart.yaml theme={null} apiVersion: v2 name: description: A minimal CRAFT solution type: application version: 0.1.0 appVersion: "0.1.0" dependencies: - name: em-service alias: api version: 0.0.15 repository: oci://ghcr.io/emergenceai/em-charts ``` ```yaml charts//values.yaml theme={null} api: fullnameOverride: -api image: repository: /-api tag: 0.1.0 service: type: ClusterIP port: 8000 env: LOG_LEVEL: "INFO" KEYCLOAK_ISSUER_URL: "https://keycloak.example.com/realms/acme" KEYCLOAK_AUDIENCE: "-api" readinessProbe: { httpGet: { path: /healthz, port: 8000 } } livenessProbe: { httpGet: { path: /healthz, port: 8000 } } ``` ```yaml charts//values.dev.yaml theme={null} api: image: { tag: dev } replicaCount: 1 env: LOG_LEVEL: "DEBUG" KEYCLOAK_ISSUER_URL: "http://localhost:8080/realms/dev" ``` ### Mirrored fixtures Canonical copies of the eight files above live at `scripts/tests/fixtures/solution-dev/tier1/`. `npm run check-snippets` compares the inline snippets to the fixtures and fails if they drift. Adds a worker component, a Postgres database, and Alembic migrations. Mirrors the [Data Governance (em-data-readiness)](/data-governance/overview) shape. Below is the **diff** from Tier 1 — apply on top of Tier 1 to get Tier 2. ### File tree (additions) ```text theme={null} / ├── packages/ │ ├── api/ (unchanged from Tier 1) │ ├── worker/ │ │ ├── pyproject.toml │ │ └── src/worker/ │ │ ├── __init__.py │ │ └── main.py │ └── common/ │ ├── pyproject.toml │ └── src/common/ │ ├── __init__.py │ ├── db.py │ └── models.py ├── alembic.ini ├── alembic/ │ ├── env.py │ └── versions/ │ └── 0001_initial.py └── charts// ├── Chart.yaml (add em-service alias for worker) └── values.yaml (add worker block + DATABASE_URL secret) ``` ### Chart.yaml diff ```yaml charts//Chart.yaml theme={null} apiVersion: v2 name: type: application version: 0.1.0 dependencies: - name: em-service alias: api version: 0.0.15 repository: oci://ghcr.io/emergenceai/em-charts # NEW for Tier 2 - name: em-service alias: worker version: 0.0.15 repository: oci://ghcr.io/emergenceai/em-charts ``` ### values.yaml diff ```yaml charts//values.yaml theme={null} api: # ... existing Tier 1 api block ... envVars: # NEW: DB connection string from secret - name: DATABASE_URL valueFrom: secretKeyRef: name: -secrets key: database-url # NEW for Tier 2 worker: image: repository: /-worker tag: 0.1.0 service: enabled: false env: LOG_LEVEL: "INFO" envVars: - name: DATABASE_URL valueFrom: { secretKeyRef: { name: -secrets, key: database-url } } ``` ### Worker entry point ```python packages/worker/src/worker/main.py theme={null} import asyncio import logging import os from common.db import session_factory logging.basicConfig(level=os.environ.get("LOG_LEVEL", "INFO")) log = logging.getLogger("worker") async def process_one() -> None: async with session_factory() as session: # ... pull a job, run it, commit ... log.info("processed job") async def main() -> None: while True: try: await process_one() except Exception: log.exception("worker iteration failed") await asyncio.sleep(1.0) if __name__ == "__main__": asyncio.run(main()) ``` ### Common DB module ```python packages/common/src/common/db.py theme={null} import os from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine engine = create_async_engine(os.environ["DATABASE_URL"], echo=False, future=True) session_factory = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False) ``` Reference implementation: see [Data Governance overview](/data-governance/overview) — the `em-data-readiness` repo follows this exact pattern. Adds an A2A agent component with LiteLLM and (optionally) a Prefect flow. Mirrors the [Data Insights (em-talk2data)](/data-insights/overview) shape. Below is the **diff** from Tier 2. ### File tree (additions) ```text theme={null} / ├── packages/ │ ├── api/ (unchanged) │ ├── worker/ (unchanged) │ ├── common/ (unchanged) │ ├── agent/ │ │ ├── pyproject.toml │ │ └── src/agent/ │ │ ├── __init__.py │ │ ├── card.py # Agent Card author │ │ ├── llm.py # LiteLLM client wrapper │ │ └── server.py # A2A protocol server │ └── flows/ # optional Prefect flows │ ├── pyproject.toml │ └── src/flows/ │ ├── __init__.py │ └── pipeline.py └── charts// ├── Chart.yaml (add em-service alias for agent + optional flows) └── values.yaml (add agent block with LLM env) ``` ### Chart.yaml diff ```yaml charts//Chart.yaml theme={null} dependencies: - { name: em-service, alias: api, version: 0.0.15, repository: oci://ghcr.io/emergenceai/em-charts } - { name: em-service, alias: worker, version: 0.0.15, repository: oci://ghcr.io/emergenceai/em-charts } # NEW for Tier 3 - { name: em-service, alias: agent, version: 0.0.15, repository: oci://ghcr.io/emergenceai/em-charts } ``` ### values.yaml additions for the agent ```yaml charts//values.yaml theme={null} agent: image: repository: /-agent tag: 0.1.0 service: type: ClusterIP port: 8001 env: LOG_LEVEL: "INFO" AGENT_NAME: "-agent" LLM_GATEWAY_URL: "https://litellm.example.com/v1" LLM_DEFAULT_MODEL: "gemini-3.5-flash" LANGFUSE_HOST: "https://langfuse.example.com" envVars: - name: LLM_GATEWAY_API_KEY valueFrom: { secretKeyRef: { name: -secrets, key: llm-gateway-api-key } } - name: LANGFUSE_PUBLIC_KEY valueFrom: { secretKeyRef: { name: -secrets, key: langfuse-public-key } } - name: LANGFUSE_SECRET_KEY valueFrom: { secretKeyRef: { name: -secrets, key: langfuse-secret-key } } readinessProbe: { httpGet: { path: /.well-known/agent-card.json, port: 8001 } } livenessProbe: { httpGet: { path: /.well-known/agent-card.json, port: 8001 } } ``` ### Agent Card ```python packages/agent/src/agent/card.py theme={null} AGENT_CARD = { "name": "-agent", "description": "", "version": "0.1.0", "url": "https://-agent.example.com", "protocolVersion": "0.3", "capabilities": {"streaming": True, "pushNotifications": False}, "skills": [ { "id": "answer", "name": "Answer questions", "description": "Answers questions using the project's data", "tags": ["qa"], "examples": ["What's the trend in last quarter's sales?"], } ], "securitySchemes": {"bearer": {"type": "http", "scheme": "bearer", "bearerFormat": "JWT"}}, "security": [{"bearer": []}], } ``` ### Register the agent Follow [Register Your First Agent](/guides/first-agent) — the agent must be registered with Assets after first deployment so other services and the UI can discover it. ### Reference implementation `em-talk2data` is the canonical reference — see [Data Insights overview](/data-insights/overview) for the full architecture (gateway service + insights agent + text2sql agent + LiteLLM via the same gateway pattern shown above). ## Snippet drift check The Tier 1 fixture files at `scripts/tests/fixtures/solution-dev/tier1/` are the source of truth for the inline snippets above. Run `npm run check-snippets` to verify they match — CI runs this check on every PR. If you change Tier 1 inline content, update the fixtures (or vice versa), then re-run `npm run check-snippets`. ## Next steps Walk through Tier 1 step by step. Multi-component packaging and per-environment overlays. Required for Tier 3 agents to be discoverable. The Tier 3 reference solution in production. # Troubleshooting Source: https://docs.emergence.ai/guides/solution-dev/troubleshooting Recognise and fix the most common errors encountered when building solutions on CRAFT — auth, secrets, image pulls, helm conflicts, LLM failures. # Troubleshooting Symptom-first reference. Find the closest match below and follow the diagnostic steps. If your symptom isn't here, file an issue with label `solution-dev-guide` and the symptom will land in the next revision. ## 401 / 403 from your service **Diagnose:** ```bash theme={null} curl -v -H "Authorization: Bearer $TOKEN" -H "X-Project-ID: $PROJECT_ID" \ http://localhost:8000/echo?msg=hi 2>&1 | head -30 ``` **Common causes:** * Token missing → request didn't include `Authorization` header * Token expired → check `exp` claim with `jq -R 'split(".") | .[1] | @base64d | fromjson' <<< "$TOKEN"` * Wrong audience → your service's `KEYCLOAK_AUDIENCE` env doesn't match the `aud` claim * Wrong issuer → `KEYCLOAK_ISSUER_URL` doesn't match the realm in the `iss` claim * JWKS cache stale after Keycloak key rotation → restart your pod (or implement a TTL cache) **Diagnose:** ```bash theme={null} # Confirm Governance sees the user with the expected roles curl -s -H "Authorization: Bearer $TOKEN" -H "X-Project-ID: $PROJECT_ID" \ http://em-runtime-governance.em-runtime:8000/governance/whoami | jq . ``` **Common causes:** * User lacks the OpenFGA tuple for the action — see [RBAC Configuration](/guides/rbac-configuration) * `X-Project-ID` points to a project the user has no access to * You're calling the permission check with the wrong `resource_uri` shape Every authenticated route on the platform expects `X-Project-ID`. Add it to your test client / SDK calls. In FastAPI, use the `project_id` dependency from [Authenticate Users](/guides/solution-dev/authenticate-users). ## Pod crashes or "secret not found" **Diagnose:** ```bash theme={null} kubectl -n em- describe pod | grep -A2 "Failed" kubectl -n em- get secret -secrets # exists? kubectl -n em- get externalsecret # if ESO; check sync status ``` **Common causes:** * ESO/Infisical hasn't synced yet → wait one sync interval, or trigger manually * Upstream secret doesn't exist → create it in GCP Secret Manager / Infisical / Vault * SecretStore points to wrong project/path → check ClusterSecretStore config * K8s ServiceAccount missing Workload Identity annotation (GCP) → see [Secrets deployment](/deployment/infrastructure/secrets) **Diagnose:** ```bash theme={null} kubectl -n em- exec -- printenv DATABASE_URL kubectl -n em- get secret -secrets -o jsonpath='{.data.database-url}' | base64 -d ``` **Common causes:** * Key name mismatch between `secretKeyRef.key` and the K8s Secret data key * The `env:` map in `values.yaml` overrides `envVars` with the same name (the [em-service env model](/deployment/helm/em-service#environment-variable-model) deduplicates with `env` winning) → rename one of them * Secret value is literally empty in upstream (you set it to `""` somewhere) Stakater Reloader needs an annotation to know which Secret to watch. For env-injected secrets via `envFrom`, `em-service` adds it by default. For custom volume-mounted secrets, you must add `secret.reloader.stakater.com/reload: ""` to `podAnnotations` — see [Manage Secrets › Rotation](/guides/solution-dev/manage-secrets#rotation). ## Image pull errors **Diagnose:** ```bash theme={null} kubectl -n em- describe pod | grep -A3 "Failed to pull" ``` **Common causes:** * `image.repository` or `image.tag` is wrong * Registry is private and `imagePullSecrets` not configured * Pull-secret expired (registry tokens often have short lifetimes) * Cluster cannot reach the registry (network policy / air-gapped environment) * Wrong CPU architecture (e.g., arm64 image on amd64 cluster) — check with `docker manifest inspect` Some registries (GHCR, ACR) enforce IP allowlists. Check the registry's audit log for the cluster's egress IP and add it to the allowlist. For GHCR specifically, the personal access token used must have `read:packages` scope. ## Helm upgrade fails Kubernetes Deployments have an immutable `.spec.selector`. Once your release exists, you cannot change `app.kubernetes.io/name`, alias names, or labels that affect the selector. **Workaround**: `helm uninstall` and `helm install` (loses pod identity for \~1 cycle) — or re-create the namespace. If you rename an `em-service` alias from `api` to `webapp`, every reference in `values.yaml` (and any `--set`) must update too. The platform won't warn — it'll just deploy with default values. Check rendered output: ```bash theme={null} helm template ./charts/ -f values.yaml | grep -A5 "kind: Deployment" ``` Helm dependency cache is per-chart. After bumping the `em-service` version in `Chart.yaml`, run `helm dependency update ./charts/` before `helm upgrade`. Stale `Chart.lock` causes the old version to be deployed silently. If a previous release left orphan resources (PVCs, PDBs), `helm install` fails with "exists and cannot be imported into the current release." Delete the orphans manually or use `helm install --replace`. ## CRD not found A CRD your chart references isn't installed. Most commonly: * Gateway API (`gateway.networking.k8s.io/v1`) → install via `kubectl apply -k 'github.com/kubernetes-sigs/gateway-api/config/crd?ref=v1.1.0'` * ExternalSecret (`external-secrets.io/v1beta1`) → install ESO via `helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace` * The platform's own CRDs (if you depend on `em-core`) → install em-core first; see [em-core Chart](/deployment/helm/em-core) Recent `em-service` chart versions target Gateway API `v1` (`gateway.networking.k8s.io/v1`), while some older clusters only have `v1beta1`. Either upgrade the cluster's Gateway API CRDs (see above), or pin `em-service` to a chart version that still targets `v1beta1`. Check the [em-service Chart](/deployment/helm/em-service) reference for the Gateway API version each release uses. ## LLM gateway errors Per-project rate limit hit. Back off (`tenacity` with `wait_random_exponential`), then surface a friendly message. **Don't** retry forever. Upstream provider rejected or gateway lost a route. Configure fallbacks in the call: ```python theme={null} # Fallback to a different provider first (claude-opus-4-8 via Vertex) # rather than to a preview model. The customtools variant is specialised # for tool-heavy agents and isn't appropriate as a general fallback. await litellm.acompletion( model="gemini-3.5-flash", messages=[...], fallbacks=["claude-opus-4-8", "vertex_ai/gemini-3.1-pro-preview"], ) ``` `LLM_GATEWAY_API_KEY` rotated but pod hasn't restarted. Check Reloader annotation; manually `kubectl rollout restart` if needed. A code path called `litellm.acompletion(...)` without `metadata={"project_id": ..., "solution": ...}`. Grep for the call sites and confirm every one passes metadata. The HTTP client's read timeout is shorter than the model's first-token latency. Bump it: `httpx.AsyncClient(timeout=httpx.Timeout(connect=5.0, read=120.0, write=5.0, pool=5.0))`. ## Postgres errors Pod-to-Postgres DNS using the wrong service name. Use the K8s service DNS: ```text theme={null} postgresql+asyncpg://:@-postgresql.em-.svc.cluster.local:5432/ ``` Not `localhost`, not the pod IP. Add a `waitForPostgres` init container, or use a startup probe that retries until the DB accepts connections. Run migrations as a Helm hook (`helm.sh/hook: pre-install,pre-upgrade`) so only one pod ever runs them at a time. ## Local dev gotchas Use `host.docker.internal` (macOS/Windows) or your host LAN IP (Linux). Or run dependencies inside Kind via `helm install`. Set `OBSTORE_PATH_STYLE=true`. MinIO requires path-style addressing. Pass `--reload-dir packages/api/src` if your code lives outside the cwd. Editor "atomic write" may not trigger watchdog → switch the editor's save mode. `docker compose ps` shows `unhealthy`? Check `docker compose logs `. The compose file in [Local Development](/guides/solution-dev/local-development) sets healthchecks; failures usually mean port conflicts or insufficient memory. ## Still stuck? Report via the 👎 thumbs at the bottom of the affected page — include: 1. The exact symptom (with command + output) 2. What you've already tried 3. Your solution's name and the cluster context # Use Shared Storage Source: https://docs.emergence.ai/guides/solution-dev/use-shared-storage Read and write objects through the platform's infrastructure-agnostic obstore client — works against S3, GCS, Azure Blob, and MinIO. # Use Shared Storage The platform exposes object storage through `obstore`, a thin abstraction that lets the same code run against AWS S3, GCS, Azure Blob, and local MinIO. Solutions use it for artifacts, intermediate data, files served to the UI, and anything else that doesn't belong in the relational database. For the platform-side abstraction model, see [Platform Overview](/platform/overview). For data-source connections (databases + storage credentials registered by customers), see [Platform › Data Connections](/platform/data-connections). ## Why obstore Three things you get for free: 1. **One code path, all clouds.** No `if AWS: ... elif GCP: ...` in your service. 2. **Credential rotation handled by the platform.** Connections fetch fresh credentials per request via the secrets pipeline. 3. **Per-tenant path isolation enforced by convention.** Every operation is scoped by `org_id` + `project_id` (passed via `X-Project-ID`). ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR subgraph SOL["Your solution"] CODE["one code path
obs.put / obs.get"] PREFIX["path prefix:
<org_id>/<project_id>
/<solution>/..."] CODE --> PREFIX end OBS["obstore client
(env-configured backend)"] PREFIX --> OBS OBS --> S3[("AWS S3")] OBS --> GCS[("GCS")] OBS --> AZ[("Azure Blob")] OBS --> MIN[("MinIO
(local dev)")] classDef sol fill:#56B4E9,stroke:#555555,color:#000 classDef act fill:#E69F00,stroke:#555555,color:#000 classDef infra fill:#0072B2,stroke:#555555,color:#fff classDef store fill:#009E73,stroke:#555555,color:#000 class CODE,PREFIX sol class OBS infra class S3,GCS,AZ,MIN store ``` **Read left-to-right**: your code calls `obs.put` / `obs.get` with a path scoped by `org_id`/`project_id`/`solution`; the obstore client picks the actual backend from env config; the same code runs against any of S3/GCS/Azure/MinIO. ## Path conventions Stick to this prefix layout for everything you write: ```text theme={null} ///// ``` Example: ```text theme={null} /acme-corp/proj-42/talk2data/conversations/abc123/messages.jsonl ``` Predictable paths make backup, garbage collection, and audit trivial. ## Provision a connection Customers register a [Data Connection](/platform/data-connections) (type `s3`, `gcs`, or `minio`) via Assets. Your solution looks it up by name when it needs to read or write. The customer registers the connection in the UI or via the Assets API, then grants your solution permission. Your code: ```python theme={null} from em_runtime_assets_sdk import AuthenticatedClient from em_runtime_assets_sdk.api.connections import get_connection async def get_storage_creds(token: str, project_id: str, conn_name: str) -> dict: client = AuthenticatedClient(base_url="http://em-runtime-assets.em-runtime:8000", token=token) conn = await get_connection.asyncio(client=client, x_project_id=project_id, name=conn_name) # conn.config holds bucket/region/endpoint; credentials fetched server-side via Secrets API return { "endpoint": conn.config.endpoint, "bucket": conn.config.bucket, "region": conn.config.region, "access_key": conn.credentials.access_key, "secret_key": conn.credentials.secret_key, } ``` For solution-owned scratch storage (logs, intermediate Prefect outputs), put bucket details directly in your chart and inject creds via secrets: ```yaml charts//values.yaml theme={null} api: env: OBSTORE_ENDPOINT: "https://s3.us-east-1.amazonaws.com" OBSTORE_BUCKET: "-artifacts" OBSTORE_REGION: "us-east-1" envVars: - name: OBSTORE_ACCESS_KEY valueFrom: { secretKeyRef: { name: -secrets, key: storage-access-key } } - name: OBSTORE_SECRET_KEY valueFrom: { secretKeyRef: { name: -secrets, key: storage-secret-key } } ``` ## Code patterns The `obstore` Python package gives you a single API across providers. Install with `uv add obstore`. ### Initialize a client from env ```python packages/api/src/api/storage.py theme={null} import os from obstore.store import S3Store, GCSStore, AzureStore, LocalStore def make_store(): backend = os.environ.get("OBSTORE_BACKEND", "s3") if backend == "s3": return S3Store( bucket=os.environ["OBSTORE_BUCKET"], endpoint=os.environ.get("OBSTORE_ENDPOINT"), # MinIO sets this; AWS doesn't access_key_id=os.environ["OBSTORE_ACCESS_KEY"], secret_access_key=os.environ["OBSTORE_SECRET_KEY"], region=os.environ.get("OBSTORE_REGION", "us-east-1"), virtual_hosted_style_request=os.environ.get("OBSTORE_PATH_STYLE", "false") != "true", ) if backend == "gcs": return GCSStore(bucket=os.environ["OBSTORE_BUCKET"]) # Workload Identity if backend == "azure": return AzureStore(container=os.environ["OBSTORE_BUCKET"]) if backend == "local": return LocalStore(prefix=os.environ.get("OBSTORE_LOCAL_PREFIX", "/tmp/obstore")) raise ValueError(f"Unknown backend: {backend}") ``` ### Put / get / list / delete ```python theme={null} import obstore as obs store = make_store() # PUT (small) — bytes in, path key out await obs.put_async(store, "acme-corp/proj-42/talk2data/conversations/abc123.json", b'{"hi":"there"}') # GET (small) data = await obs.get_async(store, "acme-corp/proj-42/talk2data/conversations/abc123.json") print(data.bytes()) # b'{"hi":"there"}' # LIST (paginated; iterate the async iterator) async for batch in obs.list_async(store, prefix="acme-corp/proj-42/"): for entry in batch: print(entry.path, entry.size) # DELETE await obs.delete_async(store, "acme-corp/proj-42/talk2data/conversations/abc123.json") ``` ### Stream a large object (avoid loading into memory) ```python theme={null} async def stream_to_disk(store, key: str, dest: str) -> None: response = await obs.get_async(store, key) with open(dest, "wb") as f: async for chunk in response.stream(min_chunk_size=8 * 1024 * 1024): f.write(chunk) ``` For multi-GB objects, prefer streaming over `bytes()`. ## Per-tenant isolation `X-Project-ID` is mandatory on every storage operation. Build the prefix from request context: ```python theme={null} from fastapi import Depends, HTTPException from typing import Annotated from .auth import current_user, project_id def storage_prefix( user: Annotated[dict, Depends(current_user)], proj: Annotated[str, Depends(project_id)], solution: str = "hello-solution", ) -> str: org = user.get("org_id") or org_id_from_issuer(user["iss"]) if not org or not proj: raise HTTPException(400, "Missing org_id or project_id") return f"{org}/{proj}/{solution}" ``` Now every read/write is scoped — there is no path a request can construct that crosses tenants. See [Platform › Multi-tenancy](/platform/multi-tenancy) for the org/project model. ## Local development with MinIO The [Local Development](/guides/solution-dev/local-development#mode-1-docker-compose-with-hot-reload) page shows the docker-compose snippet. Pre-create the bucket once: ```bash theme={null} docker run --rm --network host minio/mc \ sh -c "mc alias set local http://localhost:9000 minioadmin minioadmin && \ mc mb -p local/-artifacts" ``` Then set `OBSTORE_BACKEND=s3`, `OBSTORE_ENDPOINT=http://localhost:9000`, `OBSTORE_PATH_STYLE=true`. The same code that talks to AWS S3 in prod talks to MinIO in dev. ## Common errors Set `OBSTORE_PATH_STYLE=true`. MinIO doesn't support virtual-hosted-style addressing without DNS configuration. Check that the bucket policy / Workload Identity SA grants the required action. For GCS Workload Identity, verify the K8s ServiceAccount is annotated with `iam.gke.io/gcp-service-account=` and the GSA has `roles/storage.objectAdmin` on the bucket. AWS S3 SDKs require a region; set `OBSTORE_REGION` even if you're targeting a non-AWS endpoint via `OBSTORE_ENDPOINT`. Your `storage_prefix` helper isn't being used everywhere, OR your route accepts a user-controlled key without prefixing. Audit by grepping for `obs.get_async\|obs.put_async\|obs.delete_async` in your service and confirming every call site composes the path through `storage_prefix(...)`. ## Verification ```bash theme={null} # Round-trip test (local with MinIO) PUT_RESULT=$(curl -s -X PUT \ -H "Authorization: Bearer $TOKEN" -H "X-Project-ID: $PROJECT_ID" \ http://localhost:8000/storage/test.txt --data-binary "hello") GET_RESULT=$(curl -s \ -H "Authorization: Bearer $TOKEN" -H "X-Project-ID: $PROJECT_ID" \ http://localhost:8000/storage/test.txt) [ "$GET_RESULT" = "hello" ] && echo "round-trip OK" || echo "FAIL" ``` ## Next steps How customers register storage and DB connections. Org/project isolation — what enforces it where. Storage credentials follow the same secret-injection pattern. More storage error scenarios. # SSO Integration Source: https://docs.emergence.ai/guides/sso-integration How to configure Single Sign-On with Keycloak using OIDC or SAML for enterprise identity providers like Entra ID, Okta, and Google Workspace. # Configure SSO with Keycloak CRAFT uses **Keycloak** as its identity provider, supporting Single Sign-On (SSO) via both **OIDC** and **SAML 2.0** protocols. This guide covers configuring SSO for enterprise identity providers including Microsoft Entra ID (Azure AD), Okta, Google Workspace, and PingFederate. ## Prerequisites Before you begin, ensure you have: * Administrative access to the Keycloak admin console * Administrative access to your enterprise identity provider (IdP) * The Keycloak realm ID for your organization (realm ID = organization ID) * Network connectivity between Keycloak and the IdP (HTTPS required) ## How SSO Works in CRAFT Each organization in the platform maps to a **Keycloak realm**. When SSO is configured: 1. Users navigate to the platform login page 2. Keycloak redirects to the enterprise IdP for authentication 3. The IdP authenticates the user and returns claims to Keycloak 4. Keycloak issues a JWT containing `org_id` (from the realm) and group memberships 5. The platform uses the JWT for authorization via OpenFGA Per-tenant SSO configuration means each organization can connect to a different identity provider. One organization might use Okta while another uses Entra ID. ## Configure OIDC SSO OIDC is the recommended protocol for most identity providers. 1. Navigate to **Azure Portal** > **Microsoft Entra ID** > **App registrations** > **New registration** 2. Set the **Redirect URI** to: `https:///realms//broker/entra-id/endpoint` 3. Note the **Application (client) ID** and **Directory (tenant) ID** 4. Under **Certificates & secrets**, create a new client secret and note the value 1. In the app registration, go to **Token configuration** > **Add groups claim** 2. Select **Security groups** and set the token type to **ID** 3. Ensure the groups claim format is set to **Group ID** 1. Log in to Keycloak Admin Console 2. Select the realm matching your organization ID 3. Navigate to **Identity Providers** > **Add provider** > **OpenID Connect v1.0** 4. Configure the following: | Field | Value | | ----------------- | --------------------------------------------------------------------- | | Alias | `entra-id` | | Authorization URL | `https://login.microsoftonline.com//oauth2/v2.0/authorize` | | Token URL | `https://login.microsoftonline.com//oauth2/v2.0/token` | | Client ID | Your Entra ID application client ID | | Client Secret | Your Entra ID client secret | | Default Scopes | `openid email profile` | In the IdP configuration, add an **Attribute Importer** mapper: * **Mapper Type:** Attribute Importer * **Claim:** `groups` * **User Attribute Name:** `groups` 1. Navigate to **Okta Admin** > **Applications** > **Create App Integration** 2. Select **OIDC - OpenID Connect** and **Web Application** 3. Set the **Sign-in redirect URI** to: `https:///realms//broker/okta/endpoint` 4. Note the **Client ID** and **Client secret** 1. In the application settings, go to **Sign On** > **OpenID Connect ID Token** 2. Set **Groups claim type** to **Filter** 3. Set **Groups claim filter** to match the groups you want to sync 1. Select the realm matching your organization ID 2. Navigate to **Identity Providers** > **Add provider** > **OpenID Connect v1.0** 3. Configure: | Field | Value | | ----------------- | ------------------------------------------- | | Alias | `okta` | | Authorization URL | `https:///oauth2/v1/authorize` | | Token URL | `https:///oauth2/v1/token` | | Client ID | Your Okta client ID | | Client Secret | Your Okta client secret | | Default Scopes | `openid email profile groups` | 1. Navigate to **Google Cloud Console** > **APIs & Services** > **Credentials** 2. Create an **OAuth 2.0 Client ID** of type **Web application** 3. Add the redirect URI: `https:///realms//broker/google/endpoint` 4. Note the **Client ID** and **Client secret** 1. Select the realm matching your organization ID 2. Navigate to **Identity Providers** > **Add provider** > **Google** 3. Enter the Client ID and Client Secret 4. Set **Default Scopes** to `openid email profile` ## Configure SAML 2.0 SSO For identity providers that prefer SAML, Keycloak supports SAML 2.0 as well. Download the SAML metadata from your Keycloak realm: ``` https:///realms//protocol/saml/descriptor ``` Upload this XML to your IdP as the service provider metadata. In your SAML IdP, configure the following: | Field | Value | | -------------- | ----------------------------------------------------------------- | | Entity ID | `https:///realms/` | | ACS URL | `https:///realms//broker//endpoint` | | Name ID format | `urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress` | 1. Navigate to **Identity Providers** > **Add provider** > **SAML v2.0** 2. Import the IdP's SAML metadata XML or manually configure: * **Single Sign-On Service URL** * **Single Logout Service URL** (optional) * **Signing certificate** 3. Set **NameID Policy Format** to `email` ## Just-In-Time User Provisioning Keycloak supports **JIT provisioning** by default. When a user authenticates via SSO for the first time: 1. Keycloak creates a local user account in the realm 2. User attributes (email, name) are synced from the IdP 3. Group memberships are mapped from IdP claims to Keycloak groups 4. OpenFGA relationships are established based on group-to-role mappings JIT provisioning eliminates the need to pre-create user accounts. Users are provisioned automatically on their first SSO login. ## Enforce SSO (Disable Password Login) To require SSO for all users in an organization: 1. In the Keycloak admin console, select the organization's realm 2. Navigate to **Authentication** > **Flows** 3. Edit the **Browser** flow to remove or disable the **Username Password Form** 4. Ensure the **Identity Provider Redirector** is enabled and set to your SSO provider Keycloak client credentials for service accounts remain available even when password login is disabled. This ensures CI/CD pipelines and machine-to-machine integrations continue to function. ## Session Management Configure session behavior for your organization: | Setting | Location | Description | | ---------------- | ------------------------- | ------------------------------------------------------- | | SSO Session Idle | Realm Settings > Sessions | Idle timeout before re-authentication (default: 30 min) | | SSO Session Max | Realm Settings > Sessions | Maximum session duration (default: 10 hours) | | Remember Me | Realm Settings > Login | Allow users to extend their session | ## Group-to-Role Mapping Map IdP groups to OpenFGA roles for authorization: | IdP Group | Keycloak Group | OpenFGA Role | | ----------------- | -------------- | ------------ | | `Platform-Admins` | `admins` | `admin` | | `Developers` | `developers` | `developer` | | `Data-Analysts` | `members` | `member` | | `Viewers` | `viewers` | `viewer` | The mapping pipeline flows: **IdP groups** -> **Keycloak groups** -> **OpenFGA relations**. See [RBAC Configuration](/guides/rbac-configuration) for setting up the OpenFGA side. ## Troubleshooting Verify the redirect URI in your IdP matches the Keycloak broker endpoint exactly, including the protocol (HTTPS) and realm ID. Check for trailing slashes. Ensure group claims are configured in the IdP and a group mapper is set up in the Keycloak identity provider configuration. Verify the claim name matches between the IdP and the Keycloak mapper. Groups must be mapped from the IdP through Keycloak to OpenFGA. If the group-to-role mapping pipeline is not configured, users will authenticate but have no permissions. See the Group-to-Role Mapping section above. ## Next Steps Configure OpenFGA roles and permissions to complete the authorization setup. Learn about OIDC flows, token management, and machine-to-machine authentication. Automate user provisioning with SCIM 2.0 for SOC 2 compliance. Understand how SSO integrates with the multi-tenant architecture. # Agent Registry Source: https://docs.emergence.ai/platform/agents Unified registry for discovering, managing, and monitoring A2A agents, MCP servers, and agentskills.io skills at enterprise scale. # Agent Registry The Agent Registry provides a discovery layer for AI agents and tools. The core A2A Agent Registry is available now with full CRUD, fine-grained authorization, and event streaming. The registry fills the explicit gap in the A2A specification, which acknowledges the need for curated registries but does not prescribe a standard API for them. The A2A specification states: *"The current A2A specification does not prescribe a standard API for curated registries"* and *"The A2A community explores standardizing registry interactions or advanced discovery protocols."* The Agent Registry is designed to fill this gap with enterprise-grade capabilities. ## The Discovery Problem The A2A protocol v1.0 defines three agent discovery strategies, two of which are limited: | Strategy | Limitation | | --------------------------------------------------- | ------------------------------------------------- | | **Well-Known URI** (`/.well-known/agent-card.json`) | Requires knowing the agent's domain upfront | | **Curated Registries** | Acknowledged as needed but explicitly unspecified | | **Direct Configuration** | Hardcoded and inflexible | Meanwhile, the MCP ecosystem has its own public registry, and agentskills.io provides a portable skill format. These three ecosystems are complementary but fragmented -- no unified discovery layer exists. ## Key Capabilities RESTful CRUD for Agent Cards with full schema validation, optimistic concurrency control, and fine-grained authorization. Agents can register, update, and manage their A2A-compliant Agent Cards through the registry API. Auto-detects A2A protocol version (0.3 vs 1.0) from card structure. SHA-256 card hash for duplicate detection. **API surface:** | Method | Endpoint | Description | | ------ | ------------------------ | ------------------------------------------------ | | POST | `/agents` | Register agent (201 + Location + ETag) | | GET | `/agents` | List with pagination, status/tag/version filters | | GET | `/agents/{resource_uri}` | Get by URI (ETag) | | PUT | `/agents/{resource_uri}` | Update card with If-Match version guard | | DELETE | `/agents/{resource_uri}` | Permanent delete, cleans up authorization tuples | | POST | `/agents/validate` | Dry-run card validation (no DB write) | Periodic multi-level health probes ensure agents are reachable and functioning: | Level | Check | Purpose | | ------ | --------------------- | ---------------------------------------------------- | | **L1** | Endpoint reachability | Is the agent's URL responding? | | **L2** | Card freshness | Has the Agent Card been updated recently? | | **L3** | Protocol probe | Does the agent respond to A2A/MCP protocol requests? | All probes include SSRF protection to prevent internal network scanning. Track agents through their full lifecycle with administrative status and lifecycle stages: **Administrative Status:** PENDING, ACTIVE, SUSPENDED, DEREGISTERED **Lifecycle Stage:** EXPERIMENTAL, STABLE, DEPRECATED, RETIRED Immutable version snapshots are created on every update to an agent's registration. The registry supports version diffing and rollback to any previous version. Enterprise-grade controls including: * Approval workflows for new agent registrations * Registration policies and change windows * Audit logging of all operations * Compliance reporting * Tenant-scoped administration with optional cross-tenant federation ## Architecture ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% stateDiagram-v2 [*] --> PENDING : POST /agents/register PENDING --> ACTIVE : L1+L2+L3 health probes pass ACTIVE --> SUSPENDED : admin suspend or health failure SUSPENDED --> ACTIVE : admin reactivate ACTIVE --> DEPRECATED : lifecycle transition DEPRECATED --> RETIRED : end-of-life SUSPENDED --> DEREGISTERED : admin delete ACTIVE --> DEREGISTERED : admin delete RETIRED --> DEREGISTERED : admin delete DEREGISTERED --> [*] ``` The Agent Registry follows a stateless API architecture backed by proven platform infrastructure, OIDC authentication, fine-grained authorization, event streaming, and relational storage. ## Artifact Preview Agents produce artifacts (query results, generated reports, intermediate data) that the platform can preview without forcing a full download. The Assets service exposes a preview endpoint that returns a structured snapshot of an artifact. **Defaults** (effective with em-runtime 5.3+): * Default row limit: **500 rows** (raised from 10) * Maximum byte budget: **512 KB** per response (raised from 64 KB) * Maximum row limit: **10,000 rows** (cap regardless of caller request) **Response metadata** (`PreviewMetadata`): * `size_bytes`: total file size for tabular, text, and raw artifacts (always present) * `truncated`: boolean flag set when either the row limit or the byte budget was reached The service applies a byte-level truncation backstop after row-level limits, so a response that cuts cleanly at a record boundary still respects `max_bytes`. Callers should always check `truncated` before assuming a preview reflects the full artifact. ## MCP Tool Integrations The em-runtime MCP server exposes tool sets that agents can invoke via the Model Context Protocol. These tools proxy authenticated calls to platform services using the caller's JWT and project context. ### Talk2Data Tools When `TALK2DATA_BASE_URL` is configured, the following tools are available: | Tool | Description | | --------------------------- | ---------------------------------------------------------------------------------------- | | `generate_sql_mcp` | Generate SQL from a natural-language question against a specified data source and schema | | `execute_query_mcp` | Execute a SQL query against a data connection and return results | | `resolve_term_mcp` | Resolve a business term to its canonical definition using the data catalog | | `generate_plotly_chart_mcp` | Generate a Plotly chart specification from query results | These tools forward requests to the Talk2Data Service using the caller's bearer token and `X-Project-ID` header. If `TALK2DATA_BASE_URL` is not set, the tools return a `talk2data_not_configured` error. See [Data Insights Overview](/data-insights/overview) for Talk2Data deployment. ### Catalog Tools The MCP server also exposes metadata and search tools against the Utils service (data catalog): * **`get_schema_mcp`** — Get schema metadata for a data connection entity by FQN * **`list_metadata_mcp`** — List metadata assets with filtering * **`search_metadata_mcp`** — Full-text search across the data catalog ## Webhook Events The registry emits events after successful registration operations. | Event Category | Examples | | ---------------- | ----------------------------------------------- | | **Registration** | Agent registered, updated, deregistered | | **Health** | Health status changed, probe failed | | **Lifecycle** | Stage transition (e.g., EXPERIMENTAL to STABLE) | | **Search** | New agent matches a saved search filter | ## Well-Known URI Crawling The registry can automatically discover agents from configured domains by crawling their `/.well-known/agent-card.json` endpoints: * Scheduled crawls with configurable intervals * Caching to minimize network traffic * **Drift detection** to identify when a remote Agent Card changes * SSRF protection for all outbound requests ## Multi-Tenancy The registry supports full multi-tenant isolation: * **Namespace isolation** -- each tenant's agents are invisible to other tenants * **Tenant-scoped administration** -- per-tenant admin roles and policies * **Cross-tenant federation** -- optional sharing of selected agents across tenants * **Cost attribution** -- per-tenant usage metering, quota enforcement, and chargeback ## Target Personas | Persona | Use Case | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | | **Agent Developer** | Register, update, and manage A2A Agent Cards. Validate cards before publishing. Track health and manage versions. | | **MCP Server Developer** | Register MCP server metadata (tools, resources, prompts). Validate tool schemas. Publish to both this registry and the MCP public registry. | | **Consumer** | Search by skill tags, keywords, provider, capabilities, and protocol. Filter by health and verification status. | | **Platform Operator** | Deploy and scale the registry. Monitor health, latency, and throughput. Configure health check intervals. | | **Enterprise Admin** | Control registration policies. Configure approval workflows. Audit operations. Enforce security policies. | ## Success Metrics | Metric | Target | | ---------------------------- | ------------------------ | | API availability | 99.9% monthly | | Search p99 latency | \< 500ms | | Registration p99 latency | \< 1000ms | | Search index lag | \< 30 seconds | | Webhook first delivery | \< 60 seconds | | Registered entities capacity | 100K+ agents and servers | | Search throughput | 1K+ queries/second | ## Next Steps Configure data source connections used by agents for data access. Set up webhook notifications for agent lifecycle events. See how Data Insights uses A2A agents for conversational analytics. Understand authentication, authorization, and multi-tenant isolation. # Authentication Source: https://docs.emergence.ai/platform/authentication Keycloak multi-realm OIDC/PKCE authentication with JWT token validation, service accounts, and SSO for CRAFT. # Authentication CRAFT uses **Keycloak** as its identity provider, configured in a **multi-realm architecture** where each organization is a separate Keycloak realm. Authentication follows the **OpenID Connect (OIDC)** protocol with **PKCE** (Proof Key for Code Exchange) for browser-based clients. Keycloak is self-hosted -- the platform's identity layer runs on your infrastructure and does not depend on a cloud-native identity service. You can federate external enterprise identity providers (Entra ID, Okta, Google Workspace) through standard OIDC and SAML protocols. ## Multi-Realm Architecture Each organization maps to a dedicated Keycloak realm: ```text theme={null} Keycloak Server master realm -- Platform developers and service accounts acme-corp realm -- Acme Corporation users globex realm -- Globex Industries users initech realm -- Initech users ``` This provides complete identity isolation: * Users in one realm cannot see or access users in another * Each realm has independent password policies, session settings, and identity providers * Group memberships are realm-scoped and appear as JWT claims * SSO can be configured per-realm (e.g., SAML, LDAP, social login) ## Authentication Flow ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': { 'actorBkg': '#56B4E9', 'actorBorder': '#555555', 'actorTextColor': '#000000', 'actorLineColor': '#555555', 'signalColor': '#555555', 'signalTextColor': '#555555', 'labelBoxBkgColor': '#EEEEEE', 'labelBoxBorderColor': '#555555', 'labelTextColor': '#555555', 'loopTextColor': '#555555', 'noteBkgColor': '#F0E442', 'noteBorderColor': '#555555', 'noteTextColor': '#000000', 'activationBkgColor': '#E69F00', 'activationBorderColor': '#555555', 'fontFamily': 'sans-serif' }}}%% sequenceDiagram participant B as Browser participant KC as Keycloak Realm participant API as Platform API B->>B: Generate PKCE code_verifier + code_challenge B->>KC: Authorization request + code_challenge KC->>B: Redirect to login page B->>KC: User credentials KC-->>B: Authorization code (redirect) B->>KC: Token request + code_verifier KC-->>B: Access token (JWT) + refresh token + ID token B->>API: API request + Authorization: Bearer JWT API->>API: Validate JWT, extract org_id from realm API-->>B: Response ``` The client discovers the realm's OpenID configuration at: ```text theme={null} https://keycloak.example.com/realms/{org_id}/.well-known/openid-configuration ``` The browser initiates the OIDC authorization code flow with PKCE. A code verifier is generated client-side and the SHA-256 code challenge is sent with the authorization request. No client secret is required. The user authenticates at the Keycloak login page for their realm. Keycloak supports username/password, SSO via external IdPs (SAML, OIDC federation), LDAP, and social login. After successful login, Keycloak returns an authorization code. The client exchanges it (with the code verifier) for: * **Access token (JWT)** -- Used for API authorization * **Refresh token** -- Used to obtain new access tokens * **ID token** -- Contains user identity claims All API requests include the access token in the `Authorization: Bearer` header. The platform validates the token and extracts the organization context from the realm. ## JWT Token Structure The access token contains standard OIDC claims plus platform-specific claims: ```json theme={null} { "sub": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "iss": "https://keycloak.example.com/realms/acme-corp", "aud": "account", "azp": "em-runtime-ui", "exp": 1743696000, "iat": 1743692400, "preferred_username": "jane.smith", "email": "jane.smith@acme-corp.com", "name": "Jane Smith", "groups": ["/org-admins", "/project-developers"], "realm_access": { "roles": ["default-roles-acme-corp"] } } ``` Key claims used by the platform: | Claim | Purpose | | ------------- | -------------------------------------------------------------------------- | | `sub` | User ID for OpenFGA permission checks | | `iss` | Realm extraction -- the path segment after `/realms/` becomes `org_id` | | `groups` | Keycloak group memberships, used for group-based OpenFGA permission checks | | `azp` | Client ID of the application that requested the token | | `exp` / `iat` | Token expiration and issuance timestamps | ## Token Validation The Governance service validates tokens using a multi-step process: 1. **Extract realm** -- The token's `iss` claim is decoded (without verification) to determine the Keycloak realm 2. **Fetch JWKS** -- Public keys are fetched from the realm's JWKS endpoint (cached per realm via `lru_cache`) 3. **Verify signature** -- The token signature is validated against the realm's public key, which cryptographically proves the token was issued by the platform's Keycloak instance 4. **Check expiration** -- Standard JWT expiration (`exp`) validation 5. **Verify issuer consistency** -- The issuer from the verified token must match the initial unverified extraction 6. **Extract claims** -- `org_id`, groups, roles, and user identity are extracted into the `Auth` object * **Issuer hostname is not validated separately** -- JWKS signature verification is strictly stronger. A token signed by the platform's Keycloak private key is accepted regardless of the hostname in the issuer URL. This allows tokens issued via any entrypoint (internal service URL, API gateway, external hostname) to be accepted. * **Realm-specific validation** -- Each realm has its own signing keys. A token from `realm-a` cannot be validated against `realm-b`'s JWKS. * **Group normalization** -- Leading slashes are stripped from group names (`/org-admins` becomes `org-admins`). ## Service Accounts Service accounts are used by background workers and automated processes that need to access the platform without user interaction. They are identified by a three-check detection. See [Service Accounts](/platform/service-accounts) for a complete guide including token management, creation steps, and code examples. Service accounts authenticate via the Keycloak `master` realm, not an organization realm. Their `org_id` is `null` by default. Service account client IDs must start with the `svc-` prefix (e.g., `svc-data-readiness`, `svc-talk2data`). This is a naming convention enforced by the detection logic. Service accounts must have the `serviceAccount` role in `realm_access.roles`. This provides defense in depth -- even if someone creates a client with a `svc-` prefix, it will not be treated as a service account without the role. Service accounts use additional headers to establish context: | Header | Purpose | Required | | ---------------- | -------------------------------------------------------------------------- | -------- | | `X-Org-Id` | Specifies the organization context (since master realm tokens have no org) | Yes | | `X-On-Behalf-Of` | Identifies the original user in audit logs when acting on their behalf | Optional | The `X-On-Behalf-Of` and `X-Org-Id` headers are only trusted when the caller passes all three service account checks. Regular user tokens cannot use these headers to impersonate other users or switch organizations. ## SSO Integration Each Keycloak realm can be configured with external identity providers for single sign-on: | Provider Type | Configuration | | ------------------- | --------------------------------------------------------- | | **SAML 2.0** | Enterprise IdP integration (Okta, Azure AD, PingFederate) | | **OIDC Federation** | Connect to another OIDC provider | | **LDAP/AD** | Sync users from Active Directory or LDAP | | **Social Login** | Google, GitHub, Microsoft (per-realm configuration) | Since each organization is a separate realm, SSO configuration is fully tenant-isolated. One organization can use SAML with Okta while another uses LDAP with Active Directory. ## Obtaining Tokens ```bash theme={null} # Get a JWT token for the default admin user make get-token # Customize organization or credentials ORGANIZATION_ID=acme-corp \ ADMIN_USERNAME=jane \ ADMIN_PASSWORD=secret \ make get-token ``` ```bash theme={null} # Get a Keycloak platform admin token (master realm) make get-platform-token ``` ```python theme={null} from keycloak import KeycloakOpenID keycloak = KeycloakOpenID( server_url="https://keycloak.example.com/", client_id="my-app", realm_name="acme-corp", ) # Resource Owner Password Grant (dev/testing only) token = keycloak.token("username", "password") access_token = token["access_token"] ``` ## Next Steps Learn how JWT claims are used for OpenFGA permission checks. Understand how organizations map to Keycloak realms. See the full tenant isolation architecture. Learn how the X-Project-ID header provides project context. # Authorization Source: https://docs.emergence.ai/platform/authorization OpenFGA Relationship-Based Access Control (ReBAC) with relationship tuples, computed permissions, hierarchical inheritance, and the require_permission pattern. # Authorization CRAFT uses **OpenFGA** for authorization, implementing **Relationship-Based Access Control (ReBAC)**. Unlike traditional RBAC where roles grant flat permission sets, ReBAC models permissions as relationships between users and resources, enabling fine-grained, hierarchical access control. OpenFGA is the open-source implementation of Google's Zanzibar authorization system. It evaluates permission checks in single-digit milliseconds, making it suitable for per-request authorization in API endpoints. ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TD U["User / Group"] O["Organization"] P["Project"] A["Agent"] DC["Data Connection"] MCP["MCP Server"] U -- "owner / admin / member" --> O O -- "owns" --> P U -- "developer / operator / viewer" --> P P -- "parent_of" --> A P -- "parent_of" --> DC P -- "parent_of" --> MCP style U fill:#56B4E9,stroke:#555555,color:#000 style O fill:#E69F00,stroke:#555555,color:#000 style P fill:#E69F00,stroke:#555555,color:#000 style A fill:#009E73,stroke:#555555,color:#000 style DC fill:#009E73,stroke:#555555,color:#000 style MCP fill:#009E73,stroke:#555555,color:#000 ``` ## Core Concepts A relationship tuple is the fundamental building block. It states: *"This user/group has this relation on this resource."* ```text theme={null} user:jane owner organization:acme-corp group:org-admins admin organization:acme-corp user:bob developer project:analytics-prod ``` Tuples are stored in OpenFGA and queried during permission checks. The authorization model defines types for every resource in the platform: | Type | Service | Description | | ----------------- | ---------- | ------------------------------------------- | | `organization` | Governance | Top-level tenant boundary | | `project` | Governance | Resource grouping within an organization | | `artifact` | Assets | Generated outputs (reports, visualizations) | | `file` | Assets | Uploaded files and documents | | `data_connection` | Assets | External data source connections | | `mcp_server` | Assets | Model Context Protocol servers | | `api_server` | Assets | API server registrations | | `model` | Assets | ML model registrations | Relations represent the roles a user or group can have on a resource: | Relation | Scope | Description | | ----------- | --------------------- | ---------------------------------------------- | | `owner` | All types | Full control including deletion | | `admin` | Organization, Project | Administrative access, user/project management | | `member` | Organization | Standard membership with read access | | `developer` | Project, Resources | Create and manage resources | | `operator` | Project, Resources | Execute and operate resources | | `viewer` | Project, Resources | Read-only access | Computed permissions are derived from relations using OR logic. For example, `can_read` is granted to anyone with `viewer`, `operator`, `developer`, `admin`, or `owner` relation: ```text theme={null} define can_read: viewer or operator or developer or admin or owner ``` The full set of computed permissions: | Permission | Description | | ---------------------- | --------------------------------------------------------------- | | `can_read` | View the resource | | `can_write` | Modify the resource | | `can_delete` | Remove the resource | | `can_execute` | Run/invoke the resource (agents, MCP servers, data connections) | | `can_create_resources` | Create child resources (project-level) | | `can_share` | Share with other users | | `can_read_secrets` | View associated secrets | | `can_manage_secrets` | Create/update/delete secrets | | `can_read_metadata` | View resource metadata | | `can_manage_metadata` | Modify resource metadata | | `can_manage_projects` | Create/delete projects (organization-level) | | `can_manage_users` | Manage user memberships (organization-level) | ## Permission Inheritance The authorization model uses OpenFGA's `from` keyword to implement hierarchical permission inheritance: ```text theme={null} Projects inherit from Organizations Resources inherit from Projects Resource permissions can compute from Project permissions (e.g., can_delete from project) ``` This means an organization `admin` automatically has appropriate access to all projects and resources within their organization, without needing explicit tuples on each resource: ```bash theme={null} # This single tuple: group:org-admins admin organization:acme-corp # Grants (via inheritance): # - can_read on all projects in acme-corp # - can_write on all projects in acme-corp # - can_delete on all projects in acme-corp # - can_read on all resources in those projects # - can_write on all resources in those projects # - ... and so on ``` The inheritance chain is established via parent link tuples: ```yaml theme={null} project:analytics#organization@organization:acme-corp data_connection:pg-prod#project@project:analytics ``` ### Computed delegation: `can_delete from project` For resource types like `data_connection`, the schema computes resource-level permissions from project-level permissions. For example, `can_delete` on a data connection is satisfied if the user has delete rights on the parent project: ```text theme={null} type data_connection relations define project: [project] define can_delete: owner or admin or can_delete from project define can_execute: operator or developer or admin or owner or can_execute from project ``` This means an organization owner who has `can_delete` on every project (via the org-from-project chain above) can also delete every project-scoped resource without per-resource tuples. ### Service worker permissions on `data_connection` Service workers (background tasks like the data-readiness worker, scheduled enrichment runs, and dataset sync jobs) need to invoke data connections. The schema grants `can_execute` on `data_connection` to the service-worker role at the project level so workers can `POST /assets/data/{resource_uri}/verify` and `GET /assets/data/{resource_uri}/secret` against any connection in their assigned project, without needing a per-connection tuple. This follows the same pattern as `can_create_resources` on `project` for service workers: a single role binding at the parent grants narrow operational permissions across all child resources. ## Authorization Schema (OpenFGA DSL) The schema is defined in `openfga-schema.fga` and compiled to `openfga-schema.json` for runtime use. Here is a simplified excerpt showing the project type: ```text theme={null} type project relations define organization: [organization] # Direct assignments define owner: [user, group#member, role#assignee] define admin: [user, group#member, role#assignee] define developer: [user, group#member, role#assignee] define operator: [user, group#member, role#assignee] define viewer: [user, group#member, role#assignee] # Computed permissions with inheritance define can_read: viewer or operator or developer or admin or owner or admin from organization or owner from organization define can_write: operator or developer or admin or owner or admin from organization or owner from organization define can_delete: owner or admin or owner from organization or admin from organization define can_create_resources: developer or admin or owner or admin from organization or owner from organization ``` When modifying the OpenFGA schema, always update both the `.fga` DSL file and regenerate the `.json` runtime file by running `./scripts/generate-openfga-schema.sh`. The Governance service reads only the JSON file at startup. Adding new permissions is safe; renaming or removing is a **breaking change**. ## The require\_permission Pattern All API endpoints use the `require_permission()` dependency to enforce authorization. This is implemented in the shared `common` package and used consistently across all services: ```python theme={null} from common.permissions import Permissions, require_permission @router.delete("/data/{resource_uri}") async def delete_data_connection( resource_uri: str, _: None = Depends(require_permission( Permissions.DATA_CONNECTION, # Resource type Permissions.DATA_CONNECTION.actions.CAN_DELETE, # Required permission "resource_uri" # Path parameter with resource ID )) ): ... ``` ```python theme={null} from common.permissions import Permissions, require_permission, HEADER_PROJECT_ID @router.post("/data") async def create_data_connection( payload: DataConnectionCreateRequest, _: None = Depends(require_permission( Permissions.PROJECT, # Check on the project Permissions.PROJECT.actions.CAN_CREATE_RESOURCES, # Required permission HEADER_PROJECT_ID # From X-Project-ID header )) ): ... ``` ```python theme={null} from common.permissions import require_permission_dynamic @router.get("/{resource_type}/{resource_id}/secrets") async def list_secrets( resource_type: str, resource_id: str, _: None = Depends(require_permission_dynamic( resource_type_param="resource_type", # Resolved from path parameter permission="can_read_secrets", resource_id_param="resource_id" )) ): ... ``` ## Permission Check Flow When `require_permission()` is invoked, the following occurs: The resource ID is resolved from path parameters, query parameters, or headers based on the configuration. The `Auth` object is extracted from the validated JWT, providing `sub` (user ID) and `teams` (group memberships). The permission checker queries OpenFGA with: * **User:** `user:{sub}` and each `group:{group_name}#member` * **Relation:** The computed permission (e.g., `can_delete`) * **Object:** `{resource_type}:{resource_id}` If any user or group check returns `true`, access is granted. * **200/success:** Permission granted, the endpoint handler executes * **403 Forbidden:** Permission denied, the request is rejected * **500 Internal Server Error:** Permission system misconfigured ## Permission API The Governance service exposes a Permissions API for programmatic permission management: | Endpoint | Method | Description | | ------------------------------------ | ------ | ------------------------------------------------------- | | `/governance/permissions/check` | GET | Check if a user has a specific permission on a resource | | `/governance/permissions/grant` | POST | Grant a relation to a user or group on a resource | | `/governance/permissions/revoke` | POST | Revoke a relation from a user or group | | `/governance/permissions/set-parent` | POST | Establish a parent-child relationship for inheritance | | `/governance/permissions/delete-all` | POST | Remove all permission tuples for a resource (cleanup) | Assets and Utils never call the Permissions API directly. They use the `require_permission()` dependency, which internally forwards the user's JWT to the Governance service for permission validation via the auto-generated SDK. ## Keycloak Groups and OpenFGA The bridge between Keycloak (identity) and OpenFGA (authorization) is **Keycloak groups**: 1. Users are assigned to groups in Keycloak (e.g., `org-admins`, `project-developers`) 2. A groups mapper on the realm ensures group memberships appear in the JWT `groups` claim 3. During bootstrap, permission tuples map groups to OpenFGA relations (e.g., `group:org-admins#member` -> `admin` on `organization`) 4. At runtime, `require_permission()` checks both the user's direct tuples and their group-based tuples This means adding a user to a Keycloak group is sufficient to grant them the corresponding platform permissions -- no separate OpenFGA management is needed. ## Next Steps Understand how JWT tokens carry group claims for authorization. See how organization-level permissions are bootstrapped. Learn about project-level role assignments and inheritance. Explore the full tenant isolation architecture. # Data Connections Source: https://docs.emergence.ai/platform/data-connections Manage external data source connections with secure credential storage, connection verification, and fine-grained access control. # Data Connections Data connections represent configured links to external data sources: databases, object storage, and file systems. They are managed by the **Assets** service (port 8000) and provide a secure, unified interface for solutions to discover and access external data without handling raw credentials. Connections can be scoped to a single project or to the entire organization (org-level). See [Org-level data connections](#org-level-data-connections) below. Credentials for data connections are stored securely via the Governance Secrets API (backed by Infisical or ESO + GCP Secret Manager). The connection record itself stores only metadata; credentials are never persisted in the Assets database. ## Supported Connection Types | Type | Status | Description | | ------------ | ----------- | --------------------------------------------------------------------------------------------------------------- | | `postgres` | Implemented | PostgreSQL databases | | `redshift` | Implemented | Amazon Redshift data warehouses | | `snowflake` | Implemented | Snowflake data warehouses (Basic + JWT/key-pair auth available; OAuth and SSO planned based on customer demand) | | `mysql` | Planned | MySQL databases | | `bigquery` | Planned | Google BigQuery datasets | | `databricks` | Planned | Databricks SQL warehouses | | `couchbase` | Planned | Couchbase NoSQL databases | | `oracle` | Planned | Oracle databases | | `hive` | Planned | Apache Hive metastore | | `mongodb` | Planned | MongoDB NoSQL databases | Database connections support: * Schema and table discovery during verification (note: Snowflake performs server-level discovery and does not enumerate schemas/tables ahead of time) * Selected table filtering * Connection pooling configuration | Type | Status | Description | | ------- | ----------- | --------------------------------- | | `s3` | Implemented | Amazon S3 or S3-compatible stores | | `gcs` | Implemented | Google Cloud Storage | | `minio` | Implemented | Self-hosted MinIO instances | Storage connections support: * Bucket and prefix configuration * Credential verification * Multiple authentication methods (access key, service account, etc.) | Type | Status | Description | | ----- | ----------- | ------------------------ | | `nfs` | Implemented | NFS-mounted file systems | | `smb` | Planned | SMB/CIFS network shares | File system connections support: * Path-based configuration * Permission validation ## Connection Lifecycle Query `GET /assets/data/types` to retrieve all connection types with their credential schemas, authentication methods, and example configurations. Use `POST /assets/data/verify` to test credentials without creating a connection. For databases, this also discovers available schemas and tables for selection. `POST /assets/data` creates the connection record, stores credentials as a secret via the Governance Secrets API, and sets up ownership permissions in OpenFGA. Solutions retrieve connection metadata via `GET /assets/data/{resource_uri}` and fetch decrypted credentials via `GET /assets/data/{resource_uri}/secret` when executing queries. `POST /assets/data/{resource_uri}/verify` tests that stored credentials are still valid and the data source is accessible. ## Resource URIs Data connections use **resource URIs** as identifiers instead of UUIDs. Resource URIs follow a structured format: ```yaml theme={null} data:{org_id}:{project_id}:{connection_name} ``` For example: `data:acme-corp:analytics-prod:customer-db` If a `resource_uri` is not provided during creation, one is auto-generated from the connection name. ## Org-level data connections Data connections can be scoped to a single project (the default) or to the entire organization. Org-level connections let multiple projects share the same configured data source without per-project duplication. **Creating an org-level connection:** omit the `X-Project-ID` header (or pass `project_id=null` in the request body where supported). The created connection has its `project_id` field unset and lives at the org boundary. **Resource URI shape:** org-level connections drop the project segment. ```yaml theme={null} data:{org_id}::{connection_name} # org-level (note the empty project segment) data:{org_id}:{project_id}:{connection_name} # project-scoped (default) ``` **Permission semantics:** an org-level connection's permissions are evaluated against the organization, not a project. The OpenFGA computed-permission graph extends `can_*` checks across the org so that: * Org owners can read, write, and delete org-level connections. * Members of any project under the organization can `read` and `execute` an org-level connection if their project role grants those permissions on org-shared resources. (See [Authorization](/platform/authorization) for the full chain.) * Cross-project visibility is the explicit point of org-level connections; standard project-scoped connections remain isolated. **When to use:** a database that powers analytics across all projects in the organization, a shared S3 bucket for raw data drops, or any connection where per-project credential management would create operational drag without a security benefit. ## API Reference All endpoints require the `X-Project-ID` header and a valid JWT token. Lists all available connection types with credential schemas. Returns each connection type with its category, description, supported authentication methods, required credential fields, and example configurations. Use this to build dynamic connection forms. Lists data connections in the current project with filtering and pagination. **Access:** `can_read` on the project. **Query Parameters:** * `page`, `limit`: Pagination * `search`: Search in name and description. Whitespace-only search terms return HTTP 400. `%` and `_` characters are treated as literal text (ILIKE-escaped) so a search for `customer_db` matches the literal string, not "customer" + any character + "db". * `connection_type`: Filter by type (postgres, s3, etc.) * `status`: Filter by status (PENDING, ACTIVE, ERROR, DISABLED) * `tags`: Comma-separated tag filter (AND logic) * `sort`: Sort specification (e.g., `name:asc,created_at:desc`) * `owner_id`: Filter by owner Verifies connection credentials before saving. Tests connectivity, validates permissions, and discovers available schemas and tables for databases. **Access:** `can_create_resources` on the project. Currently supports verification for: PostgreSQL, Redshift, S3, GCS, MinIO. Creates a new data connection with inline credentials. **Access:** `can_create_resources` on the project. **Side effects:** * Creates the data connection record in the database * Auto-generates `resource_uri` from name if not provided * Stores credentials as a secret via the Governance Secrets API * Creates an ownership tuple in OpenFGA ```json Request Body theme={null} { "name": "Customer Analytics DB", "connection_type": "postgres", "description": "Production read replica for analytics", "database_config": { "host": "pg-replica.example.com", "port": 5432, "database": "analytics", "schema": "public", "selected_tables": ["customers", "orders", "products"] }, "credentials": { "username": "analytics_reader", "password": "secret" }, "tags": ["production", "analytics"] } ``` Retrieves data connection details (metadata only, no credentials). **Access:** `can_read` on the data connection. Updates a data connection. The `resource_uri` cannot be changed. **Access:** `can_write` on the data connection. Credentials can be updated by providing new values. If credentials are omitted, existing credentials are preserved. Deletes a data connection and all associated resources. **Access:** `can_delete` on the data connection. **Side effects:** * Removes the data connection record * Deletes OpenFGA permission tuples * Deletes associated secrets from the Governance Secrets API Retrieves decrypted credential values for a data connection. **Access:** `can_execute` on the data connection. This elevated permission ensures only authorized automation (agents, workflows) can access raw credentials. Secrets are never cached and access is logged for audit purposes. Verifies that an existing connection's stored credentials are still valid. **Access:** `can_execute` on the data connection. ## Credential Storage Architecture Data connection credentials follow a layered storage model: ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TD A["Assets Service (port 8000)
Data connection record: metadata, config, status
Secret names: [credentials]"] G["Governance Service (port 8000)
Secrets API"] S["Secrets Backend
(Infisical or ESO + GCP Secret Manager)
Encrypted storage · Audit logging · OpenFGA access control"] A -->|"Governance SDK call
(forwarding user JWT)"| G G --> S classDef service fill:#E69F00,stroke:#555555,color:#000 classDef secrets fill:#009E73,stroke:#555555,color:#000 class A,G service class S secrets ``` The Assets service never stores or caches credentials. Every secret retrieval goes through the Governance Secrets API, which enforces its own permission checks and audit logging. ## Data Connection Permissions Data connections support the full set of resource-level permissions: | Permission | Who Can | Use Case | | --------------------- | -------------------------------------------------- | -------------------------------------- | | `can_read` | Viewers, operators, developers, owners + inherited | View connection metadata | | `can_write` | Operators, developers, owners + inherited | Update connection settings | | `can_delete` | Developers, owners + project admin/owner | Remove connections | | `can_execute` | Operators, developers, owners + inherited | Retrieve credentials, run verification | | `can_read_secrets` | Operators, developers, owners + inherited | View associated secrets | | `can_manage_secrets` | Developers, owners + inherited | Update credentials | | `can_read_metadata` | Operators, developers, owners + inherited | View data connection metadata | | `can_manage_metadata` | Developers, owners + inherited | Update metadata and tags | ## Next Steps Learn how agents use data connections to access external data. Understand how data connections are scoped to projects. Explore the permission model for data connections. Automate data connection workflows with scheduled tasks. # LLM Gateway Source: https://docs.emergence.ai/platform/llm-gateway Operator guide for the shared LiteLLM gateway — the sole authority for agent LLM access, model allowlist, rate limits, authentication, and observability # LLM Gateway The CRAFT platform routes agent LLM traffic through a **shared LiteLLM gateway** — a single cluster-level service that holds provider credentials centrally. This page describes the target architecture from the Agent Security ADR and the operator contract for configuring the gateway. This page is for platform operators and solution-team leads. Solution developers looking for how to call an LLM from their solution code should see [Access LLMs](/guides/solution-dev/access-llms) instead. **Target vs. current deployment.** The shared gateway is the recommended model. Current em-talk2data Helm charts do not bundle a managed gateway — agents in those deployments invoke the LiteLLM SDK in-process against Vertex/Bedrock directly. Operators adopting the gateway pattern must deploy it separately and set `OPENAI_API_BASE` / `LITELLM_API_BASE` on agent deployments. ## Architecture ### Shared gateway model (Agent Security ADR) Per the Agent Security ADR, agents are **untrusted execution environments**. The gateway enforces this through two architectural constraints: 1. **No credentials in agent pods.** Provider API keys (OpenAI, Anthropic, Google) live only in the LLM Gateway's environment. Agents never receive or handle these secrets. 2. **All model access is brokered.** Every completion request passes through the gateway's allowlist check, budget enforcement, and audit pipeline before reaching a provider. The LiteLLM gateway is deployed as a **shared Kubernetes Deployment** — one set of gateway replicas serves all agents in the cluster. Agents call it via its internal ClusterIP service address. **Identity sidecar vs. LLM gateway.** The Agent Security ADR also describes a per-agent **identity sidecar** — a lightweight container that holds the workload's Kubernetes ServiceAccount JWT and authenticates the agent to platform gateways (including the LLM Gateway) using short-lived scoped tokens. This is separate from the LLM gateway itself. The identity sidecar is a per-pod security boundary; the LLM gateway is a shared cluster service. ### How requests flow ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR USER["User / orchestrator"] AGENT["Agent pod\n(no provider keys)"] IDSIDECAR["Identity sidecar\n(K8s SA JWT)"] GW["LLM Gateway\n(shared, litellm proxy)"] ALLOW["Allowlist + budget\ncheck"] ROUTE["Provider routing"] LANGFUSE["Langfuse\n(traces + cost)"] VERTEX["Google Vertex AI"] ANTH["Anthropic"] SELFHOST["Self-hosted\n(Ollama / vLLM)"] USER --> AGENT AGENT -->|"LLM_GATEWAY_URL\n(ClusterIP)"| IDSIDECAR IDSIDECAR -->|"scoped token\n+ request"| GW GW --> ALLOW ALLOW -->|"allowed"| ROUTE ALLOW -->|"denied → 403"| IDSIDECAR GW -.auto-traces.-> LANGFUSE ROUTE --> VERTEX ROUTE --> ANTH ROUTE --> SELFHOST classDef user fill:#56B4E9,stroke:#555555,color:#000 classDef agent fill:#E69F00,stroke:#555555,color:#000 classDef infra fill:#0072B2,stroke:#555555,color:#fff classDef decision fill:#F0E442,stroke:#555555,color:#000 classDef store fill:#009E73,stroke:#555555,color:#000 classDef ext fill:#CC79A7,stroke:#555555,color:#000 class USER user class AGENT agent class IDSIDECAR infra class GW infra class ALLOW decision class ROUTE infra class LANGFUSE store class VERTEX,ANTH,SELFHOST ext ``` **Credential isolation in practice:** the LLM Gateway pod holds `VERTEXAI_PROJECT`, `ANTHROPIC_API_KEY`, etc. The agent pod's only gateway-related secrets are `LLM_GATEWAY_URL` (a ClusterIP service address) and `LLM_GATEWAY_API_KEY` (a scoped gateway key with no provider privilege). Neither reveals which provider is in use, and neither can be used to call a provider directly. ### em-runtime-mcp as agent tool gateway `em-runtime-mcp` is the **tool-call gateway** — the single endpoint through which every agent invokes a platform tool. The LLM gateway and `em-runtime-mcp` are complementary: | Gateway | Handles | Enforces | | --------------------- | ------------------- | ---------------------------------------------- | | LLM Gateway (LiteLLM) | LLM completions | Model allowlist, rate limits, cost attribution | | `em-runtime-mcp` | Platform tool calls | Per-agent tool allowlist, audit events | ## Model registry ### Allowlist configuration The gateway's `config.yaml` defines the models any agent in the cluster is permitted to call. Requests for unlisted models return `400 Model not allowed`. ```yaml theme={null} # LLM Gateway config.yaml (mounted as a ConfigMap on the gateway Deployment). # CRAFT defaults route to Vertex AI; the gateway is provider-agnostic, so other # providers (OpenAI, Anthropic direct, self-hosted) can be added side-by-side. model_list: # DEFAULT for most solutions and agents. # GA on Vertex AI Model Garden (since Google I/O 2026-05-19). Outperforms # gemini-3.1-pro on 6 of 8 major benchmarks — including agentic (MCP Atlas) # and coding (Terminal-Bench) — at 3.6× the speed and a fraction of the cost. - model_name: gemini-3.5-flash litellm_params: model: vertex_ai/gemini-3.5-flash vertex_project: os.environ/VERTEXAI_PROJECT vertex_location: global # Specialized variant. Use ONLY when custom tool calling is a core # requirement (the model prioritizes registered custom functions over bash # fallbacks). Per Google's own guidance: if >50% of requests don't involve # tool calling, stay on gemini-3.5-flash. Preview SLA; global endpoint only. - model_name: gemini-3.1-pro-preview-customtools litellm_params: model: vertex_ai/gemini-3.1-pro-preview-customtools vertex_project: os.environ/VERTEXAI_PROJECT vertex_location: global # Secondary / cross-provider option — demonstrates model agnosticity - model_name: claude-opus-4-8 litellm_params: model: vertex_ai/claude-opus-4-8 vertex_project: os.environ/VERTEXAI_PROJECT vertex_location: global # Self-hosted (no key required) - model_name: ollama/llama3.1 litellm_params: model: ollama/llama3.1 api_base: http://ollama.ollama.svc.cluster.local:11434 litellm_settings: drop_params: true # tolerate extra params from different SDKs request_timeout: 120 # seconds general_settings: master_key: os.environ/LITELLM_MASTER_KEY ``` Agents call `litellm.acompletion(model="openai/gemini-3.5-flash", api_base=LLM_GATEWAY_URL, ...)` using the `model_name` alias. Switching providers is a gateway config change — no agent code changes required. **Use `gemini-3.1-pro-preview-customtools` only when extensive custom tool calling is core to the agent.** It's a fine-tuned variant that prioritizes registered custom functions over bash fallbacks. Google's own guidance: if > 50% of requests don't involve tool calling, stay on `gemini-3.5-flash` — > quality on non-tool workloads is lower on the customtools variant. Three operational constraints: **(1) global-endpoint only** — `vertex_location` must be `global`. **(2) Lower quota** — configure a fallback chain to `gemini-3.5-flash` or `claude-opus-4-8`. **(3) Preview SLA** — may be deprecated or renamed without GA notice. **Environment-variable conventions.** The gateway reads Vertex credentials from `VERTEXAI_PROJECT` and `VERTEXAI_LOCATION` (LiteLLM's documented convention, matching the em-talk2data Helm chart values). LiteLLM also accepts Google Cloud SDK standards `GOOGLE_CLOUD_PROJECT` + `GOOGLE_CLOUD_LOCATION`. Authenticate the gateway pod to Vertex AI via Workload Identity (production) or `GOOGLE_APPLICATION_CREDENTIALS` pointing to an ADC JSON (local dev). ### Provider routing | Prefix in `litellm_params.model` | Provider | | -------------------------------- | ---------------------------------- | | `vertex_ai/` | Google Vertex AI | | `openai/` | OpenAI | | `anthropic/` | Anthropic (direct) | | `gemini/` | Google Gemini (direct, non-Vertex) | | `ollama/` | Local Ollama | | `vllm/` | vLLM endpoint | | `azure/` | Azure OpenAI | To add fallbacks, list multiple entries sharing the same `model_name`: ```yaml theme={null} - model_name: gemini-3.5-flash litellm_params: model: vertex_ai/gemini-3.5-flash vertex_project: os.environ/VERTEXAI_PROJECT vertex_location: global - model_name: gemini-3.5-flash # fallback for resilience litellm_params: model: vertex_ai/claude-opus-4-8 vertex_project: os.environ/VERTEXAI_PROJECT vertex_location: global ``` ## Authentication ### Gateway API key The gateway enforces a `master_key`. Each project's agents receive a distinct scoped key provisioned by the platform secrets pipeline. The key grants access to the gateway — it carries no provider-level privilege and cannot be used to call a model provider directly. Agents receive the key via their Helm values: ```yaml theme={null} # charts/your-agent/values.yaml env: - name: LLM_GATEWAY_URL value: "http://llm-gateway.craft.svc.cluster.local/v1" - name: LLM_GATEWAY_API_KEY valueFrom: secretKeyRef: name: llm-gateway-keys key: project-key ``` The gateway's own provider secrets (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `VERTEXAI_PROJECT`, etc.) are mounted into the **gateway Deployment only** and are never visible to agent pods. ## Budget enforcement ### Rate limits ```yaml theme={null} router_settings: num_retries: 3 timeout: 120 retry_after: 5 # seconds between retries model_list: - model_name: gemini-3.5-flash litellm_params: model: vertex_ai/gemini-3.5-flash vertex_project: os.environ/VERTEXAI_PROJECT vertex_location: global tpm: 100000 # tokens per minute (cluster-wide) rpm: 500 # requests per minute (cluster-wide) ``` Limits are enforced **cluster-wide** at the shared gateway, not per-agent. When a limit is exceeded, the gateway returns `429 Too Many Requests`; clients back off exponentially. ### Cost ceilings ```yaml theme={null} litellm_settings: max_budget: 50.00 # USD, per model alias budget_duration: monthly ``` ### Overage behavior 1. Gateway returns `429` with a `Retry-After` header. 2. Agent catches `429` and applies exponential back-off with jitter. 3. If the alias has a fallback in `model_list`, the router tries it automatically. 4. If no fallback and budget exhausted, `429` propagates to the caller. Do not remove the `fallbacks` list from the router config without first confirming the upstream has headroom. A saturated primary with no fallback causes a complete LLM outage for **all agents in the cluster** — not just one pod. ## Observability ### Langfuse traces LiteLLM auto-emits traces to Langfuse when these env vars are set on the **gateway Deployment**: ```bash theme={null} LANGFUSE_HOST=https://langfuse.your-cluster.example.com LANGFUSE_PUBLIC_KEY= LANGFUSE_SECRET_KEY= ``` Enable in the gateway config: ```yaml theme={null} litellm_settings: success_callback: ["langfuse"] failure_callback: ["langfuse"] ``` Every completion call generates a Langfuse trace tagged with the `metadata` the agent passes (`project_id`, `solution`, `trace_id`) — the basis for per-project cost attribution dashboards. ### Prometheus and OpenTelemetry metrics LiteLLM exposes `/metrics` (Prometheus format) on port 4000. The platform OTEL Collector scrapes it and forwards to your observability backend. | Metric | Description | | ---------------------------------- | --------------------------------------------- | | `litellm_requests_total` | Total completion requests by model and status | | `litellm_tokens_total` | Total tokens consumed by model | | `litellm_request_duration_seconds` | Latency histogram | | `litellm_spend_usd` | Cumulative spend by model alias | OTLP export from the gateway: ```yaml theme={null} litellm_settings: service_callback: ["otel"] environment_variables: OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4317" OTEL_SERVICE_NAME: "llm-gateway" ``` ## Provider configuration ```yaml theme={null} - model_name: gemini-3.5-flash litellm_params: model: vertex_ai/gemini-3.5-flash vertex_project: os.environ/VERTEXAI_PROJECT vertex_location: global - model_name: gemini-3.1-pro-preview-customtools litellm_params: model: vertex_ai/gemini-3.1-pro-preview-customtools vertex_project: os.environ/VERTEXAI_PROJECT vertex_location: global # preview model — global endpoint only ``` Authenticate the gateway with Workload Identity (GKE) or a mounted ADC JSON (`GOOGLE_APPLICATION_CREDENTIALS`). `gemini-3.5-flash` is the recommended fast tier. ```yaml theme={null} - model_name: claude-opus-4-8 litellm_params: model: vertex_ai/claude-opus-4-8 vertex_project: os.environ/VERTEXAI_PROJECT vertex_location: global ``` Anthropic models served by Vertex AI Model Garden. Stays within the Vertex tenancy boundary — no separate Anthropic key required. ```yaml theme={null} - model_name: gpt-5.5 litellm_params: model: openai/gpt-5.5 api_key: os.environ/OPENAI_API_KEY ``` Mount `OPENAI_API_KEY` from a Kubernetes Secret into the **gateway Deployment only**. ```yaml theme={null} - model_name: claude-opus-4-8 litellm_params: model: anthropic/claude-opus-4-8 api_key: os.environ/ANTHROPIC_API_KEY ``` Mount `ANTHROPIC_API_KEY` from a Kubernetes Secret into the **gateway Deployment only**. ```yaml theme={null} - model_name: gemini-3.5-flash litellm_params: model: gemini/gemini-3.5-flash api_key: os.environ/GEMINI_API_KEY ``` Prefer the Vertex AI entry above for production — it inherits the tenant's IAM boundary rather than a long-lived API key. ```yaml theme={null} - model_name: llama3.1 litellm_params: model: ollama/llama3.1 api_base: http://ollama.ollama.svc.cluster.local:11434 ``` No API key required. The Ollama service must be reachable from the gateway namespace. ```yaml theme={null} - model_name: llama3.1-70b litellm_params: model: openai/llama3.1-70b api_base: http://vllm.vllm.svc.cluster.local:8000/v1 api_key: os.environ/VLLM_API_KEY ``` vLLM exposes an OpenAI-compatible API — use the `openai/` prefix. ## Deploying the gateway The LLM Gateway is a Kubernetes Deployment behind a ClusterIP Service. It is **not** a sidecar in each agent pod — it is a shared platform service maintained by the platform team. ```yaml theme={null} # Minimal production Deployment (add HPA + PodDisruptionBudget for HA) apiVersion: apps/v1 kind: Deployment metadata: name: llm-gateway namespace: craft spec: replicas: 2 # scale horizontally for throughput; add Redis for shared state selector: matchLabels: app: llm-gateway template: spec: containers: - name: litellm image: ghcr.io/berriai/litellm:main-latest ports: - containerPort: 4000 env: - name: LITELLM_MASTER_KEY valueFrom: secretKeyRef: name: llm-gateway-secrets key: master-key - name: VERTEXAI_PROJECT valueFrom: secretKeyRef: name: llm-gateway-secrets key: vertex-project volumeMounts: - name: config mountPath: /app/config.yaml subPath: config.yaml volumes: - name: config configMap: name: llm-gateway-config --- apiVersion: v1 kind: Service metadata: name: llm-gateway namespace: craft spec: selector: app: llm-gateway ports: - port: 80 targetPort: 4000 type: ClusterIP ``` Agents in any namespace reference the gateway as `http://llm-gateway.craft.svc.cluster.local/v1`. ## Disaster recovery — gateway unavailable Because the gateway is shared, a gateway outage affects all agents cluster-wide, not a single pod. **Symptoms:** agents receive `Connection refused` or `503 Service Unavailable` on their gateway URL. **Response:** 1. Check gateway pod health: `kubectl get pods -n craft -l app=llm-gateway` 2. Check gateway logs: `kubectl logs -n craft deployment/llm-gateway --tail=50` 3. Check provider upstream status (Vertex, OpenAI, Anthropic status pages) 4. If gateway pods are crash-looping: `kubectl rollout restart deployment/llm-gateway -n craft` 5. If a model alias is hitting rate limits, the fallback entry in `model_list` activates automatically To verify the gateway is healthy: ```bash theme={null} curl http://llm-gateway.craft.svc.cluster.local/health | jq . # Expected: {"status": "healthy"} ``` ## Next steps How to call the gateway from solution code using litellm. Platform-side observability: Langfuse traces, cost dashboards, model comparison. How provider keys and gateway keys flow through the secrets pipeline. How the gateway fits into the overall platform architecture. Connect Claude Code, Cursor, Goose, or an external agent to CRAFT's tool gateway over MCP. # MCP Server Source: https://docs.emergence.ai/platform/mcp-server Connect AI clients and external agents to CRAFT's tool gateway over the Model Context Protocol — IDE integrations, headless agent access, and tool reference # MCP Server CRAFT handles the **"what"** — governed, accurate data retrieval from your enterprise systems — so your AI tools and agents can focus on the **"why"**: validating hypotheses, identifying root causes, running what-if scenarios, and surfacing anomalies. Use `resolve_term` so your agent knows what "active customer" means in *your* warehouse before generating SQL; use `execute_query` to validate results; use `generate_plotly_chart` to surface patterns instantly. Connect Claude Code, Cursor, Goose, or a purpose-built agent to `https://craft.emergence.ai/mcp` and CRAFT becomes a governed data layer for any reasoning workflow. Two connection patterns: **(a) interactive** — IDE clients query CRAFT while you code; **(b) programmatic** — embed CRAFT as a tool server in an agent pipeline for hypothesis testing, root cause analysis, anomaly detection, or what-if modeling. ## Server details | Property | Value | | --------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | Endpoint | `https://craft.emergence.ai/mcp` | | Transport | Streamable HTTP ([MCP 2025-11-25 spec](https://modelcontextprotocol.io/specification/2025-11-25/basic/transports); SSE-compat path available) | | Auth | OAuth 2.0 PKCE (authorization code + S256, via Keycloak) | | Health | `GET https://craft.emergence.ai/health` → `{"status": "ok"}` | ## Verify the endpoint (no auth needed) Before setting up OAuth, confirm the server is reachable with the no-auth `hello_world` tool: ```bash theme={null} curl -s -X POST https://craft.emergence.ai/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"hello_world","arguments":{}}}' \ | jq . ``` Expected response: ```json theme={null} {"jsonrpc": "2.0", "id": 1, "result": {"content": [{"type": "text", "text": "Hello from CRAFT MCP server"}]}} ``` If this succeeds, network connectivity and DNS are confirmed. If you prefer a CLI tool: ```bash theme={null} npx @mcpjam/cli server doctor --url https://craft.emergence.ai/mcp ``` ## Authentication ### Interactive clients (OAuth) Supported IDE clients use the [MCP OAuth 2.0 PKCE authorization flow](https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization). On first connection, the client opens a browser window to `craft.emergence.ai`, where you sign in with your **Google** or **Microsoft** account. The access token is stored and refreshed by the client automatically — no manual token handling required. ### X-Project-ID header All tools except `hello_world` require an `X-Project-ID` header scoping the request to your CRAFT project. Find your project ID in the CRAFT console under **Settings → Project → ID**. `X-Project-ID` selects a project scope but does **not** grant access — project membership is enforced server-side against your JWT's claims. Passing an arbitrary project ID returns an authorization error if you are not a member. ### Token requirements for headless callers Programmatic callers (agents, pipelines) must obtain an access token before calling tools: | Requirement | Value | | --------------- | ------------------------------------------------------------------------ | | Issuer (`iss`) | `https://craft.emergence.ai/keycloak/realms/emergence` | | Required scopes | `openid profile email` | | Grant type | Authorization code (interactive) or Client credentials (service account) | | Token TTL | \~5 minutes; refresh token manages rotation for interactive clients | For service accounts and CI pipelines, see [Service Accounts](/platform/service-accounts) and [Authentication](/platform/authentication). ## Connect your AI client Add the CRAFT MCP server from the CLI: ```bash theme={null} claude mcp add --transport http craft https://craft.emergence.ai/mcp \ --header "X-Project-ID: " ``` Or add it to `.mcp.json` in your project root (or `~/.claude.json` for user-scope): ```json theme={null} { "mcpServers": { "craft": { "type": "http", "url": "https://craft.emergence.ai/mcp", "headers": { "X-Project-ID": "" } } } } ``` The OAuth browser flow triggers automatically on first tool call. Token refresh is handled by Claude Code. Find your project ID in the CRAFT console under **Settings → Project → ID**. Add to `.cursor/mcp.json` in your project (or `~/.cursor/mcp.json` globally): ```json theme={null} { "mcpServers": { "craft": { "type": "http", "url": "https://craft.emergence.ai/mcp", "headers": { "X-Project-ID": "${env:CRAFT_PROJECT_ID}" } } } } ``` Set `CRAFT_PROJECT_ID` as a shell environment variable — Cursor expands `${env:VAR}` at runtime. The OAuth browser window opens on first use. Create `.vscode/mcp.json` in your project: ```json theme={null} { "inputs": [ { "type": "promptString", "id": "craft-project-id", "description": "Your CRAFT project ID" } ], "servers": { "craft": { "type": "http", "url": "https://craft.emergence.ai/mcp", "headers": { "X-Project-ID": "${input:craft-project-id}" } } } } ``` VS Code prompts for the project ID once and stores it securely. OAuth handles the access token automatically. If tool calls fail silently, headers may not be forwarded — a known issue in some VS Code versions ([#264095](https://github.com/microsoft/vscode/issues/264095)). Workaround: replace the `type`/`url` block with `mcp-remote`: ```json theme={null} { "servers": { "craft": { "command": "npx", "args": [ "mcp-remote@latest", "https://craft.emergence.ai/mcp", "--header", "X-Project-ID: " ] } } } ``` Note: `mcp-remote` handles the OAuth flow locally rather than delegating it to the server. Add to `~/.config/goose/config.yaml`: ```yaml theme={null} extensions: craft: name: craft type: http url: https://craft.emergence.ai/mcp enabled: true headers: X-Project-ID: "${CRAFT_PROJECT_ID}" ``` Set `CRAFT_PROJECT_ID` in your environment. Alternatively, run `goose configure`, select **Remote Extension (Streamable HTTP)**, and the interactive wizard will prompt you for the URL and any custom headers. OAuth triggers on first connection. ## Interactive charts — MCP Apps extension `generate_plotly_chart` returns both a raw `plotly_json` response (all clients) and an embedded interactive UI resource for clients that support the [MCP Apps protocol](https://modelcontextprotocol.io/extensions/apps/overview). The embedded resource is a sandboxed iframe containing the full Plotly chart with a modebar — including an **SVG download** button — rendered inline alongside the chat without switching tabs. The interactive iframe is enabled by default (`MCP_APPS_ENABLED=true`). No client-side configuration is required; supported clients detect and render it automatically. Clients that support MCP Apps interactive chart rendering (verified 2026-05-28): | Client | MCP Apps | Notes | | ---------------------------- | :------: | --------------------------------------------------------- | | Cursor 2.6+ | ✅ | Desktop IDE; renders interactive Plotly iframe inline | | VS Code Copilot Chat v1.109+ | ✅ | Browser-rendered panel with modebar | | Goose | ✅ | Desktop app; renders iframes inline | | Postman | ✅ | Renders in Postman canvas; good for dev/testing | | MCPJam | ✅ | Inspector UI; best for debugging chart output | | Claude Code (CLI) | ❌ | Terminal — no browser context; receives raw `plotly_json` | Claude Code and other non-Apps clients receive the raw `plotly_json` response. Render it locally with `plotly.io.show(fig)` in Python or `Plotly.newPlot(...)` in JavaScript. To disable the embedded resource globally, set `MCP_APPS_ENABLED=false` on the Talk2Data service. ## Developer tools Use these tools to test and debug CRAFT's MCP server during development, especially the MCP Apps / Plotly rendering path. [MCPJam](https://app.mcpjam.com) is a dedicated MCP inspector — web app (no install), desktop app, or CLI. It supports the MCP Apps extension and renders interactive Plotly charts inline. **Web app or desktop**: Add a server with URL `https://craft.emergence.ai/mcp`, enable OAuth, and set `X-Project-ID` in the headers field: ```json theme={null} { "name": "craft", "type": "http", "url": "https://craft.emergence.ai/mcp", "headers": { "X-Project-ID": "" }, "useOAuth": true } ``` **CLI**: Run a health check and OAuth flow validation: ```bash theme={null} npx @mcpjam/cli server doctor \ --url https://craft.emergence.ai/mcp \ --header "X-Project-ID: " ``` MCPJam provides full JSON-RPC trace visibility and guided OAuth conformance checks (DCR, PKCE, CIMD) — useful for validating the MCP server before shipping tool changes. Postman can act as an MCP host (client), connecting to external MCP servers — distinct from Postman's own MCP server for managing Postman resources. Go to **Settings → MCP Hosts → Add** and configure: * **URL**: `https://craft.emergence.ai/mcp` * **Auth**: OAuth (recommended) or access token * **Headers**: `X-Project-ID: ` Postman supports the MCP Apps extension — Plotly charts appear inline in the Postman canvas. Once you have a working configuration, use Postman's **Export MCP config** button to generate a ready-to-paste config for Claude Code, Cursor, or VS Code. ## Calling CRAFT from an external agent Any MCP-compatible agent framework can use CRAFT as a tool server, bringing governed data access to any reasoning pipeline. ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR EA["External agent\n(any framework)"] GW["gateway-external\n(public HTTPS)"] MCP["em-runtime-mcp\n(tool gateway)"] ALLOW["Allowlist check\n+ audit event"] CATALOG["Catalog tools\nlist · get · search"] T2D["Talk2Data tools\nSQL · query · chart"] EA -->|"Streamable HTTP\nOAuth + X-Project-ID"| GW GW --> MCP MCP --> ALLOW ALLOW --> CATALOG ALLOW --> T2D classDef entry fill:#56B4E9,stroke:#555555,color:#000 classDef infra fill:#0072B2,stroke:#555555,color:#fff classDef decision fill:#F0E442,stroke:#555555,color:#000 classDef action fill:#E69F00,stroke:#555555,color:#000 class EA entry class GW,MCP infra class ALLOW decision class CATALOG,T2D action ``` Requires `pydantic-ai-slim[fastmcp]` and `fastmcp`: ```python theme={null} from fastmcp import Client from fastmcp.client.transports import StreamableHttpTransport from pydantic_ai import Agent from pydantic_ai.toolsets.fastmcp import FastMCPToolset transport = StreamableHttpTransport( url="https://craft.emergence.ai/mcp", headers={ "Authorization": "Bearer ", "X-Project-ID": "", }, ) toolset = FastMCPToolset(Client(transport)) agent = Agent("anthropic:claude-opus-4-8", toolsets=[toolset]) async def main(): result = await agent.run( "What databases are available? Generate SQL to count active customers." ) print(result.output) ``` `MCPServerHTTP` is deprecated — use `FastMCPToolset` with `StreamableHttpTransport`. Requires `google-adk`: ```python theme={null} from google.adk.agents import LlmAgent from google.adk.tools.mcp_tool import MCPToolset, StreamableHTTPConnectionParams agent = LlmAgent( model="gemini-3.5-flash", tools=[ MCPToolset( connection_params=StreamableHTTPConnectionParams( url="https://craft.emergence.ai/mcp", headers={ "Authorization": "Bearer ", "X-Project-ID": "", }, ) ) ], ) ``` `SseServerParams` is the legacy SSE transport. Use `StreamableHTTPConnectionParams` for the MCP 2025-11-25 streamable-HTTP transport. Any HTTP client can speak the MCP JSON-RPC protocol. List available tools: ```bash theme={null} # 1. Initialize a session curl -s -X POST https://craft.emergence.ai/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "Authorization: Bearer " \ -H "X-Project-ID: " \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"my-agent","version":"1.0"}}}' \ | jq . ``` ```bash theme={null} # 2. List tools (include MCP-Session-Id from initialize response) curl -s -X POST https://craft.emergence.ai/mcp \ -H "Content-Type: application/json" \ -H "Authorization: Bearer " \ -H "X-Project-ID: " \ -H "MCP-Session-Id: " \ -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' \ | jq '.result.tools[].name' ``` ```bash theme={null} # 3. Call a tool curl -s -X POST https://craft.emergence.ai/mcp \ -H "Content-Type: application/json" \ -H "Authorization: Bearer " \ -H "X-Project-ID: " \ -H "MCP-Session-Id: " \ -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"search_schema","arguments":{"query":"customer"}}}' \ | jq . ``` Include `MCP-Session-Id` on all requests after initialization for stateful sessions. Headless callers need a programmatic access token. Obtain one via a Keycloak service account (client credentials grant). See [Service Accounts](/platform/service-accounts) for setup. ## Available tools ### Catalog tools Always available — provide governed access to your project's data catalog. | Tool | Required parameters | Returns | | ---------------- | -------------------------------------------------------------------- | --------------------------------------------------------- | | `hello_world` | — | Server greeting; confirms connectivity (no auth required) | | `list_databases` | `resource_uri` — data connection UUID | Database metadata list | | `get_schema` | `resource_uri`, `fqn` — 2 segments = database, 3 = schema, 4 = table | Full entity schema including columns and types | | `search_schema` | `query` — Elasticsearch-style text query | Matching metadata entries across the project | ### Talk2Data tools Available when the deployment includes the Data Insights solution. See [Data Connections](/platform/data-connections) to connect your data sources. | Tool | Required parameters | Returns | | ----------------------- | ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | `generate_sql` | `question` (natural language), `database` (connection UUID), `schema` (object with `schema_name`) | `sql`, `explanation`, `assumptions` | | `execute_query` | `sql` (SELECT statements only), `database` | `columns`, `rows`, `row_count`, `truncated` | | `resolve_term` | `term` — business term (e.g., "ARR", "churn rate") | `definition`, `related_columns`, `sql_pattern` | | `generate_plotly_chart` | `data` (tabular), `chart_type` (bar, line, scatter, pie, …) | Plotly JSON figure spec (`plotly_json`) + interactive iframe for MCP Apps clients | | `sample_data` | `resource_uri`, `table_fqn` — `database.schema.table` | `columns`, `rows` (up to 100 rows preview) | `execute_query` only accepts SELECT statements — DDL, DML, and admin functions are rejected. `generate_sql` delegates to the Text2SQL A2A agent; the first call in a session may take 3–8 s (cold start), subsequent calls \~1–2 s. `generate_plotly_chart` emits per-stage MCP progress notifications (parse → validate → render → verify) forwarded as A2A status events — visible in clients that display progress. ## Per-project allowlist and governance CRAFT enforces a per-project tool allowlist — each project defines which tools each agent identity may call. This is the key difference between the CRAFT MCP gateway and a raw FastMCP server. Configure allowlists in [Platform → Authorization](/platform/authorization) (RBAC model) and [Platform → Agents](/platform/agents) (agent identity → allowlist mapping). Denied calls return a structured error: ```json theme={null} { "ok": false, "error": { "code": "tool_not_allowed", "message": "Tool 'generate_sql' is not in the allowlist for this agent identity" } } ``` Every tool call — allowed or denied — writes an audit event to the governance event log. Each event records: * Timestamp, agent identity, tool name, project ID * Input parameters (PII-scrubbed per governance policy) * Result status (allowed / denied / error) Query audit events via [Audit Log](/guides/customer-admin/audit-log). Tool calls are tracked separately from LLM completions and do not flow through the LiteLLM gateway. See [LLM Gateway](/platform/llm-gateway) for the complementary table. ## Errors and limits | Condition | HTTP | MCP error code | Remediation | | --------------------------------- | ---- | ----------------------- | ------------------------------------------------------------------------------------------------------------ | | Missing `Authorization` header | 401 | `missing_authorization` | Complete the OAuth flow; verify token is fresh | | Missing `X-Project-ID` header | 400 | `missing_project` | Add `X-Project-ID` header with your project UUID | | Tool not in allowlist | 403 | `tool_not_allowed` | Ask project admin to add tool to allowlist | | Non-SELECT SQL in `execute_query` | 422 | `validation_error` | Only SELECT statements are permitted | | Text2SQL agent unavailable | 503 | `upstream_unavailable` | Retry; `generate_sql` depends on the Talk2Data A2A agent | | Query row limit hit | 200 | — | Response includes `"truncated": true`; add a `LIMIT` or more selective `WHERE` clause to fit within 500 rows | **Limits:** * `execute_query`: server-enforced cap of 500 rows; `truncated: true` in response when exceeded * `sample_data`: up to 100 rows (`limit` parameter; default 10) * `generate_sql`: first call may take 3–8 s (Text2SQL agent cold-start); subsequent calls \~1–2 s ## Next steps The complementary gateway for LLM completions — model allowlist, rate limits, and cost attribution. Connect your data sources to enable Talk2Data tools. Track new tools, breaking changes, and platform updates. # Context Packs Source: https://docs.emergence.ai/platform/memory-context-packs Context Packs are the storage containers for agent memories, organizing related memories into retrievable, structured units that agents query for relevant context. # Context Packs A **Context Pack** is the fundamental storage unit for agent memories. It groups related memories into a named, typed container that agents query when they need relevant background knowledge before responding or acting. ## Three-Layer Architecture Context Packs use a three-layer structure that balances retrieval speed with information depth: ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TB Query["Agent Query"] --> S["Summary Layer
(fast, always loaded)"] S --> M["Metadata Layer
(selective, on match)"] M --> C["Content Layer
(full detail, on demand)"] classDef query fill:#56B4E9,stroke:#555555,color:#000 classDef layer fill:#E69F00,stroke:#555555,color:#000 class Query query class S,M,C layer ``` | Layer | Contents | When Loaded | | ------------ | ------------------------------------------------------------------- | ------------------------------------ | | **Summary** | High-level description, topic tags, last-updated timestamp | Always, fast pack discovery | | **Metadata** | Memory count, type distribution, confidence scores, access patterns | On pack selection | | **Content** | Full memory records with embeddings, provenance, relationships | On demand, specific memory retrieval | ## Two-Phase Query Model Agents retrieve context through a two-phase process that avoids loading unnecessary detail: Query Context Pack summaries to find relevant packs. Returns pack IDs, topic descriptions, and match scores. Fast, only summary layer is read. For matched packs, retrieve specific memories using semantic search within the pack's content layer. Returns ranked memory records with scores. ## Context Pack Types | Type | Purpose | Decay rate | | ---------------- | ------------------------------------------------------------- | -------------------------- | | **session** | Single conversation context, ephemeral, high detail | Aggressive (hours to days) | | **user** | Long-term user preferences and history, persistent | Conservative (months) | | **domain** | Domain knowledge, facts, rules, entity relationships | Very conservative (years) | | **project** | Project-specific context, decisions, conventions, constraints | Conservative (months) | | **team** | Shared team knowledge and conventions | Conservative (months) | | **data\_source** | Data-source-specific schema, joins, and profiling context | Conservative (months) | ## Memory Record Schema Each memory stored in a Context Pack has the following structure: ```json theme={null} { "id": "uuid", "content": "The main memory text (self-contained, context-independent)", "memory_type": "fact | experience | observation | instruction | preference | summary | glossary | ontology | textual_pattern | kpi | exemplar | join | numeric_pattern | policy", "name": "optional_label_for_direct_retrieval", "metadata": { "source": "conversation | document | agent_message", "confidence": 0.92, "created_at": "2026-04-07T10:30:00Z", "last_accessed": "2026-04-07T14:22:00Z", "access_count": 7, "superseded_by": null, "provenance": "Session #142, turn 3" }, "embedding": [0.123, -0.456, ...], "relationships": [ {"type": "supersedes", "target_id": "uuid-of-old-memory"}, {"type": "related_to", "target_id": "uuid-of-related-memory"} ], "degradation_tier": "full | summary | key_facts | existence_marker | archived" } ``` ## Managing Context Packs via API ### List Packs ```python theme={null} import httpx response = httpx.get( "https://api.example.com/utils/context-packs", headers={"Authorization": f"Bearer {token}", "X-Project-ID": project_id}, params={"pack_type": "domain"} ) packs = response.json()["data"] ``` ### Create a Pack ```python theme={null} response = httpx.post( "https://api.example.com/utils/context-packs", headers={"Authorization": f"Bearer {token}", "X-Project-ID": project_id}, json={ "name": "semiconductor_domain", "pack_type": "domain", "description": "Knowledge about semiconductor fabrication processes and terminology", "meta_data": {"tags": ["semiconductor", "fabrication", "yield"]} } ) ``` ### Query a Pack (Phase 2 retrieval) ```python theme={null} response = httpx.get( f"https://api.example.com/utils/context-packs/{pack_name}/memories", headers={"Authorization": f"Bearer {token}", "X-Project-ID": project_id}, params={ "memory_type": "fact", "page_size": 5 } ) memories = response.json()["data"] ``` ## Degradation Tiers Memories in Context Packs degrade gracefully over time when not accessed: | Tier | Description | Storage impact | | --------------------- | -------------------------------------- | -------------- | | **full** | Complete memory record with all detail | Full | | **summary** | Key points only, detail compressed | \~30% | | **key\_facts** | Core facts only, context removed | \~10% | | **existence\_marker** | Memory existed, no content | Minimal | | **archived** | Moved to cold storage, not retrieved | Off-heap | Any retrieval access re-promotes a memory to the **full** tier. This ensures frequently-used memories remain fully detailed while inactive memories are progressively compressed. ## Related Overview of the memory system and API usage. Step-by-step guide to adding memory to your agents. # Memory Service Source: https://docs.emergence.ai/platform/memory-service Multi-agent memory management for CRAFT, how agents remember and learn from interactions using Context Packs. # Memory Service The Memory Service provides multi-agent memory management for CRAFT. It gives agents the ability to remember information across conversations, accumulate domain knowledge, and personalize responses based on prior interactions. Memory is built into the **Utils service** (`em-runtime-utils`). No separate service is required — the same Utils API that handles schedules, metadata, and catalog tags also manages Context Packs and Memories. ## Key Concepts ### Memories A **memory** is a discrete, self-contained piece of information extracted from conversations, documents, or agent messages. Each memory has: * **Content**: The information itself (a fact, preference, insight, or relationship) * **Type**: The category of memory (`fact`, `preference`, `observation`, `glossary`, `policy`, etc., see [Memory Types](#memory-types) below) * **Name**: An optional label for direct retrieval * **Metadata**: Source, timestamp, confidence, access count * **Embeddings**: Vector representation for semantic search ### Context Packs A **Context Pack** is the storage container for a related set of memories. Context Packs organize memories by domain or purpose and provide efficient retrieval. See [Context Packs](/platform/memory-context-packs) for the detailed architecture. ### Memory Lifecycle ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% stateDiagram-v2 [*] --> Active : extracted + worthiness score passes Active --> Summarized : inactivity decay Summarized --> KeyFacts : further decay KeyFacts --> ExistenceMarker : further decay ExistenceMarker --> Archived : maximum decay Archived --> [*] : purged after retention period Summarized --> Active : retrieved (re-promotion) KeyFacts --> Active : retrieved (re-promotion) ExistenceMarker --> Active : retrieved (re-promotion) Archived --> Active : explicitly re-promoted ``` Memories move through a managed lifecycle: 1. **Created**, extracted from input and scored for worthiness 2. **Active**, retrievable and contributing to agent context 3. **Degraded**, fading through summarization tiers (full → key facts → existence marker) 4. **Archived**, removed from active retrieval but preserved for audit 5. **Re-promoted**, any retrieval restores a degraded memory to active state ## Memory Types Memories are categorized into two groups: ### General Types | Type | Purpose | | ------------- | --------------------------------------------------------- | | `fact` | Factual information about entities, processes, or systems | | `experience` | Learned behaviors from past interactions | | `observation` | Observed patterns or anomalies | | `instruction` | Directives or guidelines for agent behavior | | `preference` | User or agent preferences (e.g., preferred output format) | | `summary` | Condensed summaries of conversations or data | ### Data Governance Enrichment Types | Type | Purpose | | ----------------- | ------------------------------------------------- | | `glossary` | Business glossary terms and definitions | | `ontology` | Domain ontology relationships and classifications | | `textual_pattern` | Regex or text patterns identified in data columns | | `kpi` | Key performance indicator definitions | | `exemplar` | Representative data examples for reference | | `join` | Table join relationships and foreign key mappings | | `numeric_pattern` | Numeric distribution patterns in data columns | | `policy` | Data governance policies and rules | ## Using the Memory API All memory operations go through the Utils service (`/api/utils/`). Authentication uses the same JWT tokens as all other platform APIs, plus an `X-Project-ID` header to scope results to the current project. ### Add a Memory to a Context Pack Memories are created within a named Context Pack: ```python theme={null} import httpx response = httpx.post( f"https://api.example.com/utils/context-packs/{pack_name}/memories", headers={ "Authorization": f"Bearer {token}", "X-Project-ID": project_id, }, json={ "name": "sql_explanation_style", "content": { "type": "text", "text": "The user prefers detailed SQL explanations with comments", }, "memory_type": "preference", "source": {"type": "conversation"}, }, ) memory = response.json() print(memory["id"]) # UUID of the created memory ``` ### List Memories ```python theme={null} # All memories in the project (across all packs) response = httpx.get( "https://api.example.com/utils/memories", headers={"Authorization": f"Bearer {token}", "X-Project-ID": project_id}, ) all_memories = response.json() # {"data": [...], "pagination": {...}} # Memories within a specific Context Pack response = httpx.get( f"https://api.example.com/utils/context-packs/{pack_name}/memories", headers={"Authorization": f"Bearer {token}", "X-Project-ID": project_id}, ) pack_memories = response.json() ``` ### Retrieve by ID ```python theme={null} response = httpx.get( f"https://api.example.com/utils/memories/{memory_id}", headers={"Authorization": f"Bearer {token}", "X-Project-ID": project_id}, ) memory = response.json() ``` ### Retrieve by Name within a Pack ```python theme={null} response = httpx.get( f"https://api.example.com/utils/context-packs/{pack_name}/memories/{memory_name}", headers={"Authorization": f"Bearer {token}", "X-Project-ID": project_id}, ) memory = response.json() ``` ### Update a Memory ```python theme={null} response = httpx.patch( f"https://api.example.com/utils/memories/{memory_id}", headers={"Authorization": f"Bearer {token}", "X-Project-ID": project_id}, json={ "content": { "type": "text", "text": "The user prefers concise SQL explanations with minimal comments", }, }, ) ``` ### Delete a Memory ```python theme={null} # Soft delete (moves to deleted state, recoverable) response = httpx.delete( f"https://api.example.com/utils/memories/{memory_id}", headers={"Authorization": f"Bearer {token}", "X-Project-ID": project_id}, ) # Returns 200 OK with a MemoryDeleteResponse body confirming the soft-delete. # Permanent delete (irreversible) response = httpx.post( f"https://api.example.com/utils/memories/{memory_id}/permanent-delete", headers={"Authorization": f"Bearer {token}", "X-Project-ID": project_id}, ) ``` ### Lifecycle Operations ```python theme={null} # Pin a memory — keeps it in the Active tier, bypasses inactivity decay httpx.post( f"https://api.example.com/utils/memories/{memory_id}/pin", headers={"Authorization": f"Bearer {token}", "X-Project-ID": project_id}, ) # Archive a memory — moves it to Archived state httpx.post( f"https://api.example.com/utils/memories/{memory_id}/archive", headers={"Authorization": f"Bearer {token}", "X-Project-ID": project_id}, ) # Supersede — mark old_memory_id as superseded by new_memory_id httpx.post( f"https://api.example.com/utils/memories/{old_memory_id}/supersede/{new_memory_id}", headers={"Authorization": f"Bearer {token}", "X-Project-ID": project_id}, ) ``` ## Scoping Memories are scoped per organization and project: * **Organization**: determined automatically from the JWT's `org_id` claim — no separate header needed * **Project**: pass the `X-Project-ID` header on every request ```http theme={null} Authorization: Bearer X-Project-ID: ``` ## Related Deep dive into the Context Pack storage architecture. How-to guide for integrating memory into your agents. Full memory endpoint API reference. # Multi-Tenancy Source: https://docs.emergence.ai/platform/multi-tenancy Tenant isolation architecture with Keycloak realm separation, org_id scoping, dual authorization, and per-service database partitioning. # Multi-Tenancy CRAFT implements multi-tenancy at every layer of the stack. Each organization is fully isolated through a combination of **identity separation** (Keycloak realms), **authorization scoping** (OpenFGA relationship tuples), and **data partitioning** (org\_id filtering on every query). This page describes the architecture that makes this work. ## Isolation Layers ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TB REQ["Inbound Request\n(user or service account)"] KC["Keycloak Realm\n(identity isolation)"] JWT["JWT Validation\n(org_id extracted)"] OF["OpenFGA\n(authorization tuples)"] FILT["org_id Filter\n(data isolation)"] DB[("Per-Service DB\n(service isolation)")] REQ --> KC KC -->|"issues JWT\nwith org_id claim"| JWT JWT --> OF JWT --> FILT OF -->|"permission check\n(resource-scoped)"| FILT FILT --> DB classDef entry fill:#56B4E9,stroke:#555555,color:#000 classDef identity fill:#56B4E9,stroke:#555555,color:#000 classDef authz fill:#E69F00,stroke:#555555,color:#000 classDef data fill:#009E73,stroke:#555555,color:#000 classDef infra fill:#0072B2,stroke:#555555,color:#fff class REQ entry class KC,JWT identity class OF authz class FILT data class DB infra ``` The platform enforces tenant isolation through four complementary mechanisms: Each organization is a separate Keycloak realm. Users, groups, sessions, and identity provider configurations are completely isolated between realms. A user in `acme-corp` cannot see or interact with users in `globex`. OpenFGA relationship tuples are scoped to specific resources. Permission checks always include the resource ID, ensuring a user in one organization cannot access resources in another even if they somehow obtain a valid token. Every database query includes an `organization_id` filter derived from the JWT token. This is enforced at the application layer -- queries without `org_id` filters are treated as security vulnerabilities. Each platform service (Governance, Assets, Utils) owns its own PostgreSQL database. There are no cross-service foreign keys, preventing accidental data leakage through join queries. ## How org\_id Flows Through the System The organization ID is the foundation of tenant isolation. It flows through every request: The user authenticates against their organization's Keycloak realm. The JWT token's `iss` (issuer) claim contains the realm name, which becomes the `org_id`. ```text theme={null} iss: "https://keycloak.example.com/realms/acme-corp" ^^^^^^^^^ org_id ``` The Governance service validates the JWT using the realm's JWKS public keys. The `org_id` is extracted from the verified issuer claim and stored in the `Auth` object. Users in the `master` realm (platform developers) have `org_id = null`. Every API endpoint receives the `Auth` object via dependency injection. The `org_id` is used to: * **Filter list queries** to return only the organization's resources * **Scope write operations** to associate new resources with the correct organization * **Validate access** via OpenFGA permission checks When Assets or Utils need to check permissions, they forward the user's JWT token to the Governance service via auto-generated SDK calls. The Governance service re-validates the token and checks OpenFGA -- the `org_id` is never passed as a parameter that could be spoofed. ## Dual Authorization Pattern The platform uses a dual authorization pattern that combines database-level filtering with authorization checks: List endpoints apply **both** database filtering and permission checks: ```python theme={null} @router.get("/data") async def list_data_connections( ctx: RequestContextDep, # Permission check: user can read resources in this project _: None = Depends(require_permission( Permissions.PROJECT, Permissions.PROJECT.actions.CAN_READ, HEADER_PROJECT_ID )) ): # Database filter: always scope by org_id from JWT connections = await service.list_data_connections( organization_id=ctx.org_id, # From JWT, never from request body project_id=ctx.project_id, # From X-Project-ID header ) ``` This defense-in-depth approach means that even if a permission check has a bug, the database query will not return another organization's data. Single-resource endpoints rely on **OpenFGA permission checks only**: ```python theme={null} @router.get("/data/{resource_uri}") async def get_data_connection( resource_uri: str, # Permission check: user can read this specific resource _: None = Depends(require_permission( Permissions.DATA_CONNECTION, Permissions.DATA_CONNECTION.actions.CAN_READ, "resource_uri" )) ): # OpenFGA already verified the user has access connection = await service.get_data_connection( organization_id=ctx.org_id, resource_uri=resource_uri, ) ``` Even here, the `organization_id` filter is included as an additional safety measure. **Security rules enforced across all services:** * Never trust user-supplied `org_id` from request body or parameters. Always use `auth.org_id` from the JWT. * Never list resources without an `org_id` filter. Omitting the filter returns all organizations' data. * Always use `auth.org_id` when creating new resources. This ensures the resource is correctly scoped to the authenticated user's organization. ## Database Architecture Each service owns an independent PostgreSQL database with no cross-service foreign keys: ```text theme={null} PostgreSQL Server keycloak_db -- Keycloak identity data openfga_db -- OpenFGA authorization tuples infisical_db -- Infisical secrets data (present by default; unused if using ESO backend) governance_db -- Organizations, projects assets_db -- Artifacts, data connections, files, models utils_db -- Data catalog, scheduling, context packs, memories ``` Every resource table includes an `organization_id` column: | Column | Type | Purpose | | ----------------- | ------------- | -------------------------------------------------- | | `organization_id` | `VARCHAR` | Tenant partition key, always populated from JWT | | `project_id` | `VARCHAR` | Project scope within the organization | | `created_at` | `TIMESTAMPTZ` | Record creation timestamp (via `TimestampMixin`) | | `updated_at` | `TIMESTAMPTZ` | Last modification timestamp (via `TimestampMixin`) | Each service runs its own Alembic migrations independently. There are no cross-service migration dependencies, which means services can be upgraded independently without coordinating database schema changes. ## Keycloak Realm Structure When an organization is created, the Governance service provisions a complete Keycloak realm: ```text theme={null} acme-corp (realm) Groups: org-owners -- Organization owners org-admins -- Organization administrators org-members -- Standard members project-owners -- Project-level owners project-admins -- Project-level administrators project-developers -- Project-level developers project-operators -- Project-level operators project-viewers -- Project-level viewers Mappers: groups-mapper -- Ensures group memberships appear in JWT "groups" claim Clients: em-runtime-ui -- Frontend application client (PKCE) Identity Providers: (configurable per realm: SAML, OIDC, LDAP, social login) ``` ## Service Account Cross-Tenant Access Service accounts (background workers, automated processes) authenticate via the `master` realm and specify the target organization via headers: | Header | Purpose | | ---------------- | ------------------------------------- | | `X-Org-Id` | Target organization for the operation | | `X-On-Behalf-Of` | Original user ID for audit trails | These headers are only trusted when the caller passes all three service account checks: 1. Token issued by `master` realm 2. Client ID starts with `svc-` 3. `serviceAccount` role in `realm_access.roles` Regular user tokens cannot use these headers -- the platform silently ignores them for non-service-account callers. ## Service Startup and Dependencies The multi-tenant infrastructure requires a specific startup order: ```text theme={null} PostgreSQL -> Redis -> Keycloak -> OpenFGA -> [Secrets backend] -> Governance -> Assets/Utils ``` The Governance service must be running before Assets or Utils can validate permissions. If Governance is down, downstream services return 403 for all requests because permission checks fail. `make docker-run` handles the startup order automatically via Docker Compose health checks. For manual local development, start `make run-deps` first, wait for readiness, then start Governance before Assets or Utils. ## Next Steps Learn how organizations are provisioned with Keycloak realms. Understand the multi-realm JWT authentication flow. Explore OpenFGA permission inheritance across tenants. See how projects provide the second level of scoping within tenants. # Organizations Source: https://docs.emergence.ai/platform/organizations Multi-tenant organization management with Keycloak realm isolation, default group provisioning, and OpenFGA permission bootstrapping. # Organizations Organizations are the top-level tenant boundary in CRAFT. Each organization maps 1:1 to a **Keycloak realm**, providing full identity isolation between tenants. When you create an organization, the platform provisions a complete identity and authorization environment automatically. Every resource in the platform is scoped to an organization via `organization_id`. The org ID is never supplied by the client in request bodies -- it is always extracted from the JWT token's issuer claim, which identifies the Keycloak realm. ## How Organizations Work ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% graph TB API["POST /organizations"] subgraph KC["Keycloak"] REALM["New Realm\n(org_id as realm name)"] GRP["Default Groups\norg-owners / org-admins / org-members"] MAP["Groups Mapper\n(claims in JWT)"] end subgraph OF["OpenFGA"] TUPLES["Permission Tuples\nowner / admin / member relations"] end DB[("Governance DB\norg record")] JWT["JWT tokens carry\norg_id from realm issuer"] API --> REALM REALM --> GRP --> MAP GRP --> TUPLES API --> DB MAP --> JWT classDef api fill:#E69F00,stroke:#555555,color:#000 classDef kc fill:#56B4E9,stroke:#555555,color:#000 classDef authz fill:#CC79A7,stroke:#555555,color:#000 classDef db fill:#009E73,stroke:#555555,color:#000 classDef token fill:#F0E442,stroke:#555555,color:#000 class API api class REALM,GRP,MAP kc class TUPLES authz class DB db class JWT token ``` Creating an organization triggers a coordinated setup across three systems: A new Keycloak realm is created using the organization ID as the realm name. This realm holds all users, groups, and identity provider configurations for the tenant. Default groups are created within the realm to establish the role hierarchy: * `org-owners` -- Full organization control * `org-admins` -- Administrative access (user and project management) * `org-members` -- Standard membership with read access A groups mapper is configured so group memberships appear as claims in JWT tokens. Permission tuples are written to OpenFGA mapping each group to its corresponding relation: * `org-owners` -> `owner` relation on the organization * `org-admins` -> `admin` relation on the organization * `org-members` -> `member` relation on the organization This means any user added to `org-admins` in Keycloak automatically inherits `admin` permissions across the platform. An organization record is created in the Governance database with the org ID, name, description, and timestamps. ## Organization ID in JWT The organization ID is derived from the Keycloak realm embedded in the token's `iss` (issuer) claim. When a user authenticates against a realm, the platform extracts the realm name as the `org_id`: ```text theme={null} Token issuer: https://keycloak.example.com/realms/acme-corp ^^^^^^^^^ org_id = "acme-corp" ``` This design means: * The `org_id` is **cryptographically bound** to the token (it comes from the issuer verified by JWKS signature) * Clients cannot forge or override the organization context * Users in the `master` realm (platform developers) have `org_id = null` ## API Reference The Organizations API is part of the Governance service (port 8000). Creates a new organization with Keycloak realm, default groups, and OpenFGA permissions. **Access:** Platform developers only (master realm) ```json Request Body theme={null} { "id": "acme-corp", "name": "Acme Corporation", "description": "Production tenant for Acme Corp", "create_users": true } ``` The `id` must follow Keycloak realm naming rules: alphanumeric characters, hyphens, and underscores only. When `create_users` is `true`, default users are created and added to the appropriate groups for immediate access. ```json Response (201 Created) theme={null} { "id": "acme-corp", "name": "Acme Corporation", "description": "Production tenant for Acme Corp", "created_at": "2026-04-01T12:00:00Z" } ``` Retrieves organization details by ID. **Access:** Users with `can_read` permission on the organization (owners, admins, or members). ```json Response (200 OK) theme={null} { "id": "acme-corp", "name": "Acme Corporation", "description": "Production tenant for Acme Corp", "created_at": "2026-04-01T12:00:00Z", "updated_at": "2026-04-02T08:30:00Z" } ``` Deletes an organization and cascades to all its projects. Removes OpenFGA permission tuples, the Keycloak realm (including all users and groups), and database records. **Access:** Platform developers only (master realm). Idempotent: returns 204 whether the organization was deleted or was already absent. ## Bootstrap Organization During first startup, the Governance service automatically bootstraps a default organization: ```yaml theme={null} bootstrap: organization: id: "emergence" name: "Emergence" description: "Emergence organization" create_admin_user: true ``` When `create_admin_user` is `true` (the default), a user named `emergence` is created with: * **Username:** `emergence` * **Password:** `emergence` (change immediately in production) * **Group:** `org-admins` * **OpenFGA relation:** `admin` on the organization The bootstrap admin user is intended for development and initial setup. Always change the default password in production environments. ## Organization Permissions Organizations define the following computed permissions in the OpenFGA schema: | Permission | Owner | Admin | Member | | --------------------- | :---: | :---: | :----: | | `can_read` | Yes | Yes | Yes | | `can_write` | Yes | -- | -- | | `can_delete` | Yes | -- | -- | | `can_manage_projects` | Yes | Yes | -- | | `can_manage_users` | Yes | Yes | -- | | `can_read_secrets` | Yes | Yes | Yes | | `can_manage_secrets` | Yes | Yes | -- | | `can_read_metadata` | Yes | Yes | Yes | | `can_manage_metadata` | Yes | Yes | -- | ## Next Steps Learn how projects scope resources within organizations. Understand the full tenant isolation architecture. See how Keycloak realms power the authentication flow. Explore the OpenFGA permission model in depth. # CRAFT Platform Overview Source: https://docs.emergence.ai/platform/overview The shared platform layer providing governance, asset management, and utilities with built-in identity, authorization, and secrets management. # Platform Overview The platform layer gives your solutions shared infrastructure for identity, authorization, resource management, and a unified UI shell. It includes three core services (Governance, Assets, Utilities), a shared common library, and auto-generated SDKs. The platform is the foundation that every solution depends on. Solutions never implement their own authentication, authorization, or resource management -- they delegate to the platform via SDKs. ## Authentication Flow Authentication uses **Keycloak** with a multi-realm architecture where each organization is a separate realm: The user authenticates via OIDC/PKCE flow against their organization's Keycloak realm. The browser receives a JWT token. All API requests include the JWT in the `Authorization` header. The `X-Project-ID` header specifies the active project context. The Governance service validates the JWT, extracts `org_id` from the realm, and checks OpenFGA for the requested permission. If authorized, the request proceeds. If not, a 403 response is returned. Assets and Utils delegate all permission checks to Governance via SDK calls. ## Authorization Model Authorization uses **OpenFGA** for Relationship-Based Access Control (ReBAC): | Concept | Description | | ------------------------ | -------------------------------------------------------------------------- | | **Roles** | `owner`, `admin`, `member`, `developer`, `operator`, `viewer` | | **Computed permissions** | `can_read`, `can_write`, `can_delete`, `can_execute`, `can_manage_secrets` | | **Inheritance** | Projects inherit from organizations; resources inherit from projects | | **Schema** | Defined in `openfga-schema.fga` (DSL) and `openfga-schema.json` (runtime) | When modifying the OpenFGA schema, always update both the `.fga` DSL file and regenerate the `.json` runtime file. The Governance service reads only the JSON file at startup. Adding new permissions is safe; renaming or removing is a breaking change. ## Next Steps Learn how organizations provide multi-tenant isolation. Explore the unified registry for A2A agents, MCP servers, and skills. Deep dive into Keycloak multi-realm authentication. Understand OpenFGA and the ReBAC permission model. Build a new solution on top of these services — start with the 30-minute quickstart. # Projects Source: https://docs.emergence.ai/platform/projects Logical resource groupings within organizations, providing project-level scoping, permission inheritance, and the X-Project-ID header convention. # Projects Projects are the second level of resource scoping in CRAFT. Every resource (agents, data connections, artifacts, files) belongs to a project, and every project belongs to an organization. Projects provide: * **Logical separation** of resources within a single organization * **Permission inheritance** from the parent organization via OpenFGA * **Access control** through project-level roles (`owner`, `admin`, `developer`, `operator`, `viewer`) The `project_id` is never embedded in the JWT token. It is supplied via the `X-Project-ID` request header, allowing users to switch project context without re-authenticating. ## How Projects Work When a project is created, the platform performs several coordinated actions: A project record is created in the Governance database, linked to the parent organization by `organization_id`. A relationship tuple is written to OpenFGA linking the project to its organization: `project:{project_id}#organization@organization:{org_id}`. This establishes permission inheritance -- organization-level roles automatically grant access to the project. Default project groups receive appropriate permissions: * `project-owners` -> `owner` relation * `project-admins` -> `admin` relation * `project-developers` -> `developer` relation * `project-operators` -> `operator` relation * `project-viewers` -> `viewer` relation ## The X-Project-ID Header All resource APIs (Assets, Utils) require the `X-Project-ID` header to scope requests: ```bash theme={null} # List data connections in a specific project curl -H "Authorization: Bearer $TOKEN" \ -H "X-Project-ID: my-analytics-project" \ https://api.example.com/assets/data ``` This design choice has important implications: Users often work across multiple projects in the same session. Embedding `project_id` in the JWT would require re-authentication to switch projects. The header-based approach allows instant project switching with the same token. The `require_permission()` dependency reads the `X-Project-ID` header and checks OpenFGA to verify the authenticated user has the required permission on that project. If the user lacks access, a 403 is returned. List queries always filter by both `auth.org_id` (from JWT) and `project_id` (from header). This dual filter ensures data isolation even if a permission check has a gap. ## API Reference The Projects API is part of the Governance service (port 8000). Lists all projects in the user's organization with pagination. Results are filtered to only include projects the user has `can_read` permission on. **Query Parameters:** * `page` (default: 1) -- Page number * `limit` (default: 20, max: 100) -- Items per page ```json Response (200 OK) theme={null} { "data": [ { "id": "analytics-prod", "name": "Analytics Production", "description": "Production analytics project", "organization_id": "acme-corp", "created_at": "2026-04-01T12:00:00Z" } ], "pagination": { "page": 1, "limit": 20, "total": 1, "total_pages": 1 } } ``` Creates a new project within the user's organization (determined from JWT). The project is automatically linked to the organization and default groups receive permissions. **Access:** Users with `can_manage_projects` on the organization (owners or admins). ```json Request Body theme={null} { "name": "Analytics Production", "description": "Production analytics project", "external_id": "analytics-prod", "create_users": false } ``` If `external_id` is omitted, a UUID is generated for the project ID. ```json Response (201 Created) theme={null} { "id": "analytics-prod", "external_id": "analytics-prod", "name": "Analytics Production", "organization_id": "acme-corp", "created_at": "2026-04-01T12:00:00Z" } ``` Retrieves detailed information about a specific project. **Access:** Users with `can_read` permission on the project. ```json Response (200 OK) theme={null} { "name": "Analytics Production", "description": "Production analytics project", "organization_id": "acme-corp", "created_at": "2026-04-01T12:00:00Z", "updated_at": "2026-04-02T08:30:00Z" } ``` ## Project Permissions Projects define a richer role set than organizations, supporting fine-grained access control for resource management: | Permission | Owner | Admin | Developer | Operator | Viewer | | ---------------------- | :---: | :---: | :-------: | :------: | :----: | | `can_read` | Yes | Yes | Yes | Yes | Yes | | `can_write` | Yes | Yes | Yes | Yes | -- | | `can_delete` | Yes | Yes | -- | -- | -- | | `can_create_resources` | Yes | Yes | Yes | -- | -- | | `can_read_secrets` | Yes | Yes | Yes | Yes | -- | | `can_manage_secrets` | Yes | Yes | Yes | -- | -- | | `can_read_metadata` | Yes | Yes | Yes | Yes | -- | | `can_manage_metadata` | Yes | Yes | Yes | -- | -- | Project permissions also inherit from the parent organization. An organization `owner` or `admin` automatically has `can_read`, `can_write`, `can_delete`, and `can_create_resources` on all projects in their organization, even without explicit project-level role assignment. ## Service Accounts Projects support **service relations** for background workers and automated processes: | Service Relation | Grants | | ------------------ | ------------------------------------------------------------------------------------------------------------- | | `service_reader` | `can_read`, `can_read_secrets`, `can_read_metadata` | | `service_writer` | `can_read`, `can_write`, `can_read_secrets`, `can_manage_secrets`, `can_read_metadata`, `can_manage_metadata` | | `service_deleter` | `can_read`, `can_delete` | | `service_executor` | `can_execute` (for MCP servers, API servers, data connections) | Service accounts authenticate via the master realm with a `svc-` client ID prefix and are granted contextual tuples scoped to specific projects. See [Service Accounts](/platform/service-accounts) for setup instructions and token management. ## Next Steps Understand how organizations provide the top-level tenant boundary. Explore the OpenFGA permission model and permission inheritance. Manage database and storage connections within projects. Register and discover agents scoped to projects. # Schedules Source: https://docs.emergence.ai/platform/schedules Scheduled task execution with cron patterns, timezone support, one-off scheduling, pg_cron integration, and run history tracking. # Schedules The scheduling system enables time-based automation on CRAFT. Schedules are managed by the **Utils** service (port 8000) and support both recurring cron-based schedules and one-off timed executions. When a schedule triggers, it publishes a notification to a configured topic, which downstream systems (such as workflow orchestrators) can consume. Schedules use **pg\_cron** for reliable execution. Cron jobs run inside PostgreSQL, ensuring schedule triggers survive application restarts and are executed exactly by the database scheduler. ## Schedule Types Recurring schedules defined using standard cron expressions with timezone support. ```json theme={null} { "type": "cron", "cron": "0 9 * * 1-5", "timezone": "America/New_York" } ``` The cron expression above triggers at 9:00 AM Eastern Time, Monday through Friday. Internally, the expression is converted to UTC for storage and execution, then converted back to the original timezone for display. **Cron expression format:** ``` ┌───────── minute (0-59) │ ┌─────── hour (0-23) │ │ ┌───── day of month (1-31) │ │ │ ┌─── month (1-12) │ │ │ │ ┌─ day of week (0-7, 0 and 7 are Sunday) │ │ │ │ │ * * * * * ``` Single-execution schedules that trigger at a specific date and time. ```json theme={null} { "type": "one_off", "at": "2026-06-15T14:30:00", "timezone": "Europe/London" } ``` The datetime must be: * **Timezone-naive** (no `Z` or offset suffix) -- the `timezone` field specifies interpretation * **Minute-precision** -- seconds and microseconds must be zero (pg\_cron limitation) ## Notification Configuration When a schedule triggers, it publishes a notification. Currently, the `NOTIFICATION` config type is supported: ```json theme={null} { "type": "NOTIFICATION", "topic": "data-readiness.workflow.run", "extra_data": { "workflow_config_id": "wf-abc123", "environment": "production" } } ``` | Field | Description | | ------------ | ---------------------------------------------------------------------- | | `topic` | Topic string for the notification (consumed by workflow orchestrators) | | `extra_data` | Optional JSON payload included in the notification | ## Schedule Lifecycle Create a schedule with a name, schedule configuration (cron or one-off), a reference ID linking to the target resource, and a notification config. Schedules can be created in an enabled or disabled state. Enabling a schedule registers it with pg\_cron. Disabling unschedules it. Both operations are idempotent. When pg\_cron fires, a schedule run record is created with a snapshot of the schedule configuration at execution time. The notification is published to the configured topic. Query the runs API to view execution history, including status (PENDING, SUCCESS, FAILURE) and failure reasons. ## API Reference All endpoints are scoped to the authenticated user's organization. The `X-Project-ID` header optionally filters schedules by project. Creates a new schedule. **Access:** `can_manage_projects` on the organization. ```json Request Body theme={null} { "schedule_name": "Daily Analytics Refresh", "schedule": { "type": "cron", "cron": "0 6 * * *", "timezone": "America/Chicago" }, "reference_id": "wf-config-analytics-daily", "config": { "type": "NOTIFICATION", "topic": "data-readiness.workflow.run", "extra_data": { "workflow_config_id": "wf-config-analytics-daily" } }, "enabled": true } ``` ```json Response (201 Created) theme={null} { "schedule_id": "sch-a1b2c3d4" } ``` Lists all schedules in the organization. Optionally filter by project via the `X-Project-ID` header. Schedule values are returned in their original timezone for user-friendly display. Retrieves a single schedule by ID. Must belong to the authenticated user's organization. Full update of a schedule. All fields are required (this is not a partial update). Enables a schedule and registers it with pg\_cron. Idempotent if already enabled. Disables a schedule and unschedules it from pg\_cron. Idempotent if already disabled. Permanently deletes a schedule. This action cannot be undone. ### Schedule Runs Lists schedule runs with pagination. **Query Parameters:** * `page`, `limit` -- Pagination (default: page 1, 20 per page) * `order` -- Sort by `created_at`: `desc` (most recent first) or `asc` * `schedule_id` -- Filter runs for a specific schedule ```json Response theme={null} { "data": [ { "id": "run-x1y2z3", "schedule_id": "sch-a1b2c3d4", "organization_id": "acme-corp", "project_id": "analytics-prod", "status": "SUCCESS", "failure_reason": null, "created_at": "2026-04-03T06:00:00Z" } ], "pagination": { "page": 1, "limit": 20, "total": 42, "total_pages": 3 } } ``` Retrieves a single run with its full schedule snapshot -- the exact schedule configuration at the time of execution. Deletes a single schedule run record. ## Timezone Handling Timezone support is a first-class feature of the scheduling system: All cron expressions and one-off datetimes are converted to **UTC** before storage. The original timezone name is preserved alongside the expression. API responses dynamically convert UTC values back to the original timezone. A cron expression stored as `0 11 * * *` (UTC) for a schedule created with timezone `America/New_York` is returned as `0 6 * * *` (EST) or `0 7 * * *` (EDT) depending on the current DST status. Timezone values are validated against the IANA timezone database (e.g., `America/New_York`, `Europe/London`, `Asia/Tokyo`). Invalid timezone names are rejected with a descriptive error. Always use IANA timezone names (e.g., `America/New_York`) rather than fixed offsets (e.g., `UTC-5`). IANA names automatically handle daylight saving time transitions. ## Next Steps Trigger actions via event-driven webhooks. Connect schedules to data source workflows. Schedule automated agent tasks. Understand how schedules are scoped to projects. # Service Accounts Source: https://docs.emergence.ai/platform/service-accounts Service accounts enable background workers and automated processes to authenticate with CRAFT without user interaction. # Service Accounts Service accounts allow non-interactive processes, background workers, schedulers, cleanup jobs, and automated pipelines, to authenticate with CRAFT using machine identity. Service accounts authenticate against the **master realm** in Keycloak (not the organization realm), which gives them platform-level access scoped by explicit organization and project headers. ## Detection The platform identifies a request as coming from a service account when: 1. The client ID in the JWT begins with the `svc-` prefix (e.g., `svc-data-pipeline`) 2. The JWT contains the `serviceAccount` realm role 3. The authentication was performed against the Keycloak master realm Non-interactive processes using regular user tokens are rejected at service account-protected endpoints. ## Authentication Flow Service accounts use the OAuth 2.0 Client Credentials grant, no user interaction required. ```bash theme={null} curl -X POST \ "https://api.example.com/keycloak/realms/master/protocol/openid-connect/token" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=svc-my-worker" \ -d "client_secret=" ``` Response: ```json theme={null} { "access_token": "", "token_type": "Bearer", "expires_in": 3600 } ``` Service account JWTs do not carry tenant context. Pass these required headers: ```http theme={null} Authorization: Bearer X-Org-Id: org_abc123 X-Project-ID: proj_xyz789 ``` * `X-Org-Id`, the organization context (required; replaces the realm-derived org in user tokens) * `X-Project-ID`, the project scope for project-scoped endpoints (required when applicable) To attribute the request to a specific user in audit logs, also include `X-On-Behalf-Of: ` (optional). The `X-On-Behalf-Of` header is recorded in audit logs but does not grant additional permissions. Include it whenever the service account is operating on a user's behalf. The `get_headers()` helper accepts it as an optional parameter. ## Creating a Service Account In the Keycloak master realm admin console: 1. Navigate to **Clients → Create client** 2. Set **Client ID** with the `svc-` prefix (e.g., `svc-nightly-cleanup`) 3. Set **Client Protocol**: `openid-connect` 4. Set **Access Type**: `confidential` 5. Enable **Service Accounts Enabled** 6. Save and note the generated client secret In the client's **Service Account Roles** tab: 1. Select **Realm Roles** 2. Assign the `serviceAccount` role This role is required for the platform to recognize the client as a service account. Service accounts access the platform APIs with the same permission model as users. Grant the service account access to specific organizations and projects via the platform admin API or the Runtime UI. ## Use Cases | Use Case | Example | | ------------------------------- | ------------------------------------------------------------------- | | **Scheduled data pipelines** | Nightly data ingestion jobs that create agents and data connections | | **Background cleanup** | Purging expired sessions, rotating secrets, archiving old artifacts | | **Workflow orchestration** | Prefect workflows that call platform APIs to register results | | **Automated testing** | CI/CD pipelines that create isolated test resources per run | | **Inter-service communication** | Solution services calling platform APIs | ## Token Expiry and Rotation Service account tokens are short-lived (configurable, default 1 hour). Background workers should implement automatic token refresh: ```python theme={null} import httpx import time class ServiceAccountClient: def __init__(self, client_id: str, client_secret: str, keycloak_url: str): self.client_id = client_id self.client_secret = client_secret self.keycloak_url = keycloak_url self._token = None self._expires_at = 0 def _refresh_token(self): response = httpx.post( f"{self.keycloak_url}/realms/master/protocol/openid-connect/token", data={ "grant_type": "client_credentials", "client_id": self.client_id, "client_secret": self.client_secret, } ) data = response.json() self._token = data["access_token"] self._expires_at = time.time() + data["expires_in"] - 60 # 60s buffer @property def token(self) -> str: if not self._token or time.time() >= self._expires_at: self._refresh_token() return self._token def get_headers(self, org_id: str, project_id: str, on_behalf_of: str | None = None) -> dict: headers = { "Authorization": f"Bearer {self.token}", "X-Org-Id": org_id, "X-Project-ID": project_id, } if on_behalf_of: headers["X-On-Behalf-Of"] = on_behalf_of return headers ``` ## Related Overview of all authentication methods including user tokens and OIDC. How permissions are checked for service account requests. Project-level isolation and the X-Project-ID header. # Webhooks (Planned) Source: https://docs.emergence.ai/platform/webhooks Event-driven webhook delivery with HMAC signing, retry logic, and SSRF protection for CRAFT. # Webhooks Webhooks enable event-driven communication between CRAFT and external systems. When a platform event occurs (such as a schedule trigger, agent completion, or data connection status change), the webhook system delivers a signed HTTP POST request to a configured endpoint. **Coming Soon**, The webhook system described on this page is planned but not yet implemented. This page documents the target architecture and capabilities for an upcoming release. Do not build integrations against these APIs until this warning is removed. ## Architecture The webhook system follows a publish-subscribe pattern built on **Redis Streams**, consistent with the platform's infrastructure-agnostic design: ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TD E["Platform Event"] RS["Redis Streams
(event bus)"] WD["Webhook Dispatcher
HMAC sign → Deliver → Record → Retry"] EX["External System"] E --> RS --> WD --> EX classDef event fill:#56B4E9,stroke:#555555,color:#000 classDef infra fill:#0072B2,stroke:#555555,color:#fff classDef proc fill:#E69F00,stroke:#555555,color:#000 classDef ext fill:#CC79A7,stroke:#555555,color:#000 class E event class RS infra class WD proc class EX ext ``` Redis Streams is used instead of cloud-specific message brokers (SQS, Pub/Sub) to avoid cloud-specific dependencies. Redis is already part of the infrastructure stack, so no additional dependencies are required. ## Security ### HMAC Payload Signing Every webhook delivery includes an HMAC-SHA256 signature in the request headers, allowing receivers to verify that the payload was sent by the platform and has not been tampered with: ```http theme={null} POST /webhook-endpoint HTTP/1.1 Content-Type: application/json X-Webhook-Signature: sha256=a1b2c3d4e5f6... X-Webhook-Timestamp: 1743696000 X-Webhook-Event: schedule.triggered {...payload...} ``` Receivers should: 1. Extract the timestamp and signature from headers 2. Reconstruct the signed payload: `{timestamp}.{body}` 3. Compute HMAC-SHA256 using the shared secret 4. Compare the computed signature with the received one (constant-time comparison) 5. Reject payloads older than a configurable tolerance window (e.g., 5 minutes) to prevent replay attacks ### SSRF Protection The webhook dispatcher includes protection against Server-Side Request Forgery (SSRF): * Private/internal IP ranges are blocked by default (10.x.x.x, 172.16-31.x.x, 192.168.x.x, 127.x.x.x) * DNS resolution is validated before delivery * Redirect following is disabled or limited to prevent redirect-based SSRF ## Retry Logic Failed webhook deliveries are retried with exponential backoff: | Attempt | Delay | Cumulative | | --------- | ---------- | ------------ | | 1st retry | 1 minute | 1 minute | | 2nd retry | 5 minutes | 6 minutes | | 3rd retry | 30 minutes | 36 minutes | | 4th retry | 2 hours | \~2.5 hours | | 5th retry | 12 hours | \~14.5 hours | A delivery is considered successful when the receiver responds with a 2xx status code. After all retries are exhausted, the delivery is marked as failed and recorded in the delivery log. ## Event Types The platform plans to support webhook notifications for the following event categories: | Category | Events | Description | | -------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------- | | **Schedules** | `schedule.triggered`, `schedule.failed` | Schedule execution events | | **Agents** | `agent.registered`, `agent.health_changed` | Agent lifecycle events | | **Assets** | `asset.created`, `asset.updated`, `asset.deleted` | Asset lifecycle events (files, artifacts, data connections) | | **Governance** | `organization.created`, `organization.deleted`, `project.created` | Tenant-boundary events | | **Data Connections** | `connection.status_changed`, `connection.verified` | Connection state changes | | **Deployments** | `deployment.started`, `deployment.completed`, `deployment.failed` | Deployment lifecycle | ## Webhook Payload Structure Webhook payloads follow a consistent envelope format: ```json theme={null} { "id": "evt-a1b2c3d4", "type": "schedule.triggered", "timestamp": "2026-04-03T06:00:00Z", "organization_id": "acme-corp", "project_id": "analytics-prod", "data": { "schedule_id": "sch-x1y2z3", "schedule_name": "Daily Analytics Refresh", "run_id": "run-m1n2o3" } } ``` | Field | Description | | ----------------- | ----------------------------------- | | `id` | Unique event ID for idempotency | | `type` | Event type identifier | | `timestamp` | ISO 8601 UTC timestamp of the event | | `organization_id` | Organization that owns the resource | | `project_id` | Project scope (if applicable) | | `data` | Event-specific payload | Use the event `id` field for idempotency. If you receive the same event ID multiple times (due to retries), process it only once. ## Multi-Tenancy Webhooks are scoped to organizations and optionally to projects, following the same isolation model as all platform resources: * Webhook configurations are owned by an organization and optionally scoped to a project * Events are only delivered for resources within the webhook's scope * Webhook secrets (for HMAC signing) are stored via the Governance Secrets API ## Next Steps Configure time-based triggers that generate webhook events. Monitor agent lifecycle events via webhooks. Receive notifications when connection status changes. Understand how webhook endpoints authenticate with the platform. # Release Notes Source: https://docs.emergence.ai/releases/index CRAFT platform changelog. Calendar-versioned monthly releases with curated, user-facing notes. # CRAFT Release Notes CRAFT ships on a monthly cadence using **calendar versioning** (`YYYY.MM`). Each release bundles updates across the platform; user-facing changes are summarized below in [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) style. **Current Release: CRAFT 2026.06** CRAFT publishes a unified `YYYY.MM` release on or around the last business day of each calendar month. The release name reflects the month it ships in; if a release slips into the following month, the version follows the actual ship month. Underlying components continue to follow semantic versioning and may publish point releases between monthly bundles. *** ## 2026.06 *Released 2026-05-28* ### Added * **Snowflake key-pair and OAuth/SSO authentication for data connectors** — connect to Snowflake without long-lived passwords, aligning data ingestion with enterprise identity and key-rotation policies. See [Data Connections](/platform/data-connections). * **Org-level Snowflake data connection provisioning** — Snowflake connections can now be provisioned at the organization level, reducing per-project setup for teams sharing the same warehouse. See [Data Connections](/platform/data-connections). * **Memory asset filtering** — retrieve memories scoped to a specific dataset, schema, or table using the new `asset_fqn` field on the memories API, so agents can recall context without post-filtering a full memory set. See [Memory](/platform/memory-service). * **412 / 413 / 428 added to the API error reference** — the HTTP status table in [API Errors](/api-reference/errors) now documents Precondition Failed, Payload Too Large, and Precondition Required, matching what every service has been returning all along on agent-registration ETag/If-Match cycles and artifact uploads. ### Changed * **Default LLM is now `gemini-3.5-flash` (Vertex AI)** — across all framework tutorials, gateway configs, and per-call override examples. `gemini-3.5-flash` was GA'd at Google I/O 2026-05-19 and outperforms `gemini-3.1-pro` on 6 of 8 major benchmarks (including agentic MCP Atlas +5.4% and coding Terminal-Bench +6%) at 3.6× the speed. Secondary option shown for model agnosticity is now `claude-opus-4-8` on Vertex AI; OpenAI examples use `gpt-5.5`. See [Access LLMs](/guides/solution-dev/access-llms) and the [LLM Gateway](/platform/llm-gateway) operator guide. * **`gemini-3.1-pro-preview-customtools` re-scoped to specialized use only** — explicit warnings on every appearance: it is a fine-tuned variant of base `gemini-3.1-pro` that prioritizes registered custom functions over bash fallbacks; per Google's own guidance, only choose it when extensive custom tool calling is core to the agent (the model degrades on non-tool workloads if more than half of requests don't involve tools). Preview SLA, global endpoint only, Provisioned Throughput unsupported. * **SDK examples in the API reference now use the real client model** — the generated Python and TypeScript SDKs follow the openapi-python-client / openapi-generator pattern (`AuthenticatedClient` + per-endpoint module functions), not a fluent `GovernanceClient(...).foo.bar()` API that never existed. TypeScript npm scope corrected to `@emergence-ai/em-runtime-*-sdk`. See [API Authentication](/api-reference/authentication). * **Solution Developer Guide ports point at the gateway** — every direct host-port reference (`:8001`, `:8002`, `:8003`) in executable code and curl examples now uses the gateway path (`https:///api/{governance,assets,utils}/...`). The platform services all bind container port 8000; the gateway HTTPRoute fronts them. See [Local Development](/guides/solution-dev/local-development) and [Your First Agent](/guides/agent-author/your-first-agent). * **Agent-registration idempotency contract is now documented end-to-end** — `POST /assets/agents` returns 201 on first registration and 409 `RESOURCE_ALREADY_EXISTS` on a re-POST with the same name in the same project; `PUT` requires an `If-Match` header (428 Precondition Required without; 412 Precondition Failed with a stale ETag; 200 with the current one). See [Your First Agent](/guides/agent-author/your-first-agent). * **RBAC configuration rewritten against the real permissions API** — Steps 2–4 of the RBAC guide now use the live `/governance/permissions/{grant,revoke,check,set-parent,delete-all,accessible-objects}` endpoints. `check` is a `GET` with query parameters; grants and revokes are `POST` with `{resource_type, resource_id, subject, relation}` bodies. See [RBAC Configuration](/guides/rbac-configuration). * **Data connection registration documents the real schema** — the field is `connection_type` (not `type`), the enum value is `postgres` (not `postgresql`), and configuration goes under `database_config` / `storage_config` / `filesystem_config` depending on connector kind, with credentials as a separate top-level block. Pre-save validation lives at `POST /assets/data/verify`; post-save validation at `POST /assets/data/{resource_uri}/verify`. See [Data Source Setup](/guides/data-source-setup). * **Environment-variable conventions clarified for Vertex AI** — `VERTEXAI_PROJECT` / `VERTEXAI_LOCATION` (the LiteLLM convention used by CRAFT's reference Helm chart) is the canonical form; LiteLLM also accepts the Google Cloud SDK standards `GOOGLE_CLOUD_PROJECT` / `GOOGLE_CLOUD_LOCATION` set automatically by `gcloud auth application-default login`. See the [LLM Gateway](/platform/llm-gateway) operator guide. * **Top-bar nav now has icons** matching the left-nav style across all 8 tabs. * **Refreshed platform UI components** — the runtime interface now includes updated Avatar, Accordion, ContextualMenu, SegmentedControl, and Tooltip components, improving visual consistency across the platform. * **Semiconductor v0.3.9 cycle** — Knowledge tab (Documents + Agent Insights), equipment factor analysis embedded directly in generated reports, fab-dashboard-contextual chat answers, Excursion Reconciliation Agent (equipment signals vs yield outcomes), pre-generated report exports, chat runtime progress bar, wafer-map heatmaps, mid-stream agent-task cancellation, and \~30 incremental fixes. See [Semiconductor](/semiconductor/overview). ### Fixed * **Memory API documentation corrected to actual `/utils/` endpoints** — The Memory Service pages previously documented non-existent `/v2/api/memories` paths. Memory is implemented within the Utils service (`em-runtime-utils`); the correct endpoints are `POST /api/utils/context-packs/{name}/memories` (create), `GET /api/utils/memories` (list), `GET /api/utils/memories/{id}` (retrieve), `PATCH /api/utils/memories/{id}` (update), and `DELETE /api/utils/memories/{id}` (soft-delete). Context pack creation requires `pack_type` (not `type`) and pagination uses `page_size` (not `limit`). All requests require an `X-Project-ID` header for project scoping. If you built against the previously documented paths, update to the `/utils/` routes. See [Memory Service](/platform/memory-service) and [Memory Integration](/guides/memory-integration). * **Catalog summary updates** — catalog entity summaries now update correctly across all API call paths. * **Natural-language SQL generation stability** — resolved a crash when table selections were supplied without schemas, and tightened validation on chart type fields so malformed model responses no longer break query generation. See [Data Insights](/data-insights/overview). * **Gemini embedding compatibility** — embeddings produced by Gemini models are now sized to fit the vector store, preventing ingestion failures on memory and catalog writes. * **Data table interactions** — corrected spacing and search behavior in runtime data tables so filtering and selection behave as expected. * **`em-service` chart version pin corrected repo-wide** — `helm install ... --version 0.0.16` was returning "manifest unknown" because only `0.0.15` is published to the OCI registry. The earlier fix covered Quickstart and the em-service chart reference; the pin is now corrected across the whole Solution Developer Guide (starter templates, register-a-solution, package-and-deploy, overview) and the Tier 1 CI fixtures, so every scaffold pins the actually-published `0.0.15`. See [Quickstart](/guides/solution-dev/quickstart) and [em-service chart](/deployment/helm/em-service). * **`hello-solution` resource naming in Quickstart** — adding `api.fullnameOverride: hello-solution-api` to the `values.yaml` ensures `kubectl port-forward svc/hello-solution-api` resolves (without the override, em-service defaults the resource name to the release name and the documented commands return `NotFound`). *** ## 2026.05 *Released 2026-05-13* ### Added * **Catalog foreign-key graph and DDL storage** — the data catalog now traces relationships between entities through foreign-key edges (including soft FKs) and supports path-finding queries, and each catalog entity retains its `CREATE TABLE` DDL alongside profile and lineage metadata. Together these let agents and SQL-generation pipelines navigate schemas without re-deriving structure on every call. See [Data Connections](/platform/data-connections). * **MCP tools for schema exploration** — `get_schema`, `search_schema`, and `sample_data` let any MCP-aware client introspect catalog structure and preview rows from a connected warehouse, so chat surfaces outside the platform can ground answers in live schema rather than stale copies. See [MCP Server](/platform/mcp-server). * **MCP tools for natural-language analytics** — `generate_sql`, `execute_query`, `resolve_term`, and `generate_plotly_chart` expose Data Insights through MCP, so external assistants can ask questions in natural language and receive SQL, results, and charts without leaving their conversation. See [Chat with Data](/data-insights/chat-with-data). * **Wildcard catalog field search** — leading and both-ends wildcards (`*term*`, `term*`) now match field names, so columns can be located from a fragment rather than the full identifier. See [Data Connections](/platform/data-connections). * **HTTP observability metrics with swappable OTEL backends** — every platform service emits consistent HTTP latency, error, and throughput metrics routable to any OpenTelemetry-compatible backend, and a new usage-event endpoint records per-tenant API consumption for reporting. See [OpenTelemetry](/deployment/observability/opentelemetry). * **Project selector in the sidebar** — the runtime UI footer now exposes the active project and a switcher, removing a navigation hop when working across multiple projects. * **Semiconductor Knowledge tab** — a Documents and Agent Insights surface in the fab analytics dashboard so engineers can browse SOPs, ingest curated reference material, and review agent-generated analysis alongside live runs. See [Semiconductor](/semiconductor/overview). * **Equipment factor analysis with report integration** — fab analytics agents now render equipment factor visualizations directly inside generated reports, surfacing tool-level contributions to yield without a separate analysis step. See [Semiconductor](/semiconductor/overview). * **Chat with context from the fab dashboard** — the semiconductor chat agent answers questions grounded in the current dashboard view, so analysts can ask follow-ups without restating which run, wafer, or excursion they are inspecting. See [Semiconductor](/semiconductor/overview). * **Excursion Reconciliation Agent** — compares equipment signals against yield outcomes (including yield-only paths when equipment data is partial) to triangulate root causes for production excursions. See [Semiconductor](/semiconductor/overview). * **Pre-generated semiconductor report exports** — workflow runs now pre-generate downloadable report artifacts at completion instead of waiting for an export request, making large run results immediately shareable. * **Chat runtime progress bar** — long-running semiconductor chat turns now show turn-based progress, so users get visible feedback during multi-minute analysis runs. ### Changed * **Service account secrets use self-descriptive key names** — autonomous workloads read `client_id` and `client_secret` from explicitly-named secret fields rather than generic ones, making rotation and audit clearer. See [Authentication](/platform/authentication). * **Autonomous service accounts resolve consistently in audit and authorization** — request context for non-interactive callers now returns the service account's `client_id` as `user_id`, so authorization checks and audit logs trace back to the right principal regardless of whether the caller is a person or a service. See [Authorization](/platform/authorization). * **Metadata search responses include the publishing service** — each search result now carries a `service_name` field, so multi-service catalogs are easier to navigate and disambiguate. * **Data Readiness gains scoped read access to project metadata** — a new `service_metadata_reader` relation in the project authorization schema grants the Data Readiness service the minimum permissions it needs without broadening other service roles. See [Authorization](/platform/authorization). * **HTTP metrics capture the routed path, not the raw URI** — observability middleware now records the resolved route template, so dashboards group requests by endpoint instead of fragmenting on path parameters. See [OpenTelemetry](/deployment/observability/opentelemetry). * **Faster semiconductor API responses** — caching and bulk-query consolidation cut latency on analytics endpoints under sustained dashboard load, so frequently-viewed runs and excursions render faster on subsequent visits. * **Iceberg ingest path for semiconductor data generation** — the v2 data-generation workflow can now write directly into Apache Iceberg, simplifying integration with lakehouse storage for downstream analytics. See [Semiconductor](/semiconductor/overview). ### Fixed * **OIDC discovery metadata at the issuer-derived path** — well-known OIDC metadata is now served at the path RFC 8414 clients probe (including Claude Code), so MCP authentication flows complete without manual endpoint configuration. See [MCP Server](/platform/mcp-server). * **Whitespace-only catalog search returns an empty result set** — a blank or whitespace query no longer raises an error; it returns no rows, matching the behavior of every other search input. * **Navigation no longer crashes after workflow completion** — moving away from a completed workflow run in the runtime UI no longer throws. * **"Clear All" resets every score type** — clearing scores in the runtime now resets metadata scores through the unified scores endpoint, alongside the other score categories. * **Chat artifact table styling** — table outlines and header backgrounds render correctly in chat artifact previews. * **Semiconductor chat answer accuracy** — fixed a case where numbered option selections were misinterpreted as entity IDs, and added structured error parsing so agent-task failures surface readable messages instead of opaque tracebacks. See [Semiconductor](/semiconductor/overview). * **Reliable wafer-map visualizations in chat** — wafer maps now render as heatmaps rather than scatter plots, restoring readable spatial distributions in chat answers. * **Mid-stream cancellation for semiconductor agent tasks** — disconnecting the client mid-stream now correctly cancels the in-flight agent task, freeing compute and avoiding orphaned work. ### Security * **`@tanstack` packages pinned against a supply chain attack** — exact version pins prevent installation of compromised `@tanstack/*` versions discovered in the npm registry, hardening the runtime UI build against dependency confusion. *** ## 2026.04 *Released 2026-04-30* ### Added * **Conversational data analysis (chat-with-data)** — ask natural-language questions across your registered data assets and receive streaming answers with auto-generated SQL, charts, tables, and downloadable Python code. Sessions support multi-turn follow-ups, artifact attachments, and resumable event replay if the connection drops. See [Chat with Data](/data-insights/chat-with-data). * **Snowflake data connector** — connect to Snowflake warehouses for cataloging, profiling, and natural-language querying alongside existing PostgreSQL and Redshift sources. See [Data Connections](/platform/data-connections). * **Automated data intelligence enrichment** — a new eight-step pipeline runs after profiling to produce table and column descriptions, semantic classifications (Fact/Dimension/Reference), join proposals, sample questions, glossary terms, and text-pattern detections. Each finding is persisted as a searchable memory so downstream agents can reason over your catalog without re-discovery. See [Data Enrichment](/data-governance/data-enrichment). * **Business glossary backed by the memory service** — vocabulary and concept terms now live in context packs with verification status, tagging, and back-references to the tables where each term appears. See [Memory](/platform/memory-service). * **Org-level data connection provisioning** — connections can be registered once at the organization level and reused across projects, reducing per-project setup for shared warehouses. See [Data Connections](/platform/data-connections). * **Audit logging via OpenTelemetry** — platform service and workflow audit events stream through the OTEL collector alongside traces and metrics, unifying observability for compliance-sensitive deployments. See [OpenTelemetry](/deployment/observability/opentelemetry). * **Self-Improvement evals harness for the semiconductor stack** — internal evals infrastructure for the fab analytics agents, exercising agent behaviour against curated scenarios so regressions are caught before they ship. See [Semiconductor](/semiconductor/overview). * **Client-dashboard observability for semiconductor analytics** — the runtime UI's semiconductor client dashboards now emit structured telemetry to the platform OpenTelemetry pipeline, so dashboard performance and agent invocations are visible alongside backend services. * **Excursion editing and tabbed run details (semiconductor)** — engineers can now edit excursion records in place and navigate run details through a tabbed interface, reducing click depth during incident review. See [Semiconductor](/semiconductor/overview). ### Changed * **Breaking: Data quality workflow triggers no longer accept `resource_type`, `resource_uri`, or `database_type`** — tag generation, rule generation, and execution now derive these from the asset FQN, organization, and project. Tag generation accepts a new optional `industry_info` parameter for domain-specific tagging. Callers must remove the deprecated fields from trigger payloads. See [Data Governance Workflows](/data-governance/workflows). * **Data quality results now apply on completion** — DQ execution, DQ assessment, and metadata assessment workflows no longer create a pending approval; results are written directly when the workflow finishes, eliminating the manual approval step. See [Data Governance Workflows](/data-governance/workflows). * **Profiling, Metadata Assessment, and DQ Assessment workflows are now table- or schema-scoped only** — database-wide enrichment moves to the new Data Intelligence workflow, which orchestrates LLM passes across the entire catalog in a single run. See [Data Governance Workflows](/data-governance/workflows). * **Workflow run listings support multi-status filtering and multi-column sorting** — query workflow history with expressions like `?status=running,failed&sort=status:desc,started_at:asc`, and group workflows by category (ingestion, enrichment, data\_quality, assessment) in the configuration list. See [Data Governance Workflows](/data-governance/workflows). * **Column profiles include distribution analysis** — value frequency counts for low-cardinality columns, percentiles for numerics, length and case patterns for text, temporal ranges, and JSON structure breakdowns, controlled by the `DISTRIBUTION_ANALYSIS_ENABLED` flag. See [Data Profiling](/data-governance/data-profiling). * **Refreshed conversational interface for Talk2Data** — streaming UX improvements, an artifact picker panel with infinite scroll, active context bar, session context management, and reliable auto-reconnection on dropped streams. See [Chat with Data](/data-insights/chat-with-data). * **Runtime UI cycle: dashboard navigation polish, agent-card flows, and dozens of incremental fixes** — the runtime UI shipped fifty point releases this cycle covering dashboard navigation flow, agent registration UI, project-context propagation in SDK requests, table interaction polish, and component refresh (Avatar, Accordion, ContextualMenu, Tooltip). * **Resolved agent-card URLs in the semiconductor A2A client** — the fab analytics agents now use the agent's resolved well-known URL rather than its display-name URL, fixing edge cases when registering or invoking agents from custom endpoints. See [Semiconductor](/semiconductor/overview). * **More accurate Otsu threshold for semiconductor yield analysis** — added a MAD-based geometric-mean correction that prevents threshold inflation on noisy distributions, producing more reliable defect/yield segmentation. ### Fixed * **PostgreSQL ingestion to SSL-enforcing databases** — connections to AWS RDS and other managed PostgreSQL deployments no longer fail with `no pg_hba.conf entry … no encryption`; the connection pool now negotiates SSL with the same semantics as `sslmode=require`. See [Data Connections](/platform/data-connections). * **Data quality scorecards reset correctly after a clear** — DQ Score now returns N/A immediately after clearing values and reflects fresh results on the next rule execution, instead of staying pinned at 100%. * **Schema search filters by table name** — searching within a schema returns only tables whose names contain the query, instead of returning every table in the schema. * **Issues requiring attention excludes deleted datasets** — the home dashboard no longer surfaces violations for soft-deleted tables in either the list or the count. * **Profiling stability on Redshift** — large-value aggregations on event-count columns no longer overflow, and distribution analysis no longer crashes on boolean columns. See [Data Profiling](/data-governance/data-profiling). * **Catalog summary updates** — entity summaries now refresh across all API call paths, eliminating stale text after asset updates. ### Security * **MIME type validation on artifact and file uploads** — uploaded files are validated against an allowlist before storage, preventing disguised content from reaching the catalog or downstream agents. * **SQL injection hardening in dataset search** — catalog search escapes user-supplied wildcards and validates query input before composing `ILIKE` filters. * **Stricter data connection name validation** — connection names containing special characters that could break downstream SQL or filesystem paths are rejected at the API boundary. *** ## 2026.03 *Released 2026-03-31* ### Added * **Intelligent context for AI agents** — an 8-step LLM enrichment pipeline runs automatically after profiling, producing asset-granular memories (semantic layer, table summaries, sample questions, JOIN proposals, text patterns) that agents can retrieve at query time. See [Data Enrichment](/data-governance/data-enrichment). * **Memory** — agents can now store and retrieve structured context packs and typed memories (glossary, exemplar, summary, observation, ontology, policy, join) through the Utils service, so they recall task-specific knowledge across sessions. See [Memory](/platform/memory-service). * **Business Glossary** — glossary terms are now persisted as first-class memories with full CRUD, verification, and tagging, replacing the previous placeholder. See [Data Governance](/data-governance/overview). * **Chat with Data** — a new conversational interface for asking natural-language questions of your data, with multi-turn sessions, server-sent responses, a durable per-turn event log for replay, and auto-generated session titles. See [Chat with Data](/data-insights/chat-with-data). * **Column distribution analysis** — column profiles now include histograms: value-frequency counts for low-cardinality columns, percentiles for high-cardinality numerics, length and case analysis for text, temporal ranges, and JSON structure analysis. See [Data Profiling](/data-governance/data-profiling). * **Data Quality approval workflows** — generated DQ tags and rules can be previewed (including SQL and validation status) before being persisted, so reviewers approve before commit. See [Data Governance](/data-governance/overview). * **Hardened semiconductor code-run execution** — the fab analytics code-run service now handles out-of-memory failures gracefully without crashing the agent, with improved observability into long-running runs and corrected log message streaming so users see live progress instead of buffered batches. See [Semiconductor](/semiconductor/overview). * **Persistent client-dashboard auth across tabs (semiconductor)** — fab analytics users no longer need to re-authenticate when opening run details in a new browser tab. * **Runtime UI: initial dashboard navigation, chart controls, and agent integration** — the runtime UI shipped 19 point releases this cycle covering the foundational dashboard navigation flow, chart-control polish, log message streaming, and authentication persistence across the sidebar. ### Changed * **Event-driven ingestion and auto-chained profiling** — creating a data connection now automatically triggers ingestion via the platform event bus, and ingestion auto-chains into profiling on completion. No more manual workflow kickoff. See [Data Connections](/platform/data-connections). * **Simpler DQ workflow inputs** — tag generation, rule generation, and execution no longer require `resource_type`, `resource_uri`, or `database_type` parameters — these are derived internally from the asset FQN. Tag generation also accepts an optional `industry_info` parameter for domain-specific tagging. See [Data Governance](/data-governance/overview). * **DQ execution, DQ assessment, and metadata assessment no longer require approval** — results are available immediately on completion, since these are read-only analyses. Tag and rule *generation* still require approval (preview before commit). See [Data Governance](/data-governance/overview). * **Faster metadata assessments at scale** — scoring and violation detection now run in-memory against the catalog SDK rather than issuing \~28 database queries per asset, sized to handle large catalogs without timing out. See [Data Governance](/data-governance/overview). * **OpenTelemetry instrumentation across the platform** — runtime services now emit traces, metrics, and structured logs to your OpenTelemetry collector, supporting end-to-end request tracing. See [OpenTelemetry](/deployment/observability/opentelemetry). * **Richer catalog responses and filters** — catalog entities now include `row_count`, `column_count`, and `table_count`; metadata responses support `include_children` to expand nested assets; asset list endpoints accept `key_prefix` and `owner_id` filters; tag operations support bulk get/store/delete. See [Data Connections](/platform/data-connections). ### Fixed * **Cross-tenant resource URI collisions in authorization** — the authorization engine now keys resource URIs by tenant, so the same `resource_uri` in different tenants no longer shares permissions. See [Authorization](/platform/authorization). * **Catalog re-ingestion no longer blocked after database-level deletes** — removing assets directly from the database used to leave orphan references that prevented re-ingestion through the API; re-ingestion now succeeds cleanly. See [Data Connections](/platform/data-connections). * **Data connection secrets are cleaned from the secrets backend on deletion** — deleting a connection now removes its credentials from secret storage, eliminating orphaned secrets. See [Data Connections](/platform/data-connections). * **Multi-replica migration race condition** — concurrent platform replicas no longer race when applying database migrations on startup; an advisory lock now serializes migration runs safely. * **More predictable workflow approval and execution** — workflow approval no longer fails on Keycloak token refresh, status updates no longer deadlock under concurrent load, and workflow runs against external Keycloak no longer fail with JWT issuer mismatches. ### Removed * **MinIO removed as the Chat with Data artifact backend** — query results (Parquet, charts, JSON) now live in the platform Artifacts API, authenticated via service-worker tokens. Operators no longer need to provision MinIO alongside Chat with Data. See [Chat with Data](/data-insights/chat-with-data). * **Debug `/config` endpoint removed from production** — internal configuration is no longer exposed over HTTP. ### Security * **Streaming file uploads** — large file and artifact uploads now stream to storage rather than buffering the entire payload in memory, removing a denial-of-service vector on multi-gigabyte uploads. See [Data Connections](/platform/data-connections). * **Stricter resource URI validation in DQ workflows** — DQ tag, rule, and execution workflows now validate resource URIs and propagate organization and project context end-to-end, preventing cross-project asset references. *** ## 2026.02 *Released 2026-02-27* ### Added * **Data Quality workflows for Data Governance** — automated DQ tag generation, rule generation, and rule execution, plus scorecard, trend, and detail endpoints for tracking quality metrics across assets. See [Data Governance](/data-governance/overview) and [Data Profiling](/data-governance/data-profiling). * **Talk to Data foundation** — initial release of the multi-agent system that powers natural-language conversations with structured data sources, including shared libraries and orchestration scaffolding for downstream solutions. See [Data Insights](/data-insights/overview) and [Chat with Data](/data-insights/chat-with-data). * **Table and column profiling with Amazon Redshift support** — a new Profiler workflow scans tables and columns to seed governance assessments, and a Redshift adapter extends profiling beyond Snowflake and Postgres. See [Data Connections](/platform/data-connections). * **Workflow approval flow with real-time status streaming** — Data Governance workflows now support a reviewer approval step before runs are finalized (with reset-to-pending), and a Server-Sent Events stream on workflow endpoints replaces client-side polling for status updates. See [Workflows](/data-governance/workflows). * **Bulk catalog metadata endpoints and entity versioning** — update metadata for many catalog entities in a single request, and track changes over time with major/minor versioning on catalog metadata entities. * **Multivariate excursion detection and report visualizations for the semiconductor solution** — detect lot excursions across correlated variables (not just single-metric thresholds), and generate charts directly inside fab analytics reports through a new visualization agent. See [Semiconductor](/semiconductor/overview). ### Changed * **Connection health visibility** — data connections now expose `status` and `status_message`, and list responses include `owner_id`, so administrators can diagnose connection failures and identify owners without extra requests. See [Data Connections](/platform/data-connections). * **More descriptive workflow status messages** — Prefect-backed workflows now emit structured status messages from tasks and lifecycle hooks, so operators see meaningful progress instead of opaque task names. * **Standardized pagination and error envelope across Data Governance endpoints** — consistent page, size, and cursor semantics and a unified error response shape throughout the API surface. See [Data Governance](/data-governance/overview). * **Breaking: Data Quality workflow APIs split rule generation from execution** — DQ rule generation and execution are now distinct workflow steps; integrations that previously called a single combined endpoint must adopt the two-step generate-then-execute pattern. See [Workflows](/data-governance/workflows). ### Fixed * **Multi-realm Keycloak bootstrap and CORS coverage** — bootstrap against newer Keycloak versions now succeeds, and CORS preflight is enabled on the remaining platform APIs that previously rejected browser requests. See [Authentication](/platform/authentication). * **Database migration safety and connection-pool stability** — migrations now fail fast when multiple Alembic heads are detected, preventing partial schema upgrades, and nested savepoints in workflow event creation are now committed on success, eliminating a slow connection-pool leak under sustained load. * **SSE streaming robustness** — malformed `Last-Event-ID` header values now return 400 instead of crashing the streaming endpoint with a 500. ### Removed * **Breaking: Legacy `domain_id` and `client_id` fields removed from data catalog entities** — these tenant-extension fields are no longer recognized in catalog payloads; callers must migrate to the standard organization and project model. See [Data Connections](/platform/data-connections). *** ## 2026.01 *Released 2026-01-31* ### Added * **Data connection management API** — programmatically register, list, update, and remove PostgreSQL and Redshift connections, with built-in connectivity verification so misconfigured credentials are caught before a workflow runs. See [Data Connections](/platform/data-connections). * **Object storage APIs** — first-class clients and endpoints for managing object-storage assets across deployments, so agents can read and write blobs without bespoke integration code. * **Scoped secrets at organization, project, and resource levels** — store and retrieve secrets at the granularity your workflow actually needs, with permission checks that inherit from the owning resource so secret access stays consistent with the rest of your authorization model. See [Authorization](/platform/authorization). * **Agent guardrails** — opt-in safety checks on agent inputs and outputs, providing a foundation for content moderation, PII handling, and policy enforcement on agentic workflows. See [Agents](/platform/agents). * **Data generation agent for semiconductor workflows** — an interactive agent with web-search capability guides fab engineers through synthetic dataset generation, including reproducible runs via configurable seeds. See [Semiconductor](/semiconductor/overview). * **Data catalog and metadata enrichment workflows** — LLM-powered metadata enrichment over a versioned catalog of databases, schemas, tables, and columns; ingestion runs are tracked end-to-end with configurable retry behavior and live status reporting. See [Data Enrichment](/data-governance/data-enrichment). ### Changed * **A2A protocol for semiconductor chat** — chat services now run on the standard A2A SDK, with end-user authentication propagated through each agent request, aligning the semiconductor solution with the platform's agent-to-agent communication model. See [Semiconductor](/semiconductor/overview). * **Unified bootstrap across platform services** — a single bootstrap flow now provisions identity realms, organizations, permission grants, and database state in one pass, and surfaces project-level readiness in the bootstrap status response. * **Stateless yield-excursion workflow** — yield-excursion pipelines now pass data via explicit dictionaries between steps rather than shared state, making runs easier to reason about, retry, and observe. See [Semiconductor](/semiconductor/overview). * **Standardized workflow naming and a service-ingestion workflow type** — workflow names follow a single convention across ingestion, enrichment, and analysis runs, and service ingestion is now a first-class workflow type. See [Data Governance Workflows](/data-governance/workflows). ### Fixed * **Pearson correlation in excursion analysis** — corrected a case where the Pearson coefficient was applied to inputs it shouldn't have been, restoring expected correlation results in yield-excursion runs. See [Semiconductor](/semiconductor/overview). * **Windows compatibility for self-improvement runs** — resolved path and environment issues that prevented self-improvement workflows from running on Windows hosts. * **Local Docker Compose startup for semiconductor services** — fixed build and runtime errors that blocked bringing up all semiconductor services together, easing local end-to-end testing. * **Reports now reference the analysis run, not the underlying orchestrator run** — generated reports link to the correct analysis run ID, so report URLs match what users see in the analysis history. * **Catalog asset generation** — addressed a case where catalog asset generation could silently no-op under certain bootstrap states. ### Security * **Permission enforcement on all secrets endpoints** — every create, read, update, and delete on the secrets API is now gated on the caller's authorization against the owning resource, closing a gap where checks based on secret type alone could be bypassed. See [Authorization](/platform/authorization). * **Container base images pinned to immutable digests** — workflow runner images now reference specific tool versions instead of `:latest`, preventing supply-chain drift if a published tag is moved or replaced. * **Hardened API surface** — standardized exception handling and additional input validation across runtime services reduce the risk of information disclosure via unhandled errors. See [API Reference](/api-reference/introduction). # SDKs & Client Libraries Source: https://docs.emergence.ai/sdks/overview Auto-generated Python and TypeScript SDKs for CRAFT APIs # SDKs & Client Libraries CRAFT provides auto-generated SDKs for **Python** and **TypeScript**, built from OpenAPI specifications on every release. Type-safe Python client with async support, Pydantic models, and full API coverage. Fully typed TypeScript client for Node.js and browser environments. ## Available SDKs | Package | Python (PyPI) | TypeScript (npm) | API | | -------------- | --------------------------- | ----------------------------------------- | ------------------------------------------------- | | **Governance** | `em-runtime-governance-sdk` | `@emergence-ai/em-runtime-governance-sdk` | Organizations, Projects, Permissions | | **Assets** | `em-runtime-assets-sdk` | `@emergence-ai/em-runtime-assets-sdk` | Data Connections, Artifacts, Files, Models | | **Utils** | `em-runtime-utils-sdk` | `@emergence-ai/em-runtime-utils-sdk` | Data Catalog, Scheduling, Context Packs, Memories | ## SDK Generation Pipeline SDKs are auto-generated from OpenAPI specs using [openapi-python-client](https://github.com/openapi-generators/openapi-python-client) and the OpenAPI Generator for TypeScript. The pipeline runs on every final release: FastAPI apps export their schemas via `app.openapi()` at build time. OpenAPI specs are fed to code generators producing typed clients with models, API classes, and authentication helpers. Python SDKs publish to Google Artifact Registry. TypeScript SDKs publish to the `@emergence-ai` npm scope. SDK versions match the platform release version (e.g., platform v5.28.0 → SDK v5.28.0). SDK versions may lag behind the latest platform service versions. For example, the Runtime UI may use Assets SDK v5.27.0 while the Assets service is at v5.28.0. SDKs are backward-compatible, newer service versions serve older SDK clients without issue. Check the release notes for current versions. ## Authentication All SDKs use JWT bearer token authentication via Keycloak: ```python theme={null} from em_runtime_governance_sdk import AuthenticatedClient client = AuthenticatedClient( base_url="https://your-instance.emergence.ai", token="your-jwt-token" ) ``` ```typescript theme={null} import { Configuration, OrganizationsApi } from '@emergence-ai/em-runtime-governance-sdk'; const config = new Configuration({ basePath: 'https://your-instance.emergence.ai', accessToken: 'your-jwt-token', }); const api = new OrganizationsApi(config); ``` ## Registry Configuration SDKs are hosted on Google Artifact Registry (private). Configure your package manager: ```bash theme={null} # Add the Artifact Registry index uv add em-runtime-governance-sdk --index-url https://us-central1-python.pkg.dev/aip-artifacts-store/aipy/simple/ # Or with pip pip install em-runtime-governance-sdk \ --index-url https://us-central1-python.pkg.dev/aip-artifacts-store/aipy/simple/ ``` ```bash theme={null} # Configure the @emergence-ai scope echo "@emergence-ai:registry=https://us-central1-npm.pkg.dev/aip-artifacts-store/aip-npm/" >> .npmrc # Install npm install @emergence-ai/em-runtime-governance-sdk ``` Registry authentication requires Google Cloud credentials. See your platform admin for access tokens or Workload Identity setup. ## Next Steps Installation, usage patterns, error handling Setup, type safety, async patterns Interactive API playground with all endpoints Keycloak OIDC setup and token management # Python SDK Source: https://docs.emergence.ai/sdks/python Type-safe Python client for CRAFT APIs # Python SDK The CRAFT Python SDK provides type-safe, auto-generated clients for all platform APIs. Built with Pydantic models and httpx for async support. ## Installation ```bash theme={null} # Install all SDKs uv add em-runtime-governance-sdk em-runtime-assets-sdk em-runtime-utils-sdk \ --index-url https://us-central1-python.pkg.dev/aip-artifacts-store/aipy/simple/ # Or install individually uv add em-runtime-governance-sdk \ --index-url https://us-central1-python.pkg.dev/aip-artifacts-store/aipy/simple/ ``` ## Quick Start ### Create a Client ```python theme={null} from em_runtime_governance_sdk import AuthenticatedClient # Create an authenticated client client = AuthenticatedClient( base_url="https://your-instance.emergence.ai/governance", token="your-jwt-token", ) ``` ### List Organizations ```python theme={null} from em_runtime_governance_sdk.api.organizations import list_organizations from em_runtime_governance_sdk.models import OrganizationListResponse response = list_organizations.sync(client=client) for org in response.data: print(f"{org.id}: {org.name}") ``` ### Create a Project ```python theme={null} from em_runtime_governance_sdk.api.projects import create_project from em_runtime_governance_sdk.models import ProjectCreate project = create_project.sync( client=client, body=ProjectCreate( name="My Project", description="Data analysis workspace", ), ) print(f"Created project: {project.id}") ``` ### Create a Data Connection ```python theme={null} from em_runtime_assets_sdk import AuthenticatedClient as AssetsClient from em_runtime_assets_sdk.api.data import create_data_connection from em_runtime_assets_sdk.models import DataConnectionCreate assets_client = AssetsClient( base_url="https://your-instance.emergence.ai/assets", token="your-jwt-token", ) connection = create_data_connection.sync( client=assets_client, body=DataConnectionCreate( name="analytics-db", connection_type="postgres", description="Analytics PostgreSQL database", ), ) ``` ## Async Usage All SDK methods support async via the `.asyncio()` variant: ```python theme={null} import asyncio from em_runtime_governance_sdk.api.organizations import list_organizations async def main(): response = await list_organizations.asyncio(client=client) for org in response.data: print(f"{org.id}: {org.name}") asyncio.run(main()) ``` ## Error Handling ```python theme={null} from em_runtime_governance_sdk.types import Response from httpx import HTTPStatusError try: response = list_organizations.sync_detailed(client=client) if response.status_code == 200: data = response.parsed elif response.status_code == 401: print("Token expired — refresh and retry") elif response.status_code == 403: print("Insufficient permissions") except HTTPStatusError as e: print(f"HTTP error: {e.response.status_code}") ``` ## Package Reference | Package | Covers | Import | | --------------------------- | ------------------------------------------------- | ------------------------------------------- | | `em-runtime-governance-sdk` | Organizations, Projects, Permissions, Roles | `from em_runtime_governance_sdk import ...` | | `em-runtime-assets-sdk` | Artifacts, Data Connections, Files, Models | `from em_runtime_assets_sdk import ...` | | `em-runtime-utils-sdk` | Data Catalog, Scheduling, Context Packs, Memories | `from em_runtime_utils_sdk import ...` | ## Next Steps TypeScript/JavaScript client guide Interactive API playground # TypeScript SDK Source: https://docs.emergence.ai/sdks/typescript Fully typed TypeScript client for CRAFT APIs # TypeScript SDK The CRAFT TypeScript SDK provides fully typed clients for all platform APIs, with support for Node.js and browser environments. ## Installation ```bash theme={null} # Configure the @emergence-ai npm scope echo "@emergence-ai:registry=https://us-central1-npm.pkg.dev/aip-artifacts-store/aip-npm/" >> .npmrc # Install all SDKs npm install @emergence-ai/em-runtime-governance-sdk \ @emergence-ai/em-runtime-assets-sdk \ @emergence-ai/em-runtime-utils-sdk ``` ## Quick Start ### Configure the Client ```typescript theme={null} import { Configuration, OrganizationsApi, ProjectsApi, } from '@emergence-ai/em-runtime-governance-sdk'; const config = new Configuration({ basePath: 'https://your-instance.emergence.ai/governance', accessToken: 'your-jwt-token', }); const orgsApi = new OrganizationsApi(config); const projectsApi = new ProjectsApi(config); ``` ### List Organizations ```typescript theme={null} const { data: orgs } = await orgsApi.listOrganizations(); orgs.data.forEach(org => { console.log(`${org.id}: ${org.name}`); }); ``` ### Create a Project ```typescript theme={null} const { data: project } = await projectsApi.createProject({ projectCreate: { name: 'My Project', description: 'Data analysis workspace', }, }); console.log(`Created project: ${project.id}`); ``` ### Create a Data Connection ```typescript theme={null} import { Configuration as AssetsConfig, DataApi, } from '@emergence-ai/em-runtime-assets-sdk'; const assetsConfig = new AssetsConfig({ basePath: 'https://your-instance.emergence.ai/assets', accessToken: 'your-jwt-token', }); const dataApi = new DataApi(assetsConfig); const { data: connection } = await dataApi.createDataConnection({ dataConnectionCreate: { name: 'analytics-db', connectionType: 'postgres', description: 'Analytics PostgreSQL database', }, }); ``` ## Error Handling ```typescript theme={null} import { AxiosError } from 'axios'; try { const { data } = await orgsApi.listOrganizations(); } catch (error) { if (error instanceof AxiosError) { switch (error.response?.status) { case 401: console.error('Token expired — refresh and retry'); break; case 403: console.error('Insufficient permissions'); break; default: console.error(`API error: ${error.response?.status}`); } } } ``` ## Token Refresh Pattern ```typescript theme={null} import { Configuration } from '@emergence-ai/em-runtime-governance-sdk'; // Dynamic token provider const config = new Configuration({ basePath: 'https://your-instance.emergence.ai/governance', accessToken: async () => { // Fetch fresh token from Keycloak const token = await refreshKeycloakToken(); return token; }, }); ``` ## Package Reference | Package | Covers | Scope | | ----------------------------------------- | ------------------------------------------------- | ------------ | | `@emergence-ai/em-runtime-governance-sdk` | Organizations, Projects, Permissions | `governance` | | `@emergence-ai/em-runtime-assets-sdk` | Artifacts, Data Connections, Files, Models | `assets` | | `@emergence-ai/em-runtime-utils-sdk` | Data Catalog, Scheduling, Context Packs, Memories | `utils` | ## Next Steps Python client guide Interactive API playground # Authentication Source: https://docs.emergence.ai/security/authentication Deep dive into Keycloak authentication, OIDC flows, JWT token management, and machine-to-machine authentication on CRAFT. # Authentication CRAFT uses **Keycloak** for all authentication. Every API request requires a valid JWT token, and each organization maps to a dedicated Keycloak realm. This page covers OIDC flows, token management, machine-to-machine authentication, and session configuration. ## Authentication Architecture Browser-based applications authenticate using the **OIDC Authorization Code flow with PKCE**: 1. The frontend redirects the user to the Keycloak realm login page 2. The user authenticates (password, SSO, or MFA) 3. Keycloak returns an authorization code to the frontend callback URL 4. The frontend exchanges the code for JWT tokens (access + refresh + ID) 5. The access token is included in all API requests as `Authorization: Bearer ` The PKCE extension protects against authorization code interception attacks in public clients (browser-based applications that cannot securely store a client secret). Service accounts, CI/CD pipelines, and automation tools authenticate using the **Keycloak client credentials grant**: ```bash theme={null} curl -X POST \ "https:///realms//protocol/openid-connect/token" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=" \ -d "client_secret=" ``` Client credentials are managed entirely by Keycloak -- there is no custom API key lifecycle. Keycloak handles credential rotation, revocation, and expiration. ## JWT Token Structure Access tokens issued by Keycloak contain the following claims used by the platform: | Claim | Source | Description | | -------- | ------------------ | ---------------------------------------------------------- | | `sub` | Keycloak | Unique user identifier | | `iss` | Keycloak realm URL | Issuer URL including the realm ID | | `org_id` | Derived from realm | Organization ID (realm ID = org ID) | | `groups` | Keycloak groups | Group memberships for role mapping | | `aud` | Keycloak client | Audience -- identifies the intended recipient of the token | | `exp` | Keycloak | Token expiration timestamp | | `iat` | Keycloak | Token issuance timestamp | The `project_id` is never stored in the JWT. It is passed via the `X-Project-ID` HTTP header or as a path parameter on each API request. This allows a single token to access multiple projects within the same organization. ## Token Validation Platform services validate JWT tokens using the following process: The service fetches the Keycloak JWKS (JSON Web Key Set) from the realm's well-known endpoint: `https:///realms//protocol/openid-connect/certs` The token's RS256 signature is verified against the public key from the JWKS endpoint. The service validates the `iss` (issuer) and `exp` (expiration) claims. Expired or malformed tokens receive a `401 Unauthorized` response. Audience (`aud`) validation is not currently enforced; this is a known limitation tracked for remediation. The `org_id` is extracted from the realm portion of the issuer URL. All subsequent queries are scoped to this organization. ## Multi-Realm Architecture Each organization in the platform is a separate Keycloak realm: | Aspect | Implementation | | ------------------------ | ----------------------------------------------- | | **Realm ID** | Equals the organization ID | | **User directory** | Each realm has its own user store | | **IdP configuration** | SSO providers are configured per realm | | **Client registrations** | Applications and service accounts are per realm | | **Session management** | Sessions are isolated per realm | This architecture provides complete tenant isolation at the authentication level. A user authenticated in one realm cannot access resources in another realm's organization. ## Single Sign-On (SSO) Keycloak supports SSO via both OIDC and SAML 2.0 for enterprise identity providers: | IdP | Protocol | Configuration | | ------------------ | -------- | -------------------------------------- | | Microsoft Entra ID | OIDC | App registration + group claims | | Okta | OIDC | Application integration + group filter | | Google Workspace | OIDC | OAuth client credentials | | PingFederate | SAML 2.0 | SP metadata exchange | ### SSO Features * **Per-tenant configuration**: Each organization connects to its own IdP * **Just-in-time provisioning**: Users are created on first SSO login * **Forced SSO**: Organizations can disable password login, requiring all users to authenticate via the configured IdP * **Session management**: Configurable idle timeout, max session duration, and forced re-authentication See the [SSO Integration guide](/guides/sso-integration) for step-by-step configuration. ## Anonymous Access The platform supports optional anonymous read-only access for public resources: | Setting | Value | | ------------------ | ------------------------------------------------------------------------------------------------ | | **Scope** | Read-only access to PUBLIC entities via search and list endpoints | | **Rate limit** | 30 requests per minute (target; must be enforced at ingress/WAF layer by deploying organization) | | **Redaction** | Security schemes and credentials are omitted from responses | | **Default** | Disabled per tenant; must be explicitly enabled | | **Implementation** | `OptionalAuth` dependency returns `None` for unauthenticated requests | Anonymous endpoints must be explicitly allowlisted in the service configuration. The standard `get_auth()` dependency always requires a valid JWT. ## Rate Limiting Rate limiting must be enforced at the ingress/WAF layer by the deploying organization. The following targets are recommended: | Operation Type | Recommended Limit | Enforcement | | --------------------------------- | ----------------- | ------------------ | | Writes (POST, PUT, PATCH, DELETE) | 60/min | Ingress/WAF policy | | Reads (GET, POST :search) | 300/min | Ingress/WAF policy | | Bulk operations | 10/min | Ingress/WAF policy | | Anonymous reads | 30/min | Ingress/WAF policy | When rate limiting is configured, the ingress layer should return `429 Too Many Requests` with a `Retry-After` header and include standard rate limit response headers (`X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`). Rate limiting is not enforced by default in the platform. Deploying organizations must configure rate limiting at their ingress controller or WAF (e.g., GCP Cloud Armor, AWS WAF, Nginx rate limiting). ## Token Management Best Practices Use the refresh token to obtain new access tokens before the access token expires. Keycloak refresh tokens have a longer lifetime than access tokens. Implement proactive refresh (e.g., refresh when 75% of the token lifetime has elapsed) rather than waiting for a 401 response. Rotate client secrets periodically. Keycloak supports multiple active client secrets during rotation, allowing a grace period for credential updates across services. Never store tokens in localStorage for browser applications. Use httpOnly cookies or in-memory storage with the token refresh flow. For server-side applications, store tokens in encrypted session state. ## Troubleshooting Check the token's `iss` claim matches the expected Keycloak realm URL. Verify the Keycloak server is reachable and the JWKS endpoint returns valid keys. Clock skew between services can cause premature token expiration. Verify the redirect URI configured in the IdP matches the Keycloak broker endpoint. Check that the Keycloak realm has the identity provider enabled and the browser flow is configured correctly. Ensure a group mapper is configured in the Keycloak client settings. The mapper must be set to add group memberships to the access token (not just the ID token). ## Next Steps Configure SSO with Entra ID, Okta, or Google Workspace. Set up OpenFGA roles and permissions for authorization. Learn how to authenticate API requests with JWT tokens. Review TLS and network policies for securing authentication flows. # GDPR Compliance Source: https://docs.emergence.ai/security/compliance/gdpr How CRAFT supports GDPR compliance with data handling, Right to Be Forgotten, audit logging, and data residency controls. # GDPR Compliance CRAFT is designed to support organizations in meeting their obligations under the **General Data Protection Regulation (GDPR)**. This page covers data handling, Right to Be Forgotten (RTBF) implementation, audit logging, and data residency controls. CRAFT provides the technical mechanisms for GDPR compliance. Organizations are responsible for configuring and operating the platform in accordance with their specific GDPR obligations, Data Protection Impact Assessments (DPIAs), and legal counsel. ## Data Processing Principles The platform implements GDPR's data processing principles at the architecture level: | Principle | Implementation | | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Lawfulness** | Platform enforces authenticated access controls; deploying organizations must establish and document their Article 6 lawful basis (e.g., legitimate interests, contract performance, consent) for each processing activity in their Records of Processing Activities (RoPA) | | **Purpose limitation** | Data connections are scoped to specific projects with defined purposes | | **Data minimization** | Agent Cards apply data minimization to provider contact information | | **Accuracy** | Audit logs record all data modifications with timestamps | | **Storage limitation** | Configurable retention policies per tenant and data type | | **Integrity and confidentiality** | Encryption at rest (AES-256) and in transit (TLS 1.2 minimum) | | **Accountability** | Immutable audit logs with full traceability | ## Personal Data Inventory The platform processes the following categories of personal data: | Category | Location | Sensitivity | Purpose | Suggested Lawful Basis | | --------------------------- | ------------------ | ------------ | ------------------------ | ----------------------------------------------- | | User identity (email, name) | Keycloak realms | Restricted | Authentication | Contract performance or legitimate interests | | User IP address | Audit logs | Restricted | Security monitoring | Legitimate interests | | User agent string | Audit logs | Internal | Debugging | Legitimate interests | | Agent Card provider contact | Assets database | Confidential | Agent registration | Contract performance | | Session data | Redis, PostgreSQL | Confidential | Multi-turn conversations | Contract performance | | Query results | Solution databases | Confidential | Data analytics | Determined by organization based on data source | The "Suggested Lawful Basis" column provides guidance only. Organizations must confirm the lawful basis for each processing activity in their specific deployment context and document it in their Records of Processing Activities (RoPA). ## Right to Be Forgotten (RTBF) The platform implements RTBF using **crypto-shredding** -- destroying the encryption key rather than locating and deleting every instance of the data. ### How Crypto-Shredding Works When audit log entries are created, PII fields (`actor` identity, `ip_address`) are encrypted using a per-principal encryption key. An administrator or the data subject initiates an RTBF request via the Governance API. The principal's encryption key is permanently deleted. Without the key, encrypted fields become unrecoverable. The audit log entry structure is preserved (action, resource\_type, resource\_id, timestamp, trace\_id), but the `actor` field displays `[REDACTED]`. ### Data Deletion Matrix When an RTBF request is processed, data is deleted across all storage systems: | Storage | Action | Timing | | ----------------------------- | --------------------------------------------------------- | ---------------------------------------- | | **PostgreSQL** | Hard-delete rows from all tables | Immediate | | **OpenFGA** | Delete authorization relationship tuples | Immediate | | **Object storage** | Delete skill package artifacts | Immediate | | **Redis** | Invalidate cached entries | Immediate | | **Event bus** (Redis Streams) | Events are ephemeral (7-day retention) | No action needed beyond retention expiry | | **WAL/PITR backups** | Backup retention must not exceed RTBF compliance deadline | Per backup retention policy | | **Operational logs** | PII limited to tenant\_id and actor; 90-day max retention | Retention-based expiry | WAL and PITR backups may contain personal data. GDPR controllers have one month to respond to RTBF requests (extendable by two months for complex cases under Art. 12(3)). Recommended maximum backup retention is 30-90 days to stay within RTBF response obligations. The default infrastructure configuration retains 7 daily backups with 7-day transaction log retention. Long-term backups containing personal data require either crypto-shredding capability or documented justification under Art. 17(3) exemptions. ## Audit Logging Every state-changing operation produces an immutable audit entry: | Field | Type | Description | | --------------- | --------- | ---------------------------------------------------------- | | `id` | ULID | Unique entry identifier | | `tenant_id` | string | Tenant scope | | `action` | string | e.g., `agent.register`, `agent.approve`, `policy.update` | | `actor` | string | Authenticated principal (encrypted with per-principal key) | | `actor_type` | enum | USER, SERVICE\_ACCOUNT, SYSTEM | | `resource_type` | string | Target resource type | | `resource_id` | string | Target resource identifier | | `timestamp` | timestamp | When the action occurred | | `details` | JSONB | Old/new values, approval comments, policy violations | | `ip_address` | string | Client IP (encrypted with per-principal key) | | `trace_id` | string | OpenTelemetry trace ID for end-to-end correlation | ### Audit Log Properties * **Append-only**: No updates or deletes in normal operation * **Retention**: Configurable per tenant (default 1 year; extended retention only where required by specific regulation, e.g., financial record-keeping) * **Export**: `GET /governance/admin/audit-logs?format=csv` for compliance reporting * **SIEM integration**: Forward to Splunk, Datadog, or SIEM via OpenTelemetry Collector (customer-configurable exporters) * **Filterable**: By actor, action, date range, and resource ## Breach Notification GDPR Article 33 requires that in the event of a personal data breach, the controller must notify the competent supervisory authority without undue delay and, where feasible, no later than **72 hours** after becoming aware of it. Article 34 requires notification to affected data subjects where the breach is likely to result in a high risk to their rights and freedoms. The platform supports breach detection and timely reporting through: * **SIEM integration**: Real-time forwarding of security events via OpenTelemetry Collector for anomaly detection * **Audit log analysis**: Identify unauthorized access patterns via audit log queries and export * **OpenTelemetry alerting**: Configure alerts for unusual access patterns (high volume queries, off-hours access, privilege escalation) * **Trace correlation**: `trace_id` in audit logs enables end-to-end investigation of suspected breaches Organizations must establish their own breach notification procedures, including designated contacts for supervisory authority notification, internal escalation chains, and data subject communication templates. The platform provides detection and evidence tools, not the notification process itself. ## Data Residency For GDPR Article 44-49 compliance (international data transfers): * **Region labels**: Each tenant is tagged with a data residency region (EU, US, APAC, custom) * **Geo-fenced storage**: Database partitioning by region ensures EU data stays in EU * **Regional API endpoints**: Optional per-region API gateway routing (e.g., `eu.platform.example.com`) * **Cross-region search**: Federated queries can span regions only with explicit opt-in * **Data location attestation**: API endpoint returns storage location metadata per entity ### Legal Transfer Mechanisms Technical geo-fencing alone does not constitute GDPR-compliant transfer controls. Cross-border data flows require a valid legal transfer mechanism under GDPR Chapter V: * **Standard Contractual Clauses (SCCs)** (Art. 46(2)(c)) -- the most common mechanism for international transfers * **Adequacy decisions** (Art. 45) -- transfers to countries with an adequate level of data protection * **Binding Corporate Rules** (Art. 47) -- for intra-group international transfers Organizations should review their Data Processing Agreements (DPAs) with cloud providers to ensure EU-standard protections for transferred data, including appropriate SCCs. ## Consent and Access Controls | GDPR Right | Platform Mechanism | | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Right of access** (Art. 15) | Audit logs and data export APIs. Organizations must respond to DSARs within one month of receipt (Art. 12(3)), extendable by two months for complex requests | | **Right to rectification** (Art. 16) | Standard CRUD APIs for data correction | | **Right to erasure** (Art. 17) | Crypto-shredding + data deletion matrix | | **Right to restrict processing** (Art. 18) | Tenant-level freeze / per-resource access controls | | **Right to data portability** (Art. 20) | JSON/CSV export APIs | | **Right to object** (Art. 21) | Organizations must implement a process to receive and act on individual objection requests. Platform supports per-user processing restrictions via access controls. Tenant-level processing settings provide additional scope controls | ## Data Protection Impact Assessment (DPIA) When deploying the platform for processing personal data, consider the following for your DPIA: Map the flow of personal data through the platform: authentication tokens, audit logs, data connections, query results, and agent interactions. Identify which components process personal data and the legal basis for each. Configure the platform to collect only the data necessary for your use case. Disable anonymous access logging if not needed. Configure audit log retention to the minimum required period. If using cloud-managed services (Cloud SQL, RDS, Azure Database), ensure your Data Processing Agreement (DPA) with the cloud provider covers the data being processed. If using cloud-based LLM APIs, a DPA with the LLM provider is required. A formal DPIA is mandatory under Art. 35 for processing likely to result in high risk, including: systematic profiling, large-scale processing of sensitive data (e.g., healthcare via Data Insights), and automated decision-making. Engage your Data Protection Officer (DPO) per Art. 37 in reviewing the DPIA, and consult your supervisory authority under Art. 36 if residual risk remains high after mitigation. ## Compliance Reporting The platform provides compliance reports via the admin API: | Report | Content | | ----------------------------- | --------------------------------------------------------- | | **Data processing inventory** | All data connections, their purposes, and access patterns | | **Access review** | All users and their permissions per tenant | | **Audit trail export** | Full audit log for a specified date range | | **RTBF confirmation** | Confirmation of data deletion across all storage systems | ## Next Steps See how the platform maps to SOC 2 Trust Service Criteria. Review HIPAA compliance considerations for healthcare deployments. Understand data classification levels and encryption requirements. Configure backup retention to align with GDPR requirements. # HIPAA Compliance Source: https://docs.emergence.ai/security/compliance/hipaa HIPAA compliance considerations for deploying CRAFT in healthcare environments with PHI data handling requirements. # HIPAA Compliance This page covers considerations for deploying CRAFT in healthcare environments where **Protected Health Information (PHI)** may be processed. The platform's security architecture provides the technical controls needed to support HIPAA compliance when properly configured. HIPAA compliance is a shared responsibility. The platform provides technical safeguards, but organizations must implement administrative safeguards (policies, training, BAAs) and physical safeguards appropriate to their deployment model. Consult with your compliance team and legal counsel. ## HIPAA Applicability HIPAA may apply when CRAFT is used to: * Query healthcare databases containing patient records via Data Insights * Profile and govern healthcare data assets via Data Governance * Process data from Electronic Health Records (EHR) systems * Generate analytics or visualizations from clinical data In these scenarios, the platform acts as a **Business Associate** processing PHI on behalf of the Covered Entity (healthcare organization). ## HIPAA Safeguards Mapping ### Technical Safeguards (45 CFR 164.312) | HIPAA Requirement | Section | Platform Control | | ------------------------------ | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Access control** | 164.312(a)(1) | Keycloak multi-realm authentication with OIDC; OpenFGA ReBAC with role-based permissions | | **Unique user identification** | 164.312(a)(2)(i) | UUID-based user identifiers in Keycloak; JWT `sub` claim for audit attribution | | **Emergency access procedure** | 164.312(a)(2)(ii) | Admin override capabilities; Keycloak emergency access accounts. Organizations must document emergency access procedures including: invocation criteria, authorization chain, mandatory post-use audit review, and account lifecycle controls | | **Automatic logoff** | 164.312(a)(2)(iii) | Session idle timeout and max duration configurable via Keycloak realm settings (Keycloak defaults: 30 min idle, 10 hr max). HIPAA deployments should configure per Minimum Viable HIPAA Configuration (15 min idle, 8 hr max recommended) | | **Encryption and decryption** | 164.312(a)(2)(iv) | AES-256 encryption at rest; TLS 1.3 for external, TLS 1.2 minimum for internal | | **Audit controls** | 164.312(b) | Immutable audit logs with ULID identifiers, configurable retention | | **Integrity controls** | 164.312(c)(1) | Parameterized SQL queries; schema validation; Pydantic request/response validation | | **Authentication** | 164.312(d) | Multi-factor authentication via Keycloak; SSO with enterprise IdPs | | **Transmission security** | 164.312(e)(1) | TLS 1.3 for all external communication; TLS 1.2 minimum for internal. mTLS available via service mesh | ### Administrative Safeguards (45 CFR 164.308) These require organizational implementation supported by platform capabilities: | HIPAA Requirement | Section | Platform Support | | --------------------------------- | ------------- | ------------------------------------------------------------------------------- | | **Security management process** | 164.308(a)(1) | Audit logs for risk assessment; vulnerability scanning in CI/CD | | **Workforce security** | 164.308(a)(3) | OpenFGA role assignments; SCIM 2.0 deprovisioning; access reviews | | **Information access management** | 164.308(a)(4) | Project-level access isolation; role-based computed permissions | | **Security awareness training** | 164.308(a)(5) | Organization responsibility (not platform-provided) | | **Security incident procedures** | 164.308(a)(6) | OpenTelemetry trace correlation; SIEM integration via OpenTelemetry Collector | | **Contingency plan** | 164.308(a)(7) | Backup/restore procedures; disaster recovery checklist; multi-zone availability | | **Evaluation** | 164.308(a)(8) | Periodic access reviews; compliance reporting APIs | | **BAA with subcontractors** | 164.308(b)(1) | Required with cloud providers when using managed services | ### Physical Safeguards (45 CFR 164.310) Physical safeguards are primarily the responsibility of the deployment environment: | HIPAA Requirement | Platform Relevance | | ----------------------------- | ------------------------------------------------------------------- | | **Facility access controls** | Cloud provider data center security (GCP, AWS, Azure SOC 2 reports) | | **Workstation use/security** | Organization responsibility for client devices | | **Device and media controls** | Encryption at rest; crypto-shredding for data disposal | ## PHI Data Flow Controls When the platform processes PHI, apply these additional controls: ### Data Connection Security Create a dedicated read-only database user for PHI data connections. Grant access only to the specific tables and schemas needed. Never grant write access unless explicitly required. Set `ssl_mode: verify-full` on all data connections to PHI databases. This ensures encrypted connections with server identity verification. Use project-level isolation to restrict which teams can query PHI databases. Assign the `viewer` role to analytics users who should only read query results, and `developer` to those who configure data connections. HIPAA requires 6-year retention for policies, procedures, and documentation (45 CFR 164.530(j)(2)). Audit log retention is a separate organizational decision based on risk assessment. Configure `retention_years: 7` as a recommended default to cover both documentation and audit trail needs. ### Data Minimization | Control | Implementation | | ---------------------- | ----------------------------------------------------------------------------------------------------- | | Query result retention | Configure session TTL to minimize PHI retention in platform databases | | Cache expiration | Set Redis TTL to the minimum needed for session continuity | | Log redaction | Ensure PHI is not logged in operational logs; limit to tenant\_id and actor | | Profiling scope | When profiling PHI databases, limit profiling to metadata (schema, statistics) rather than raw values | ## Minimum Viable HIPAA Configuration For healthcare deployments, apply these configuration overrides: ```yaml theme={null} # Helm values for HIPAA-compliant deployment security: tls: minVersion: "1.3" # External traffic; internal defaults to 1.2 minimum enforceExternal: true # enforceInternal: true # Requires service mesh for internal TLS 1.3 keycloak: sessions: idleTimeout: "15m" # HIPAA automatic logoff (Keycloak default: 30m) maxLifespan: "8h" # Keycloak default: 10h mfa: enabled: true policy: "required" # Require MFA for all users audit: retentionYears: 7 # Recommended: covers HIPAA 6-year documentation retention + buffer piiEncryption: true # SIEM integration: configure OpenTelemetry Collector exporters per your SIEM provider # otelCollector: # exporters: # syslog: { ... } # Customer-configurable; not pre-configured dataConnections: sslMode: "verify-full" # Enforce verified SSL readOnly: true # Default to read-only connections ``` ## Business Associate Agreement (BAA) **Critical for Data Insights deployments:** If the platform sends PHI to a cloud-based LLM API (e.g., for natural-language analysis or metadata enrichment), a BAA with the LLM provider is **required**. This is a HIPAA violation without a BAA. Self-hosted LLMs (configurable via `api_base`) eliminate this requirement. When using cloud-managed services with PHI, ensure BAAs are in place: | Service | BAA Required With | Notes | | -------------------------- | ------------------------------- | ----------------------------------------------------------------------- | | Cloud SQL / RDS / Azure DB | Cloud provider | Covers PHI stored in managed databases | | Memorystore / ElastiCache | Cloud provider | If PHI is cached in Redis | | GCS / S3 / Azure Blob | Cloud provider | If PHI-derived artifacts are stored | | Secrets backend | Infisical or GCP Secret Manager | If using hosted secrets management | | LLM providers | LLM vendor | **Required** if PHI is sent to LLM APIs for analysis. See warning above | ## Breach Notification HIPAA requires notification to affected individuals within 60 calendar days of **discovery** of the breach (45 CFR 164.404(b)). Additionally, notification to the Secretary of HHS is required (45 CFR 164.408), and notification to prominent media outlets may be required for breaches affecting more than 500 residents of a state (45 CFR 164.406). The platform supports breach detection through: * **Audit log analysis**: Identify unauthorized access patterns via audit log queries * **SIEM integration**: Real-time forwarding of security events for anomaly detection * **OpenTelemetry alerting**: Configure alerts for unusual access patterns (high volume queries, off-hours access, privilege escalation) ## Next Steps Review GDPR controls that complement HIPAA requirements. See the full SOC 2 Trust Service Criteria mapping. Understand data classification levels for PHI handling. Configure MFA and session management for HIPAA compliance. # SOC 2 Controls Source: https://docs.emergence.ai/security/compliance/soc2 SOC 2 Trust Service Criteria mapping for CRAFT, covering security, availability, processing integrity, confidentiality, and privacy. # SOC 2 Controls Mapping This page maps CRAFT's security controls to the **SOC 2 Trust Service Criteria (TSC) 2017** (revised 2022) defined by the AICPA. CRAFT is the core platform powering the Emergence product suite. The mapping covers all five trust service categories: Security, Availability, Processing Integrity, Confidentiality, and Privacy. This mapping documents the platform's built-in controls. Organizations pursuing SOC 2 Type II certification must also implement organizational controls (policies, procedures, training) and engage a qualified auditor. ## Security (Common Criteria) Security controls form the foundation of all SOC 2 categories. ### CC6: Logical and Physical Access Controls | Criteria | Control | Platform Implementation | | -------- | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CC6.1 | Logical access security | Keycloak multi-realm JWT authentication with OIDC/PKCE | | CC6.2 | User registration and authorization | OpenFGA ReBAC with role-based permissions (owner, admin, member, developer, operator, viewer) | | CC6.3 | Access removal | Keycloak user deactivation + OpenFGA relation cleanup; SCIM 2.0 deprovisioning from IdP | | CC6.6 | Boundary protection | Kubernetes NetworkPolicies, ingress TLS termination, SSRF protection | | CC6.7 | Data transmission protection | TLS 1.3 for external traffic, TLS 1.2 minimum for internal (due to managed service compatibility -- Cloud SQL, Redis, Keycloak). mTLS available via service mesh (Istio, Linkerd) | | CC6.8 | Unauthorized software prevention | Container-based deployment, immutable images, no SSH access to pods | ### CC7: System Operations | Criteria | Control | Platform Implementation | | -------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | CC7.1 | Detection of vulnerabilities | Container image scanning in CI/CD, dependency pinning with `==` | | CC7.2 | Anomaly detection | OpenTelemetry traces/metrics/logs with Grafana LGTM alerting | | CC7.3 | Security incident evaluation | Immutable audit logs with SIEM integration via OpenTelemetry Collector (customer-configurable exporters) | | CC7.4 | Incident response | Audit trail with trace\_id correlation for investigation; configurable alerting thresholds for detection. Organizations must maintain a documented IR plan with roles and escalation paths | ### CC8: Change Management | Criteria | Control | Platform Implementation | | -------- | ------------------------ | ---------------------------------------------------------------------- | | CC8.1 | Change authorization | GitHub PR reviews, changeset-based versioning, CI/CD gates | | CC8.2 | Infrastructure changes | Terraform IaC with plan/apply workflow, ArgoCD GitOps | | CC8.3 | Configuration management | Helm values files, immutable Alembic migrations, version-driven builds | ## Availability | Criteria | Control | Platform Implementation | | -------- | ------------------------- | ----------------------------------------------------------------------------------- | | A1.1 | Processing capacity | Kubernetes autoscaling (GKE Autopilot, HPA), Redis connection pooling | | A1.2 | Recovery from disruptions | Multi-database backup strategy, point-in-time recovery, disaster recovery checklist | | A1.3 | Environmental protections | Cloud provider infrastructure (GKE, EKS, AKS) with multi-zone availability | ### Backup and Recovery Controls | Control | Implementation | | ---------------------- | ----------------------------------------------------------------------------------------------- | | Automated backups | Cloud-managed daily backups with configurable retention (30 days default) | | Point-in-time recovery | WAL archiving for PostgreSQL PITR | | Backup testing | Regular restore validation as part of DR procedures (frequency defined in organization DR plan) | | Startup order | Documented service dependency DAG for orderly recovery | ## Processing Integrity | Criteria | Control | Platform Implementation | | -------- | --------------------- | ---------------------------------------------------------------------------- | | PI1.1 | Processing accuracy | SQLAlchemy ORM with strict typing, Pydantic request/response validation | | PI1.2 | Input validation | FastAPI automatic request validation, search query injection prevention | | PI1.3 | Processing monitoring | Prefect workflow orchestration with status tracking, health check monitoring | | PI1.4 | Output completeness | Auto-generated SDKs ensure API contract compliance, E2E test coverage | ### Data Integrity Controls | Control | Implementation | | -------------------- | -------------------------------------------------------------------------------------- | | Schema enforcement | SQLAlchemy 2.0+ with strict type annotations, Alembic migration validation | | API contract testing | Auto-generated Python/TypeScript SDKs from OpenAPI specs, E2E tests via generated SDKs | | Query safety | `plainto_tsquery()` for user input, parameterized queries, no string interpolation | | Webhook integrity | HMAC-SHA256 signed payloads with unique delivery IDs (planned -- not yet released) | ## Confidentiality | Criteria | Control | Platform Implementation | | -------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | C1.1 | Confidential information identification | Four-level data classification (Public, Internal, Confidential, Restricted). Query result sensitivity inherits from the underlying data source; organizations must classify their data connections appropriately | | C1.2 | Confidential information disposal | Crypto-shredding for RTBF, configurable retention policies, backup rotation | Query result sensitivity inherits from the underlying data source. Healthcare data may be Restricted, while business analytics may be Confidential. Organizations must classify their data connections appropriately. See [Data Classification](/security/data-classification) for the full taxonomy. ### Encryption Controls | Control | Implementation | | --------------------- | -------------------------------------------------------------------------------------------- | | Encryption at rest | AES-256 via cloud-managed encryption, CMEK support | | Encryption in transit | TLS 1.3 (external), TLS 1.2 minimum (internal). mTLS available via service mesh | | Key management | Cloud KMS or provider-managed keys, per-principal keys for audit PII | | Secrets management | Platform Secrets API (Infisical or ESO + GCP Secret Manager), never stored in code or config | ## Privacy | Criteria | Control | Platform Implementation | | -------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | P1.1 | Privacy notice | Data processing purposes documented per data connection | | P2.1 | Choice and consent | Configurable per-tenant data processing settings | | P3.1 | Personal information collection | Minimal PII collection (email, IP for audit); data minimization applied | | P4.1 | Use, retention, and disposal | Configurable retention per data type, automated cleanup | | P5.1 | Access to personal information | Audit log export, data portability APIs (JSON/CSV) | | P6.1 | Disclosure to third parties | No user data shared by default. LLM-powered features (when enabled) send user queries and schema context to the configured LLM provider; DPA/BAA required. Cloud provider DPAs for managed services. Self-hosted LLM option eliminates third-party data transfer | | P7.1 | Quality of personal information | User profile sync via SCIM, Keycloak-managed identity | | P8.1 | Complaints handling | Admin audit trail for RTBF request tracking | ## Access Review Controls The platform supports periodic access reviews for SOC 2 compliance: Administrators can review all RBAC grants per tenant using the Governance API. The review includes role assignments, last login timestamps, and permission inheritance paths. Automated detection of agent registrations whose publisher account has been deactivated. Orphaned resources are flagged for review or ownership transfer. Entities with no updates and consecutive failing health checks for a configurable number of days are flagged for review. Stale entities may indicate abandoned resources or decommissioned services. When users are deactivated in the enterprise IdP, SCIM 2.0 sync automatically removes their Keycloak realm access and cleans up OpenFGA relations. ## Evidence Collection For SOC 2 audit evidence, the platform provides: | Evidence | Source | Export Format | | ------------------- | -------------------------------- | ----------------- | | Access logs | Audit log API | JSON, CSV | | Change history | Git commits, ArgoCD sync history | Git log, API | | Configuration state | Helm values, Terraform state | YAML, JSON | | Vulnerability scans | CI/CD pipeline artifacts | SARIF, JSON | | Backup verification | Cloud provider backup logs | Provider-specific | | Incident records | Alert history, trace correlation | JSON | ## Next Steps Learn about GDPR-specific controls and Right to Be Forgotten. Review HIPAA compliance considerations for healthcare deployments. Deep dive into the authentication architecture. Review data classification levels and encryption standards. # Data Classification Source: https://docs.emergence.ai/security/data-classification Data classification levels, encryption at rest and in transit, and data handling requirements across CRAFT. # Data Classification CRAFT classifies all data processed by the platform into defined sensitivity levels with corresponding handling requirements. This page covers the classification taxonomy, encryption standards, and data handling policies. ## Classification Levels **Level 0 -- Public** Data intended for public consumption. No access control required. | Attribute | Value | | ------------------------- | ------------------------------------------------------ | | **Access** | No authentication required | | **Encryption at rest** | Inherits from platform default (cloud-managed AES-256) | | **Encryption in transit** | Recommended (TLS) | | **Retention** | Per content policy | **Examples:** Public Agent Cards, public documentation, published API specifications, marketplace listing descriptions. **Level 1 -- Internal** Data intended for internal use within an organization. Accessible to all authenticated members. | Attribute | Value | | ------------------------- | ---------------------------------------------- | | **Access** | Authenticated users with `can_read` permission | | **Encryption at rest** | Required | | **Encryption in transit** | Required (TLS 1.2 minimum) | | **Retention** | Per organization policy | **Examples:** Agent metadata, project configurations, workflow definitions, health check results, non-sensitive audit log entries. **Level 2 -- Confidential** Sensitive business data requiring restricted access. Disclosure could cause significant business impact. | Attribute | Value | | ------------------------- | -------------------------------------------------------------- | | **Access** | Role-based, minimum `developer` or `operator` role | | **Encryption at rest** | Required (AES-256) | | **Encryption in transit** | Required (TLS 1.2 minimum; TLS 1.3 where supported by service) | | **Retention** | Per data governance policy, with right-to-erasure support | **Examples:** Customer database contents accessed via data connections, query results, data profiling output, enrichment metadata, analytics insights. **Level 3 -- Restricted** Highly sensitive data. Unauthorized disclosure could cause severe business, legal, or regulatory impact. | Attribute | Value | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Access** | `admin` or `owner` role, with audit logging | | **Encryption at rest** | Required (AES-256, CMEK where available) | | **Encryption in transit** | Required (TLS 1.2 minimum for all traffic). mTLS for internal services requires service mesh deployment (e.g., Istio, Linkerd) -- not included by default | | **Retention** | Regulatory minimum, with crypto-shredding for erasure | **Examples:** Data connection credentials, API secrets, Keycloak realm configurations, HMAC webhook secrets, user PII (email, IP address), audit log actor identities. ## Encryption at Rest All persistent data is encrypted at rest using industry-standard encryption: | Component | Encryption Method | Key Management | | ------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | **PostgreSQL** | Cloud-managed encryption (AES-256) | Google-managed or CMEK (Cloud SQL), AWS KMS (RDS), Azure-managed (Flexible Server) | | **Redis** | Cloud-managed encryption where available | Provider-managed keys | | **Secrets backend** | Application-level secrets: Infisical (envelope encryption) or GCP Secret Manager (KMS-managed) | Backend-specific key hierarchy | | **Object storage** | Server-side encryption (AES-256) | Provider-managed or CMEK | | **Webhook secrets** | Envelope encryption | Per-subscription encryption keys | | **Audit log PII** | Per-principal encryption keys | Enables crypto-shredding for RTBF | ### Customer-Managed Encryption Keys (CMEK) For organizations with strict key management requirements, the platform supports CMEK on cloud-managed services: Use Cloud KMS keys with Cloud SQL and GCS. Configure via Terraform: ```hcl theme={null} resource "google_sql_database_instance" "main" { encryption_key_name = google_kms_crypto_key.database.id } ``` Use AWS KMS keys with RDS and S3. Configure via Terraform with `kms_key_id` parameters. Use Azure Key Vault keys with Azure Database for PostgreSQL. Configure via Terraform with customer-managed key references. ## Encryption in Transit All network communication is encrypted using TLS: | Channel | Minimum TLS Version | Notes | | --------------------------- | ------------------- | -------------------------------------------------- | | External API traffic | TLS 1.3 | Mandatory for all client-to-platform communication | | Internal service-to-service | TLS 1.2 | TLS 1.3 recommended | | Database connections | TLS 1.2 | `sslmode=require` or higher | | Redis connections | TLS 1.2 | In-transit encryption enabled | | Keycloak to IdP | TLS 1.2 | Required for OIDC and SAML flows | | Webhook deliveries | TLS 1.2 | HTTPS required in production environments | ### Mutual TLS (mTLS) For zero-trust Kubernetes deployments, the platform supports mTLS for internal service communication via a service mesh (Istio, Linkerd). ## Data Handling by Component | Component | Data Handled | Classification | Special Handling | | ------------------- | ------------------------------------------------- | ----------------------- | --------------------------------------------------------------------------- | | **Governance** | Organizations, users, roles | Internal - Restricted | PII in audit logs encrypted per-principal | | **Assets** | Artifacts, data connections, files, models | Internal - Restricted | Credentials stored via platform Secrets API (Infisical or ESO), never in DB | | **Utils** | Data catalog, scheduling, context packs, memories | Internal - Confidential | Memory content encrypted at rest; context pack metadata access-controlled | | **Data Insights** | Query results, SQL, visualizations | Confidential | Customer data accessed read-only via connections | | **Data Governance** | Profiles, metadata, DQ rules | Confidential | Metadata derived from customer data | | **Keycloak** | User identities, sessions | Restricted | Realm isolation, password hashing | | **OpenFGA** | Authorization tuples | Internal | No PII, relationship data only | ## Data Residency For organizations with data sovereignty requirements: * **Region labels**: Each tenant is tagged with a data residency region (EU, US, APAC, custom) * **Geo-fenced storage**: Database partitioning by region ensures data stays within designated boundaries * **Regional endpoints**: Optional per-region API routing (e.g., `eu.platform.example.com`) * **Cross-region search**: Federated queries can span regions with explicit opt-in; results indicate source region ## Data Retention | Data Type | Default Retention | Regulatory Override | | ------------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Audit logs | 1 year | Extended retention where required by specific regulation (e.g., financial record-keeping). Note: HIPAA's 6-year retention (45 CFR 164.530(j)(2)) applies to policies and documentation, not specifically to audit logs | | Operational logs | 90 days | -- | | Session data | 30 days after last activity | -- | | Webhook delivery logs | 30 days | -- | | Data profiling results | Organization-defined (recommend aligning with data connection retention) | Per GDPR RTBF; storage limitation principle requires justification for extended retention | | Event bus (Redis Streams) | 7 days | -- | ## Next Steps Learn about GDPR data handling, Right to Be Forgotten, and audit requirements. Review TLS configuration, SSRF protection, and network policies. See how data classification maps to SOC 2 Trust Service Criteria. Understand backup encryption and retention for disaster recovery. # Network Security Source: https://docs.emergence.ai/security/network-security Network policies, SSRF protection, TLS configuration, and network segmentation for CRAFT. # Network Security CRAFT implements defense-in-depth network security including TLS encryption, SSRF protection, Kubernetes network policies, and DNS security. This page covers the network security architecture and configuration. ## TLS Configuration All external and internal communication is encrypted with TLS: | Channel | Protocol | Minimum Version | Certificate Management | | --------------------- | -------- | --------------- | ----------------------------------------------------------------- | | Client to API gateway | HTTPS | TLS 1.3 | Managed certificates (Let's Encrypt, cloud-managed, or custom CA) | | Service to service | HTTPS | TLS 1.2 | Internal CA or service mesh | | Service to PostgreSQL | TLS | TLS 1.2 | Cloud-managed or custom CA | | Service to Redis | TLS | TLS 1.2 | Cloud-managed | | Keycloak to IdP | HTTPS | TLS 1.2 | IdP-managed certificates | | Webhook deliveries | HTTPS | TLS 1.2 | Target server certificates | TLS 1.0 and 1.1 are not supported. The platform enforces TLS 1.2 as a minimum and recommends TLS 1.3 for all external connections. ## SSRF Protection All outbound HTTP requests are protected against Server-Side Request Forgery (SSRF) attacks. This includes JWS `jku` fetch, health checks, webhook delivery (planned), and crawler operations. ### Protection Layers | Protection | Implementation | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Private IP blocking** | Resolved IPs validated against RFC 1918 (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), RFC 4193 (fc00::/7), link-local (169.254.0.0/16, fe80::/10), and loopback (127.0.0.0/8, ::1) | | **DNS pinning** | DNS resolution performed before connection; resolved IP pinned to prevent TOCTOU rebinding attacks | | **HTTPS enforcement** | `jku` URLs must use HTTPS; webhook targets must use HTTPS in production (when implemented) | | **Redirect limiting** | At most 2 redirects followed, with IP re-validation after each redirect | | **Domain allowlists** | Configurable per feature (e.g., `jku` domains, crawl domains) | | **Response size limits** | Crawler responses limited to 1 MB | | **Timeout enforcement** | `jku` fetch: 5 seconds; health checks: 10 seconds (configurable) | ### Per-Context Behavior Different outbound request types apply SSRF rules differently: | Context | Private IPs | SSRF Check | Notes | | ---------------- | ------------------------ | ---------- | ----------------------------------------------------------------------- | | Health checks | Skip with UNKNOWN status | Yes | Cards with private-IP endpoints are accepted; health checks are skipped | | JWS `jku` fetch | Reject | Yes | HTTPS required; domain allowlist applies | | Crawler | Reject | Yes | Admin-configured domains only | | Webhook delivery | Reject | Yes | HTTPS required in production; fire-and-forget | ## Kubernetes Network Policies The platform uses Kubernetes NetworkPolicies to restrict pod-to-pod communication: ```yaml theme={null} # Example: Governance service network policy apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: governance-netpol spec: podSelector: matchLabels: app: governance policyTypes: - Ingress - Egress ingress: - from: - podSelector: matchLabels: app: assets - podSelector: matchLabels: app: utils ports: - port: 8000 egress: - to: - podSelector: matchLabels: app: keycloak ports: - port: 8080 - to: - podSelector: matchLabels: app: openfga ports: - port: 8080 - to: - podSelector: matchLabels: app: postgresql ports: - port: 5432 ``` ### Network Segmentation The platform services are segmented into security zones: Receives traffic from external clients via the ingress controller or API gateway. Only the API endpoints and Keycloak login pages are exposed externally. Contains the platform services (Governance, Assets, Utils) and solution services (Data Insights, Data Governance). Services communicate via internal cluster networking with TLS. Contains PostgreSQL, Redis, and other data stores. Only accessible from the Application Zone. No direct external access is permitted. Contains Keycloak and OpenFGA. Keycloak is partially exposed (login pages) while OpenFGA is internal-only. ## Ingress Security ### Ingress Controller Configuration The platform's ingress controller (NGINX or cloud-managed) enforces: | Setting | Value | Purpose | | ------------------------- | ------------------- | ------------------------------------------------------------------------------------------- | | `ssl-redirect` | `true` | Force HTTPS for all connections | | `hsts-max-age` | `31536000` | HTTP Strict Transport Security (1 year) | | `hsts-include-subdomains` | `true` | Apply HSTS to all subdomains | | `proxy-body-size` | `50m` | Limit request body size (matches skill package upload limit) | | `rate-limiting` | Customer-configured | Per-route rate limiting at the ingress level (must be configured by deploying organization) | ### Web Application Firewall (WAF) WAF protection is recommended for all production deployments, especially those processing PHI, financial data, or sensitive enterprise data: * **GCP**: Cloud Armor security policies with OWASP Top 10 managed rules * **AWS**: AWS WAF with AWS Managed Rules (Core Rule Set, Known Bad Inputs) * **Azure**: Azure Front Door or Application Gateway WAF with OWASP 3.2 ruleset Deployments without WAF should document this as an accepted risk in their security posture. At minimum, enable the cloud provider's managed OWASP Top 10 rule group and IP reputation-based blocking. ## DNS Security | Protection | Description | | -------------------- | ----------------------------------------------------------- | | **DNSSEC** | Recommended for external DNS zones to prevent DNS spoofing | | **Internal DNS** | Kubernetes CoreDNS for service discovery within the cluster | | **DNS pinning** | SSRF protection pins resolved IPs to prevent DNS rebinding | | **TXT verification** | Agent domain verification via DNS TXT records | ## Service Mesh (Optional) For zero-trust deployments, the platform supports service mesh integration: * Automatic mTLS between all pods * Authorization policies for fine-grained access control * Traffic management and observability * PeerAuthentication for strict mTLS enforcement * Automatic mTLS with minimal configuration * Lightweight proxy with low resource overhead * Service profiles for per-route reliability settings ## Webhook Security When released, outbound webhook payloads will include security headers for integrity verification: | Header | Description | | --------------------- | ---------------------------------------- | | `X-Webhook-Id` | Unique delivery ID for idempotency | | `X-Webhook-Timestamp` | ISO 8601 timestamp for replay protection | | `X-Webhook-Signature` | HMAC-SHA256 signature of the payload | Receivers should: 1. Verify the HMAC-SHA256 signature using the shared secret 2. Reject payloads older than 5 minutes (replay protection) 3. Use the delivery ID for idempotent processing Webhook secrets support rotation with a 24-hour overlap period. During rotation, both the old and new secrets are valid for signature verification. ## Troubleshooting Check Kubernetes NetworkPolicies. Verify the source pod has egress rules allowing traffic to the destination pod on the correct port. Use `kubectl describe networkpolicy` to inspect active policies. Verify the target URL resolves to a public IP address. If the target is behind a private DNS, add the domain to the appropriate allowlist. Health checks for agents with private IPs return UNKNOWN status by design. Check certificate validity and chain completeness. Verify the server supports TLS 1.2 or higher. For internal services, ensure the internal CA certificate is trusted by all pods. ## Next Steps Review the authentication architecture and token management. Understand encryption requirements by data classification level. Configure network-related Helm values for your deployment. Infrastructure-level networking configuration for Kubernetes. # CRAFT Security Overview Source: https://docs.emergence.ai/security/overview CRAFT security architecture: Keycloak authentication, OpenFGA authorization, secrets management (ESO or Infisical), multi-tenant isolation, and OpenTelemetry observability. # Security Overview Security is embedded into every layer of CRAFT. The security model is built on four pillars: **authentication** (who you are), **authorization** (what you can do), **secrets management** (protecting credentials), and **multi-tenant isolation** (keeping organizations separate). ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR EXT["External Request"] TLS["TLS / Gateway"] AUTHN["Keycloak AuthN
JWT Validation"] AUTHZ["OpenFGA AuthZ
Permission Check"] HANDLER["Request Handler"] DB["PostgreSQL
org_id-scoped queries"] DATA["Data"] EXT --> TLS --> AUTHN --> AUTHZ --> HANDLER --> DB --> DATA style EXT fill:#CC79A7,stroke:#555555,color:#000 style TLS fill:#0072B2,stroke:#555555,color:#fff style AUTHN fill:#D55E00,stroke:#555555,color:#000 style AUTHZ fill:#D55E00,stroke:#555555,color:#000 style HANDLER fill:#E69F00,stroke:#555555,color:#000 style DB fill:#009E73,stroke:#555555,color:#000 style DATA fill:#009E73,stroke:#555555,color:#000 ``` ## Security Architecture **Technology:** Keycloak with multi-realm JWT and OIDC/PKCE Keycloak provides the identity layer for the entire platform. Each organization is a separate Keycloak **realm**, providing complete identity isolation between tenants. | Feature | Details | | ---------------------- | ------------------------------------------------------------- | | **Protocol** | OpenID Connect (OIDC) with PKCE for browser clients | | **Token format** | JWT containing `org_id` (from realm), groups, and user claims | | **Multi-realm** | Each organization = one Keycloak realm (realm ID = org ID) | | **SSO support** | Enterprise SSO via SAML or OIDC identity provider federation | | **Session management** | Token refresh, revocation, and session timeout policies | The `project_id` is never stored in the JWT. It is passed via the `X-Project-ID` HTTP header or as a path parameter, allowing users to switch project context without re-authenticating. **Technology:** OpenFGA with Relationship-Based Access Control (ReBAC) OpenFGA provides fine-grained authorization based on relationships between entities rather than static role assignments. This enables permission inheritance and computed permissions. ### Roles | Role | Scope | Description | | ----------- | ---------------------- | ---------------------------------------------------- | | `owner` | Organization | Full control over the organization and all resources | | `admin` | Organization / Project | Administrative access to manage users and settings | | `member` | Project | Standard access to project resources | | `developer` | Project | Create and modify agents, connections, and workflows | | `operator` | Project | Deploy, schedule, and monitor resources | | `viewer` | Project | Read-only access to all resources | ### Computed Permissions Permissions are **computed at query time** from role relationships, not statically assigned: | Permission | Derived From | | -------------------- | --------------------------------- | | `can_read` | All roles (viewer and above) | | `can_write` | member, developer, admin, owner | | `can_delete` | admin, owner | | `can_execute` | developer, operator, admin, owner | | `can_manage_secrets` | admin, owner | ### Permission Inheritance Permissions flow downward through the resource hierarchy: ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR Org["Organization"] --> Proj["Project"] --> Res["Resource
(Agent, Data Connection, etc.)"] classDef level fill:#0072B2,stroke:#555555,color:#fff class Org,Proj,Res level ``` A user who is an `admin` of an organization automatically has admin-level computed permissions on all projects and resources within it. Solutions must never implement their own permission checks. All authorization is delegated to the Governance service via the `require_permission()` decorator or SDK calls. If Governance is unavailable, all downstream services return 403. The platform supports two secrets backends, choose based on your deployment environment. See [Secrets Management](/security/secrets-management) for a full comparison. | Layer | Options | Scope | Examples | | ------------------------- | ---------------------------------- | ---------------------------------------- | ------------------------------------------------------ | | **Application (cloud)** | ESO + GCP Secret Manager | Platform services, synced to K8s Secrets | Data connection credentials, API keys | | **Application (on-prem)** | Infisical | Platform services | Data connection credentials, API keys, encryption keys | | **Infrastructure** | GCP Secret Manager (via Terraform) | Terraform-managed resources | Database passwords, service account keys | **Key properties:** * Secrets are never exposed in API responses * Per-organization namespace isolation (Infisical) or IAM-scoped access (GCP SM) * Credentials are injected at runtime only when needed * Auto-rotation supported: ESO + Stakater Reloader (cloud), Infisical rotation policies (self-hosted) Never commit secrets or credentials to configuration files or source code. Use the platform Secrets API for application secrets and GCP Secret Manager or your cloud provider's equivalent for infrastructure secrets. Multi-tenancy provides complete isolation between organizations at every layer of the stack: | Layer | Isolation Mechanism | | ------------------ | ------------------------------------------------------------------------ | | **Authentication** | Separate Keycloak realm per organization | | **Authorization** | Per-organization OpenFGA relationship tuples | | **Data** | All database queries filtered by `org_id` from JWT | | **Secrets** | Per-organization isolation (Infisical namespaces or GCP SM IAM bindings) | | **Storage** | Tenant-scoped object storage paths | ### Security Invariants These rules are enforced across all platform services: 1. The `org_id` **always** comes from the authenticated JWT token, never from request parameters 2. **List operations** filter by both `org_id` (from JWT) AND `project_id` (from header) at the database level 3. **Write operations** stamp the `org_id` from the JWT onto new resources 4. **No cross-service foreign keys** exist between databases 5. A missing org\_id filter on a list query is treated as a **security vulnerability** Platform services run entirely within your infrastructure. External data egress depends on the integrations and model providers you configure -- for example, LLM calls via LiteLLM are sent to the provider you choose (Gemini, GPT, Claude, or a self-hosted model). You control what data leaves your environment. ## Dual Authorization Pattern The platform enforces authorization through a dual pattern that combines database filtering with OpenFGA checks: | Operation Type | Database Filter | OpenFGA Check | | ------------------------ | -------------------------------------- | ------------------------------------------------- | | **List operations** | Filter by `auth.org_id` + `project_id` | `require_permission()` for project access | | **Get-by-ID operations** | Not needed | `require_permission()` handles authorization | | **Create operations** | Stamp `auth.org_id` on resource | `require_permission()` for project write access | | **Delete operations** | Not needed | `require_permission()` for resource delete access | This dual pattern prevents two classes of vulnerabilities: (1) listing resources across organizations (database filter) and (2) accessing specific resources without permission (OpenFGA check). ## Authentication Flow The user authenticates via OIDC/PKCE flow against their organization's Keycloak realm. The browser receives a JWT token containing `org_id` and group memberships. The client includes the JWT in the `Authorization: Bearer ` header and the active project in the `X-Project-ID` header. The Governance service validates the JWT signature, expiry, and issuer. It extracts `org_id` from the realm claim. OpenFGA evaluates the user's relationship to the requested resource. Computed permissions are resolved at query time based on role inheritance. If authorized, the request proceeds with the authenticated context. Database queries are scoped to `org_id` and `project_id`. ## Network Security | Control | Implementation | | ---------------------- | -------------------------------------------------------------------------------------------------------------- | | **SSRF protection** | Webhook and data connection endpoints validate target URLs against allowlists | | **HMAC signatures** | Webhook payloads will be signed with HMAC for integrity verification (planned -- not yet released) | | **TLS** | All inter-service communication uses TLS in production | | **Private networking** | Cloud deployments use private VPC networking; self-managed clusters should use equivalent network segmentation | | **Firewall rules** | Terraform-managed ingress/egress rules | ## Observability and Audit The platform provides comprehensive observability for security monitoring: | Component | Technology | Purpose | | -------------- | --------------------- | ----------------------------------------------- | | **Traces** | OpenTelemetry + Tempo | Distributed request tracing across services | | **Metrics** | OpenTelemetry + Mimir | Service health, latency, and throughput metrics | | **Logs** | OpenTelemetry + Loki | Structured logging with correlation IDs | | **Dashboards** | Grafana | Visualization and alerting for all telemetry | All telemetry is exported via the **OTLP** (OpenTelemetry Protocol) standard -- a vendor-neutral protocol compatible with any OTLP-capable backend. You choose your observability stack; the Grafana LGTM bundle (Loki, Grafana, Tempo, Mimir) is included by default but never required. ## Security Best Practices * Enable TLS for all ingress and inter-service communication * Configure Keycloak password policies and brute-force detection * Set up regular secret rotation (Infisical rotation policies or ESO + Stakater Reloader) * Monitor OpenFGA permission check latency and error rates * Review audit logs for unauthorized access attempts * Use private networking for database and cache connections * Never implement permission checks directly -- always use `require_permission()` * Always use `auth.org_id` from JWT, never from request parameters * Never store secrets in configuration files or environment variables in source control * Use the Governance SDK for all authentication and authorization operations * Validate webhook signatures before processing events (when webhook feature is released) * Use parameterized queries to prevent SQL injection (enforced by SQLAlchemy ORM) ## Next Steps Deep dive into Keycloak configuration and OIDC flows. Understand data classification and handling policies. Learn about network-level security controls and firewall configuration. Step-by-step guide for configuring roles and permissions. # Secrets Management Source: https://docs.emergence.ai/security/secrets-management Overview of secrets management options for CRAFT, including ESO + GCP Secret Manager and Infisical as equal deployment alternatives. # Secrets Management CRAFT supports two secrets management backends. Both operate through the same **Governance Secrets API** abstraction, applications access secrets through a unified interface regardless of which backend is deployed. ## Secrets API Abstraction The Governance service exposes a Secrets API that proxies secret read/write operations to the configured backend. This decouples application code from the underlying secrets provider. ```text theme={null} Application → Governance Secrets API → [ ESO + GCP Secret Manager | Infisical ] ``` ## Option A: ESO + GCP Secret Manager **Recommended for cloud deployments on GCP.** Uses the [External Secrets Operator](https://external-secrets.io/) (ESO) to sync secrets from GCP Secret Manager into Kubernetes Secrets, with [Stakater Reloader](https://github.com/stakater/Reloader) for automatic pod restarts on secret changes. ### Architecture ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart LR TF["Terraform
(GKE module)"] -->|provisions| SM["GCP Secret Manager"] SM -->|synced by| ESO["External Secrets
Operator"] ESO -->|creates| KS["Kubernetes Secrets"] KS -->|mounted in| Pod["Application Pod"] RLD["Stakater Reloader"] -->|watches| KS RLD -->|restarts| Pod classDef infra fill:#0072B2,stroke:#555555,color:#fff classDef process fill:#E69F00,stroke:#555555,color:#000 classDef data fill:#009E73,stroke:#555555,color:#000 classDef app fill:#56B4E9,stroke:#555555,color:#000 class TF,ESO,RLD infra class SM process class KS data class Pod app ``` ### Components | Component | Purpose | | ----------------------------- | ---------------------------------------------------------------------------- | | **GCP Secret Manager** | Cloud-native secret store, encrypted at rest, audit-logged, IAM-controlled | | **External Secrets Operator** | Kubernetes operator that syncs GCP SM secrets into K8s Secrets on a schedule | | **ClusterSecretStore** | Cluster-scoped ESO resource that configures the GCP SM connection | | **ExternalSecret** | Namespace-scoped resource mapping a GCP SM secret to a K8s Secret | | **Stakater Reloader** | Watches K8s Secrets for changes; triggers rolling restarts of annotated pods | ### Secret Naming Convention Secrets in GCP Secret Manager follow a `-` naming pattern. Specific names are environment-specific. ### Authentication ESO authenticates to GCP Secret Manager using [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity), no service account keys are stored in the cluster. ### Rotation When a secret value is updated in GCP Secret Manager: 1. ESO detects the change on its next sync interval (configurable, default 1h) 2. The K8s Secret is updated 3. Stakater Reloader detects the change (via annotation `reloader.stakater.com/auto: "true"`) and triggers a rolling restart For immediate rotation, manually trigger an ESO sync or restart the affected deployment. Secrets injected as environment variables require a pod restart to take effect, even after ESO syncs the updated value to the Kubernetes Secret. Stakater Reloader (included in reference infrastructure) automates this by triggering rolling restarts when annotated Secrets change. For security incidents, manually restart the affected deployment immediately after triggering an ESO sync. Consider reducing the default 1-hour sync interval for high-sensitivity secrets. ### Deployment See [Infrastructure Secrets](/deployment/infrastructure/secrets) for the step-by-step deployment guide. *** ## Option B: Infisical **Recommended for on-premises and cloud-agnostic deployments.** [Infisical](https://infisical.com/) provides application-level envelope encryption with a self-hostable server and native Kubernetes authentication. ### Architecture Infisical runs as a subchart within the em-runtime Helm release and stores secrets in its own PostgreSQL database (`infisical`). Services authenticate via Infisical's Kubernetes machine identity. ### Components | Component | Purpose | | ----------------------- | ----------------------------------------------------------------------- | | **Infisical server** | Self-hosted secret management platform (deployed as Helm subchart) | | **Infisical SDK** | Python/TypeScript SDK used by em-runtime services at startup | | **Machine Identity** | Kubernetes service account-based authentication (no static credentials) | | **Envelope encryption** | Secrets encrypted at rest with a per-secret encryption key hierarchy | ### Configuration Infisical is enabled by default in the em-runtime Helm chart: ```yaml theme={null} infisical: enabled: true resources: requests: cpu: "250m" memory: "512Mi" ``` To use an external Infisical instance instead of the subchart: ```yaml theme={null} infisical: enabled: false governance: extraEnvVars: - name: INFISICAL_URL value: "https://infisical.example.com" - name: INFISICAL_PROJECT_ID value: "your-project-id" ``` ### Rotation Infisical supports secret rotation via its dashboard or API. Service pods must be restarted to pick up new secret values (unlike ESO + Reloader which handles this automatically). Infisical requires manual pod restarts to pick up rotated secrets. For regulated deployments, configure Stakater Reloader (included in reference infrastructure) to watch Infisical-managed Kubernetes Secrets and trigger automated rolling restarts on change. Document the rotation procedure including verification steps. *** ## Comparison | Attribute | ESO + GCP Secret Manager | Infisical | | ---------------------- | ------------------------- | ---------------------------------- | | **Deployment model** | Cloud-native (GCP) | Self-hosted or cloud-agnostic | | **Cloud dependency** | Requires GCP | No cloud dependency | | **Encryption at rest** | GCP KMS (managed) | Envelope encryption (self-managed) | | **Auto-rotation** | Yes (Stakater Reloader) | Manual pod restart required | | **Audit logging** | GCP Cloud Audit Logs | Infisical audit logs | | **On-prem support** | No | Yes | | **Helm chart** | ESO Helm chart (separate) | Infisical subchart (included) | | **Auth mechanism** | GCP Workload Identity | Kubernetes machine identity | ## Choosing a Backend * **GKE (GCP cloud deployment)**: ESO + GCP Secret Manager is the default and is provisioned by Terraform * **On-premises or non-GCP cloud**: Infisical (included in Helm chart as subchart) * **Multi-cloud (EKS/AKS)**: Either, ESO supports AWS Secrets Manager and Azure Key Vault; Infisical is cloud-agnostic Both backends are supported and maintained. The choice depends on your deployment environment, not the platform version. ## Related Step-by-step guide to deploying ESO or Infisical in your cluster. How credentials are classified and protected within the platform. Helm values for enabling and configuring secrets backends. Network-level controls that protect secret access paths. # CRAFT Semiconductor Overview Source: https://docs.emergence.ai/semiconductor/overview Overview of the CRAFT semiconductor fab analytics platform for AI-assisted data analysis, workflow orchestration, and manufacturing insights. # Semiconductor Analytics The Semiconductor solution provides AI-assisted data analytics for semiconductor fabrication (fab) environments. It enables fab engineers to query manufacturing data using natural language, run automated analysis workflows, and gain insights into production processes, the AI agent generates and executes Python code on behalf of the engineer. ## Who Is It For Analyze manufacturing data, identify yield issues, and investigate process deviations using conversational AI instead of manual SQL queries. Monitor process parameters, detect drift, and correlate process variables with quality outcomes using AI-assisted analytics. Track defect rates, analyze wafer maps, and identify root causes of quality issues with automated data exploration. Access high-level dashboards and generate reports on fab performance, throughput, and equipment utilization. ## Key Capabilities Fab engineers interact with manufacturing data through a natural-language chat interface: * "What is the yield trend for product X over the past 30 days?" * "Which tools have the highest defect rates this week?" * "Compare process parameters between high-yield and low-yield lots" * "Show me wafer map defect patterns for the last 100 wafers" The system generates Python analysis code, executes it in the sandboxed Code Execution service using the platform's data-access layer, and returns results with interactive visualizations. Prefect-orchestrated workflows automate analysis and data management: * **Yield excursion detection**: Automated excursion detection with E1–E7 and parameter detectors, root cause analysis, and report generation (PDF, PPT, DOCX) * **Cross-product analysis**: Compare yield and excursion patterns across products * **Data ingestion**: Automated data extraction from fab systems (Optimal+) * **Feedback learning**: Daily policy refinement from user feedback The solution connects to standard semiconductor data sources: * Manufacturing Execution Systems (MES) * Equipment Data Collection (EDC) databases * Wafer test and sort databases * Inline metrology measurement databases * Defect inspection and review systems ## Architecture The Semiconductor solution follows the same architectural patterns as the other CRAFT solutions: | Component | Role | | ---------------------------- | -------------------------------------------------------------- | | **Conversational interface** | Natural-language queries against fab data | | **AI agents** | Pydantic AI agent with MCP tools for fab analytics | | **Code execution** | Sandboxed Python execution for data analysis and visualization | | **Workflow engine** | Prefect workflows for automated analysis pipelines | | **Platform integration** | Authentication, authorization, and data connections | ### Platform Integration The Semiconductor solution integrates with the shared CRAFT: * **Authentication**: Keycloak OIDC with organization-specific user management * **Authorization**: JWT-based access control with organization-scoped data queries * **Data connections**: Database connections managed via the data source connection system * **Multi-tenancy**: All data isolated per organization The Semiconductor solution operates as an independent platform with its own microservices (Backend API, AI Agent, MCP Server, Client Dashboard), shared library, and infrastructure. See the detailed architecture documentation for the full system design. ## Semiconductor-Specific Analytics | Analysis Type | Description | | ----------------------- | ---------------------------------------------------------------------------------------- | | **Yield analysis** | Lot-level and wafer-level yield calculation and trending | | **Excursion detection** | Automated detection of yield anomalies using E1–E7, WAT, WSP, and multivariate detectors | | **Process correlation** | Statistical correlation between process parameters and quality outcomes | | **Bin analysis** | Test bin distribution analysis and Pareto charts | | **Report generation** | Automated RCA and insight reports exportable as PDF, DOCX, and PPT | | **SPC monitoring** | Control chart generation with out-of-control detection | ## Data Model The solution understands standard semiconductor manufacturing data structures: | Entity | Description | | --------------------------- | ---------------------------------------------------------------------------------------------------------- | | **Products** | Semiconductor device families, characterized by technology node, foundry, and business unit | | **Lots** | Manufacturing batches with user-friendly IDs (e.g., `L2401-001`) for natural reference in chat and reports | | **Wafers** | Individual wafers within lots with WAT, WSD, and FTD measurements | | **Excursion Events** | Detected yield anomalies with detector type, severity, and affected lots/wafers | | **Reports** | Generated analysis reports (RCA, Insight) with sections, charts, and export support | | **Workflow Runs** | Execution records for Prefect workflows with inputs, status, and outputs | | **Policies** | Learned excursion detection rules with versioning from user feedback | | **Alerts** | Notifications triggered by excursion detection or workflow events | | **Data Source Connections** | Configured connections to external fab data systems |