> ## Documentation Index > Fetch the complete documentation index at: https://docs.emergence.ai/llms.txt > Use this file to discover all available pages before exploring further. # Data Connections > Manage external data source connections with secure credential storage, connection verification, and fine-grained access control. # Data Connections Data connections represent configured links to external data sources: databases, object storage, and file systems. They are managed by the **Assets** service (port 8000) and provide a secure, unified interface for solutions to discover and access external data without handling raw credentials. Connections can be scoped to a single project or to the entire organization (org-level). See [Org-level data connections](#org-level-data-connections) below. Credentials for data connections are stored securely via the Governance Secrets API (backed by Infisical or ESO + GCP Secret Manager). The connection record itself stores only metadata; credentials are never persisted in the Assets database. ## Supported Connection Types | Type | Status | Description | | ------------ | ----------- | --------------------------------------------------------------------------------------------------------------- | | `postgres` | Implemented | PostgreSQL databases | | `redshift` | Implemented | Amazon Redshift data warehouses | | `snowflake` | Implemented | Snowflake data warehouses (Basic + JWT/key-pair auth available; OAuth and SSO planned based on customer demand) | | `mysql` | Planned | MySQL databases | | `bigquery` | Planned | Google BigQuery datasets | | `databricks` | Planned | Databricks SQL warehouses | | `couchbase` | Planned | Couchbase NoSQL databases | | `oracle` | Planned | Oracle databases | | `hive` | Planned | Apache Hive metastore | | `mongodb` | Planned | MongoDB NoSQL databases | Database connections support: * Schema and table discovery during verification (note: Snowflake performs server-level discovery and does not enumerate schemas/tables ahead of time) * Selected table filtering * Connection pooling configuration | Type | Status | Description | | ------- | ----------- | --------------------------------- | | `s3` | Implemented | Amazon S3 or S3-compatible stores | | `gcs` | Implemented | Google Cloud Storage | | `minio` | Implemented | Self-hosted MinIO instances | Storage connections support: * Bucket and prefix configuration * Credential verification * Multiple authentication methods (access key, service account, etc.) | Type | Status | Description | | ----- | ----------- | ------------------------ | | `nfs` | Implemented | NFS-mounted file systems | | `smb` | Planned | SMB/CIFS network shares | File system connections support: * Path-based configuration * Permission validation ## Connection Lifecycle Query `GET /assets/data/types` to retrieve all connection types with their credential schemas, authentication methods, and example configurations. Use `POST /assets/data/verify` to test credentials without creating a connection. For databases, this also discovers available schemas and tables for selection. `POST /assets/data` creates the connection record, stores credentials as a secret via the Governance Secrets API, and sets up ownership permissions in OpenFGA. Solutions retrieve connection metadata via `GET /assets/data/{resource_uri}` and fetch decrypted credentials via `GET /assets/data/{resource_uri}/secret` when executing queries. `POST /assets/data/{resource_uri}/verify` tests that stored credentials are still valid and the data source is accessible. ## Resource URIs Data connections use **resource URIs** as identifiers instead of UUIDs. Resource URIs follow a structured format: ```yaml theme={null} data:{org_id}:{project_id}:{connection_name} ``` For example: `data:acme-corp:analytics-prod:customer-db` If a `resource_uri` is not provided during creation, one is auto-generated from the connection name. ## Org-level data connections Data connections can be scoped to a single project (the default) or to the entire organization. Org-level connections let multiple projects share the same configured data source without per-project duplication. **Creating an org-level connection:** omit the `X-Project-ID` header (or pass `project_id=null` in the request body where supported). The created connection has its `project_id` field unset and lives at the org boundary. **Resource URI shape:** org-level connections drop the project segment. ```yaml theme={null} data:{org_id}::{connection_name} # org-level (note the empty project segment) data:{org_id}:{project_id}:{connection_name} # project-scoped (default) ``` **Permission semantics:** an org-level connection's permissions are evaluated against the organization, not a project. The OpenFGA computed-permission graph extends `can_*` checks across the org so that: * Org owners can read, write, and delete org-level connections. * Members of any project under the organization can `read` and `execute` an org-level connection if their project role grants those permissions on org-shared resources. (See [Authorization](/platform/authorization) for the full chain.) * Cross-project visibility is the explicit point of org-level connections; standard project-scoped connections remain isolated. **When to use:** a database that powers analytics across all projects in the organization, a shared S3 bucket for raw data drops, or any connection where per-project credential management would create operational drag without a security benefit. ## API Reference All endpoints require the `X-Project-ID` header and a valid JWT token. Lists all available connection types with credential schemas. Returns each connection type with its category, description, supported authentication methods, required credential fields, and example configurations. Use this to build dynamic connection forms. Lists data connections in the current project with filtering and pagination. **Access:** `can_read` on the project. **Query Parameters:** * `page`, `limit`: Pagination * `search`: Search in name and description. Whitespace-only search terms return HTTP 400. `%` and `_` characters are treated as literal text (ILIKE-escaped) so a search for `customer_db` matches the literal string, not "customer" + any character + "db". * `connection_type`: Filter by type (postgres, s3, etc.) * `status`: Filter by status (PENDING, ACTIVE, ERROR, DISABLED) * `tags`: Comma-separated tag filter (AND logic) * `sort`: Sort specification (e.g., `name:asc,created_at:desc`) * `owner_id`: Filter by owner Verifies connection credentials before saving. Tests connectivity, validates permissions, and discovers available schemas and tables for databases. **Access:** `can_create_resources` on the project. Currently supports verification for: PostgreSQL, Redshift, S3, GCS, MinIO. Creates a new data connection with inline credentials. **Access:** `can_create_resources` on the project. **Side effects:** * Creates the data connection record in the database * Auto-generates `resource_uri` from name if not provided * Stores credentials as a secret via the Governance Secrets API * Creates an ownership tuple in OpenFGA ```json Request Body theme={null} { "name": "Customer Analytics DB", "connection_type": "postgres", "description": "Production read replica for analytics", "database_config": { "host": "pg-replica.example.com", "port": 5432, "database": "analytics", "schema": "public", "selected_tables": ["customers", "orders", "products"] }, "credentials": { "username": "analytics_reader", "password": "secret" }, "tags": ["production", "analytics"] } ``` Retrieves data connection details (metadata only, no credentials). **Access:** `can_read` on the data connection. Updates a data connection. The `resource_uri` cannot be changed. **Access:** `can_write` on the data connection. Credentials can be updated by providing new values. If credentials are omitted, existing credentials are preserved. Deletes a data connection and all associated resources. **Access:** `can_delete` on the data connection. **Side effects:** * Removes the data connection record * Deletes OpenFGA permission tuples * Deletes associated secrets from the Governance Secrets API Retrieves decrypted credential values for a data connection. **Access:** `can_execute` on the data connection. This elevated permission ensures only authorized automation (agents, workflows) can access raw credentials. Secrets are never cached and access is logged for audit purposes. Verifies that an existing connection's stored credentials are still valid. **Access:** `can_execute` on the data connection. ## Credential Storage Architecture Data connection credentials follow a layered storage model: ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%% flowchart TD A["Assets Service (port 8000)
Data connection record: metadata, config, status
Secret names: [credentials]"] G["Governance Service (port 8000)
Secrets API"] S["Secrets Backend
(Infisical or ESO + GCP Secret Manager)
Encrypted storage · Audit logging · OpenFGA access control"] A -->|"Governance SDK call
(forwarding user JWT)"| G G --> S classDef service fill:#E69F00,stroke:#555555,color:#000 classDef secrets fill:#009E73,stroke:#555555,color:#000 class A,G service class S secrets ``` The Assets service never stores or caches credentials. Every secret retrieval goes through the Governance Secrets API, which enforces its own permission checks and audit logging. ## Data Connection Permissions Data connections support the full set of resource-level permissions: | Permission | Who Can | Use Case | | --------------------- | -------------------------------------------------- | -------------------------------------- | | `can_read` | Viewers, operators, developers, owners + inherited | View connection metadata | | `can_write` | Operators, developers, owners + inherited | Update connection settings | | `can_delete` | Developers, owners + project admin/owner | Remove connections | | `can_execute` | Operators, developers, owners + inherited | Retrieve credentials, run verification | | `can_read_secrets` | Operators, developers, owners + inherited | View associated secrets | | `can_manage_secrets` | Developers, owners + inherited | Update credentials | | `can_read_metadata` | Operators, developers, owners + inherited | View data connection metadata | | `can_manage_metadata` | Developers, owners + inherited | Update metadata and tags | ## Next Steps Learn how agents use data connections to access external data. Understand how data connections are scoped to projects. Explore the permission model for data connections. Automate data connection workflows with scheduled tasks.