> ## Documentation Index
> Fetch the complete documentation index at: https://docs.emergence.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Data Enrichment
> How the Data Governance solution uses LLMs to enrich metadata with descriptions, classifications, and data quality rule suggestions.
# Metadata Enrichment
The metadata enrichment pipeline uses **LLMs** to automatically generate descriptions, classifications, and data quality rule suggestions for profiled data assets. This transforms raw schema information into rich, governed metadata that helps data teams understand and trust their data.
## What Gets Enriched
| Enrichment | Description |
| --------------------------- | ---------------------------------------------------------------------- |
| **Business description** | Human-readable description of the table's purpose and contents |
| **Domain classification** | Business domain the table belongs to (e.g., Sales, Finance, HR) |
| **Sensitivity level** | Data classification level (Public, Internal, Confidential, Restricted) |
| **Data steward suggestion** | Recommended owner based on domain and content |
| **Relationship summary** | Description of how the table relates to other tables |
| Enrichment | Description |
| ------------------------ | --------------------------------------------------------------- |
| **Business description** | What the column represents in business terms |
| **Data type assessment** | Whether the declared type matches actual content |
| **PII classification** | Whether the column contains personally identifiable information |
| **Format pattern** | Expected format (email, phone, date, currency, etc.) |
| **Valid value range** | Expected min/max or enumerated values |
| **DQ rule suggestions** | Recommended data quality rules for this column |
## Enrichment Pipeline
```mermaid theme={null}
%%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%%
graph TB
API["Data Governance API\n(trigger)"]
FLOW["Prefect Flow\n(enrich_metadata)"]
CTR["ConcurrentTaskRunner\n(parallel tables)"]
CTX["prepare_tables_and_schema()\n(full schema + profiling context)"]
SEM["asyncio.Semaphore\n(LLM API rate gate)"]
LLM["Vertex AI\n(gemini-3.5-flash)"]
VAL["Validation\n(PII cross-check,\nschema alignment)"]
DB[("datareadiness DB\n(enriched metadata)")]
subgraph Outputs["Generated Metadata"]
DESC["Business descriptions"]
PII["PII classifications"]
DQR["DQ rule suggestions"]
end
API --> FLOW --> CTR --> CTX --> SEM --> LLM --> VAL --> Outputs --> DB
classDef trigger fill:#56B4E9,stroke:#555555,color:#000
classDef orch fill:#E69F00,stroke:#555555,color:#000
classDef gate fill:#F0E442,stroke:#555555,color:#000
classDef llm fill:#CC79A7,stroke:#555555,color:#000
classDef result fill:#009E73,stroke:#555555,color:#000
classDef store fill:#0072B2,stroke:#555555,color:#fff
class API trigger
class FLOW,CTR,CTX orch
class SEM gate
class LLM,VAL llm
class DESC,PII,DQR result
class DB store
```
The enrichment workflow is orchestrated by Prefect and uses the OpenAI API for LLM-powered analysis:
The workflow gathers schema information and profiling results for the target tables. Schema context (all tables and relationships) is assembled using `prepare_tables_and_schema()` to give the LLM full database context.
Schema and profiling data are sent to the LLM with structured prompts. The LLM generates metadata enrichments based on:
* Column names and types
* Profiling statistics (completeness, uniqueness, patterns)
* Table relationships and foreign keys
* Cross-table context (schema\_info for understanding relationships)
Generated metadata is validated against the schema. PII classifications are cross-checked with column patterns and profiling data.
Enriched metadata is stored in the Data Governance database, linked to the corresponding data asset records.
## Schema Context for Better Enrichment
The enrichment pipeline provides full schema context to the LLM for higher-quality results:
```python theme={null}
from dq.utils.schema_helpers import prepare_tables_and_schema
# Gather schema context
schema_info, tables_info = await prepare_tables_and_schema(connection)
# schema_info: all tables in the database (for relationship understanding)
# tables_info: selected tables for enrichment
# Pass both to the LLM prompt
prompt = build_enrichment_prompt(
tables=tables_info,
schema_context=schema_info # LLM understands cross-table relationships
)
```
Providing full schema context allows the LLM to understand foreign key relationships and generate more accurate business descriptions. For example, it can identify that `customer_id` in the `orders` table references the `customers` table and generate appropriate descriptions.
## Concurrency Control
Enrichment workflows use the same hybrid concurrency model as profiling:
| Level | Mechanism | Control |
| -------------- | ---------------------- | --------------------------------------------- |
| **Flow-level** | `ConcurrentTaskRunner` | Max concurrent table enrichment tasks |
| **Task-level** | `asyncio.Semaphore` | Max concurrent LLM API calls within each task |
The task-level semaphore is particularly important for enrichment to avoid hitting LLM API rate limits:
```python theme={null}
@task
async def enrich_table(table_info, schema_context):
semaphore = asyncio.Semaphore(config.max_concurrent_llm_calls)
columns = table_info["columns"]
async def enrich_column(column):
async with semaphore:
return await llm_enrich(column, schema_context)
results = await asyncio.gather(*[enrich_column(c) for c in columns])
return results
```
## LLM Configuration
The enrichment pipeline uses LiteLLM and is provider-agnostic — defaults to
Vertex AI but works with any LiteLLM-supported provider:
| Setting | Source | Description |
| ---------------------------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `VERTEXAI_PROJECT` / `VERTEXAI_LOCATION` | Environment variables | Vertex AI project and region (default `global`). LiteLLM also accepts the Google Cloud standards `GOOGLE_CLOUD_PROJECT` / `GOOGLE_CLOUD_LOCATION`. |
| `GOOGLE_APPLICATION_CREDENTIALS` | Environment variable | Path to ADC JSON (local dev). Use Workload Identity in production. |
| `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` | Environment variables | Only required if you swap the model to a non-Vertex provider |
| Model | `config.yaml` | LiteLLM model identifier (e.g., `vertex_ai/gemini-3.5-flash`, `vertex_ai/claude-opus-4-8`, `openai/gpt-5.5`) |
| Max tokens | `config.yaml` | Response token limit |
| Temperature | `config.yaml` | Creativity level (low for factual metadata) |
Enrichment prompts include database schema information and profiling statistics. Ensure your LLM provider's data handling policies are compatible with the sensitivity level of your data. For Restricted data, consider using self-hosted LLMs.
## DQ Rule Generation
As part of enrichment, the LLM suggests data quality (DQ) rules for each column:
| Rule Type | Example |
| ---------------- | ------------------------------------------------ |
| **Completeness** | "email column should be 100% non-null" |
| **Format** | "phone column should match pattern XXX-XXX-XXXX" |
| **Range** | "age column should be between 0 and 150" |
| **Uniqueness** | "customer\_id should be unique" |
| **Referential** | "product\_id should exist in products table" |
| **Consistency** | "end\_date should be after start\_date" |
Generated rules are stored as structured metadata and can be used by the DQ monitoring system to track quality over time.
## Enrichment Results
Access enrichment results via the Data Governance API:
```json theme={null}
{
"table": "customers",
"enriched_at": "2026-04-03T12:00:00Z",
"business_description": "Customer master data containing contact information and account details for all registered customers.",
"domain": "Sales",
"sensitivity": "Confidential",
"columns": [
{
"name": "email",
"business_description": "Primary email address for customer communication",
"pii_classification": "PII",
"suggested_rules": [
{"type": "completeness", "threshold": 1.0},
{"type": "format", "pattern": "email"}
]
}
]
}
```
## Next Steps
Profile your data before enrichment to provide statistical context.
Learn about the Prefect orchestration system that runs enrichment.
Understand PII classification in the context of GDPR.
Review how enrichment outputs map to data classification levels.