> ## Documentation Index
> Fetch the complete documentation index at: https://docs.emergence.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Data Profiling
> How the Data Governance solution profiles data assets at the column and table level to assess data quality, completeness, and statistical characteristics.
# Data Profiling
Data profiling is the foundation of the Data Governance solution. It analyzes connected databases to produce statistical profiles of tables and columns, identifying data quality issues, completeness gaps, and structural anomalies.
## What Data Profiling Captures
For each column in a profiled table, the system computes:
| Metric | Description |
| -------------------------- | -------------------------------------------- |
| **Completeness** | Percentage of non-null values |
| **Uniqueness** | Ratio of distinct values to total rows |
| **Data type distribution** | Actual types vs declared column type |
| **Min / Max / Mean** | Statistical bounds for numeric columns |
| **Standard deviation** | Spread of numeric values |
| **Pattern analysis** | Common formats (email, phone, date patterns) |
| **Top N values** | Most frequent values and their counts |
| **Outlier detection** | Values beyond 2 standard deviations |
For each table:
| Metric | Description |
| -------------------------- | ------------------------------- |
| **Row count** | Total number of records |
| **Column count** | Number of columns |
| **Completeness score** | Average column completeness |
| **Freshness** | Last update timestamp |
| **Primary key analysis** | Key uniqueness and completeness |
| **Foreign key validation** | Referential integrity checks |
| **Duplicate detection** | Potential duplicate rows |
## Profiling Workflow
```mermaid theme={null}
%%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%%
graph TB
API["Data Governance API\n(trigger / schedule)"]
FLOW["Prefect Flow\n(profile_database)"]
CTR["ConcurrentTaskRunner\n(max 10 tables parallel)"]
SEM["asyncio.Semaphore\n(20 column analyses / table)"]
SRC[("Source Database\n(via data connection)")]
subgraph Results["Results"]
COL["Column metrics\n(completeness, uniqueness,\npatterns, outliers)"]
TBL["Table metrics\n(row counts, freshness,\nPK/FK validation)"]
end
DB[("datareadiness DB\n(results + timestamps)")]
SC["Scorecard"]
API --> FLOW --> CTR --> SEM --> SRC
SEM --> COL & TBL
COL & TBL --> DB --> SC
classDef trigger fill:#56B4E9,stroke:#555555,color:#000
classDef orch fill:#E69F00,stroke:#555555,color:#000
classDef gate fill:#F0E442,stroke:#555555,color:#000
classDef src fill:#0072B2,stroke:#555555,color:#fff
classDef result fill:#009E73,stroke:#555555,color:#000
class API trigger
class FLOW,CTR orch
class SEM gate
class SRC src
class COL,TBL,DB,SC result
```
Data profiling is orchestrated as a Prefect workflow:
An administrator or data steward selects the tables and schemas to profile via the Data Governance API.
The Prefect flow launches with `ConcurrentTaskRunner` for parallel table processing. Each table is profiled as an independent task.
For each table, the workflow queries the database to compute column-level statistics. Queries are optimized using sampling for large tables.
Profiling results are stored in the Data Governance database (`datareadiness`) with timestamps for historical tracking.
A data quality scorecard summarizes the profiling results across all tables, highlighting issues that need attention.
## Concurrency Control
Profiling workflows use hybrid concurrency to efficiently process large databases without overwhelming the source:
```python theme={null}
@flow(task_runner=ConcurrentTaskRunner(max_workers=10))
async def profile_database(tables: list[str]):
# Process up to 10 tables concurrently
tasks = [profile_table(table) for table in tables]
await asyncio.gather(*tasks)
@task
async def profile_table(table: str):
semaphore = asyncio.Semaphore(20)
# Within each table, run up to 20 column analyses concurrently
async with semaphore:
await analyze_column(column)
```
| Level | Mechanism | Configuration |
| -------------- | ------------------------------------- | ------------- |
| **Flow-level** | `ConcurrentTaskRunner(max_workers=N)` | `config.yaml` |
| **Task-level** | `asyncio.Semaphore(N)` | `config.yaml` |
Adjust the `max_workers` and semaphore limits based on your source database capacity. Start with conservative values and increase based on observed database load.
## Integration with Data Connections
Profiling uses data connections registered in the platform's Assets service:
1. The data steward selects a registered data connection
2. The profiling workflow retrieves connection credentials from the Secrets API at runtime
3. A read-only database session is established for profiling queries
4. All profiling queries use the read-only user configured during [data source setup](/guides/data-source-setup)
Profiling queries can be resource-intensive on large tables. Schedule profiling runs during off-peak hours or configure sampling thresholds for tables with millions of rows.
## Profiling Results
Profiling results are accessible via the Data Governance API:
```bash theme={null}
# Get profiling results for a table by FQN (e.g. analytics-db.public.customers)
curl -X GET \
"https:///api/data-readiness/data-assets/profiles/tables/by-fqn?fqn=" \
-H "Authorization: Bearer $TOKEN" \
-H "X-Project-ID: "
```
Results include:
```json theme={null}
{
"table": "customers",
"profiled_at": "2026-04-03T10:00:00Z",
"row_count": 150000,
"columns": [
{
"name": "email",
"completeness": 0.98,
"uniqueness": 0.97,
"pattern": "email",
"top_values": ["gmail.com", "outlook.com", "yahoo.com"]
},
{
"name": "phone",
"completeness": 0.72,
"uniqueness": 0.95,
"pattern": "phone_us",
"null_count": 42000
}
],
"quality_score": 0.85
}
```
## Scheduling Profiles
Profiling can be scheduled for recurring execution via the platform's scheduling system:
* **Daily profiles**: Track data quality trends over time
* **Weekly profiles**: Standard cadence for most datasets
* **On-demand**: Triggered manually for new data sources or after schema changes
## Next Steps
Enrich metadata after profiling with LLM-powered descriptions.
Learn about the Prefect workflow orchestration system.
Connect a database to start profiling.
Understand how profiling data is classified and protected.