> ## Documentation Index
> Fetch the complete documentation index at: https://docs.emergence.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Observability

> Understanding LLM-specific observability for CRAFT, why it differs from infrastructure observability and how Langfuse and OpenTelemetry complement each other.

# LLM Observability

CRAFT uses two complementary observability systems that operate at different layers:

* **OpenTelemetry**, infrastructure-level: HTTP requests, database queries, Redis, service health
* **Langfuse**, LLM-level: model calls, token usage, cost, prompt quality, evaluation results

Both are valuable; neither replaces the other.

## Why LLM Observability Is Different

Traditional observability (metrics, traces, logs) was designed for deterministic systems. LLM-powered applications introduce unique observability challenges:

| Challenge              | Why it matters                                                                                                             |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| **Non-determinism**    | The same prompt can produce different outputs. Standard error rates don't capture quality degradation.                     |
| **Token economics**    | Cost is proportional to token usage, not request count. A single call can cost $0.001 or $1.00 depending on prompt length. |
| **Prompt engineering** | Changing a prompt is a configuration change, not a code change, but it can dramatically affect output quality.             |
| **Evaluation**         | "Is this response correct?" requires domain-specific evaluation, not just latency or error rate checks.                    |
| **Multi-turn context** | A user session spans multiple LLM calls. Standard tracing doesn't capture the logical conversation flow.                   |

## What Each System Captures

### OpenTelemetry (Infrastructure)

```text theme={null}
Service → [HTTP/DB/Redis spans] → OTel Collector → Grafana
```

| Signal      | Examples                                              |
| ----------- | ----------------------------------------------------- |
| **Metrics** | Request rate, P99 latency, error rate, pod CPU/memory |
| **Traces**  | End-to-end request flow through services              |
| **Logs**    | Service logs, error messages, audit events            |

Answers: "Is the service up? Is it slow? Are there errors?"

### Langfuse (LLM)

```text theme={null}
LLM call → [LiteLLM callback] → Langfuse
```

| Signal          | Examples                                                    |
| --------------- | ----------------------------------------------------------- |
| **Traces**      | Complete LLM session with all turns, tools, and context     |
| **Metrics**     | Tokens used, estimated cost, latency per model call         |
| **Evaluations** | LLM-as-a-judge quality scores, human labels, rubric results |
| **Prompts**     | Version history, usage statistics, A/B comparison           |

Answers: "Is the AI producing quality output? What's it costing? Which prompt version is better?"

## Architecture

```mermaid theme={null}
%%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#555555', 'fontFamily': 'sans-serif', 'edgeLabelBackground': '#ffffff'}}}%%
flowchart TB
    subgraph App["Application Services"]
        Svc["em-talk2data / em-semi"]
        LLM["LiteLLM (LLM proxy)"]
    end

    subgraph OTel["Infrastructure Observability"]
        OTelC["OTel Collector"]
        Prom["Metrics Backend"]
        Grafana["Grafana"]
    end

    subgraph LFuse["LLM Observability"]
        LF["Langfuse"]
    end

    Svc -->|HTTP spans| OTelC
    Svc --> LLM
    LLM -->|LiteLLM callbacks| LF
    LLM -->|LLM API calls| ExtLLM["Gemini / GPT / Claude"]

    OTelC --> Prom --> Grafana
    LF --> LFDash["Langfuse Dashboard"]

    classDef app fill:#E69F00,stroke:#555555,color:#000
    classDef otel fill:#009E73,stroke:#555555,color:#000
    classDef lf fill:#0072B2,stroke:#555555,color:#fff
    classDef ext fill:#CC79A7,stroke:#555555,color:#000
    class Svc,LLM app
    class OTelC,Prom,Grafana otel
    class LF,LFDash lf
    class ExtLLM ext
```

## Integration Pattern

The platform uses **LiteLLM** as a provider-agnostic LLM proxy. LiteLLM natively supports Langfuse as a callback handler, requiring no changes to application code.

When `LANGFUSE_HOST` is set, LiteLLM automatically:

1. Records each LLM API call to Langfuse (prompt, completion, model, tokens, cost)
2. Groups calls into sessions by conversation ID
3. Reports evaluation scores if evaluators are configured

## Langfuse Trace Anatomy

A Langfuse trace for a Data Insights session might look like:

```text theme={null}
Session: "user-query-schema-explain"
  └─ Trace: /api/chat (3.2s)
       ├─ Span: schema_lookup (0.3s) [DB query]
       ├─ Span: LLM call (2.5s)
       │    Model: gemini-3.5-flash
       │    Input tokens: 1,240
       │    Output tokens: 380
       │    Cost: $0.0018
       │    Quality score: 0.87 (LLM-as-judge)
       └─ Span: format_response (0.4s)
```

## Cost Tracking

Langfuse aggregates LLM costs across all calls, enabling:

* Cost per conversation / session / user
* Cost trends over time
* Model comparison (cost vs. quality tradeoff)
* Budget alerting (configurable thresholds)

## Related

<CardGroup cols={2}>
  <Card title="Langfuse" icon="magnifying-glass" href="/deployment/observability/langfuse">
    Deploy and configure Langfuse.
  </Card>

  <Card title="OpenTelemetry" icon="wave-square" href="/deployment/observability/opentelemetry">
    Infrastructure observability with OTel.
  </Card>
</CardGroup>