LLM Evaluation and Observability for Production Systems

Shipping an LLM feature is easy compared with knowing whether it still works next month. Prompt drift, index drift, model updates, and changing user behavior all quietly erode quality.

This tutorial is a practical stack for evaluation and observability that I use on production GenAI and RAG systems. It is not a vendor tour. It is a workflow you can implement with your current tools.

Split the Problem: Offline vs Online

• Offline evaluation: fixed datasets, repeatable scores, release gates
• Online observability: live traffic, traces, user feedback, cost and latency

You need both. Offline catches regressions before deploy. Online catches reality after users arrive.

Step 1: Build a Golden Set

Start with 50 to 200 real examples. Pull from support tickets, search logs, or sales FAQs. Include:

• Common easy cases
• Ambiguous questions
• Out-of-scope requests
• Permission-sensitive asks
• Known failure modes from past incidents

For each item, store the user input, expected behavior, and any required citations or tool calls. For RAG, store the documents that should be retrieved when possible.

Step 2: Define Metrics That Match the Job

For RAG answer systems

• Retrieval hit rate / recall@k
• Answer groundedness (claims supported by retrieved context)
• Correctness against a reference or rubric
• Refusal quality on out-of-scope questions

For agents and workflows

• Task success rate
• Wrong tool call rate
• Steps to completion
• Human escalation rate

For all systems

• p50 / p95 latency
• Cost per successful outcome
• Error and timeout rate

Step 3: Automate Offline Evals in CI

Run the golden set on every meaningful change: prompt, chunking, embedding model, generator model, or tool schema.

# Pseudocode for a release gate
results = run_eval_suite(dataset="golden_v3", config=candidate)
assert results.groundedness >= baseline.groundedness - 0.02
assert results.task_success >= baseline.task_success - 0.03
assert results.p95_latency_ms <= sla.p95_latency_ms
assert results.cost_per_success <= budget.cost_per_success * 1.15

Fail the build when quality or cost regresses past thresholds. Manual "looks good to me" reviews do not scale.

Step 4: Trace Every Production Request

Log a structured trace per request:

• Request ID, user segment, feature flag
• Retrieved document IDs and scores
• Model name, temperature, token counts
• Tool calls and outcomes
• Final answer and latency breakdown

Without traces, you cannot debug "it got worse on Tuesday." With traces, you can see whether retrieval, the model, or a tool caused the failure.

Step 5: Collect Online Feedback Signals

Useful signals, in order of reliability:

1. Task completion events (ticket closed, form submitted, checkout finished)
2. Explicit thumbs up / down with optional reason codes
3. Regeneration or edit rate
4. Escalation to human support

Do not optimize only for thumbs. Users downvote style issues and upvote fluent wrong answers. Pair feedback with outcome metrics.

Step 6: Watch Cost Like a Product Metric

Enterprise GenAI cost problems usually come from silent volume growth or agent loops, not from the first invoice. Track:

• Tokens and USD per route / model
• Cost per successful task, not per raw call
• Retry and fallback multipliers
• Top prompts or users by spend

Set alerts when daily cost rises more than a set percent without a matching success-rate gain.

A Minimal Weekly Review Cadence

• Monday: scan failed traces and top negative feedback clusters
• Midweek: patch prompts or retrieval rules for the top issue
• Friday: rerun offline suite and compare to last release

This is boring. It is also how systems stay trustworthy after the launch blog post.

If you cannot explain quality and cost with numbers, you do not have a production AI system yet. You have a hosted experiment.

Common Mistakes

• Evaluating only final answers and ignoring retrieval quality
• Changing models in production with no golden-set comparison
• Logging prompts without redacting secrets
• Using LLM-as-judge with no calibration against human labels
• Optimizing average latency while p95 is on fire

Bottom Line

Evaluation is how you decide what to ship. Observability is how you keep it alive. Put a golden set, release gates, traces, and cost dashboards in place before you scale traffic. Everything else is optional polish.