A good playground session is not evidence
Generative systems are easy to demonstrate because people naturally choose cooperative prompts and recognize plausible language as progress. Production traffic is different: inputs are incomplete, permissions change, source material conflicts, tools time out, and users discover paths the project team did not imagine.
Evaluation turns “it looked good” into a release decision. Microsoft Foundry provides evaluators, tracing, monitoring, and Application Insights integration, but the organization must still define what success means for its workflow.
Start with the decision the system supports
Do not begin by selecting generic metrics. Write the operating claim first:
Given an approved case record, the assistant produces a grounded draft that a qualified reviewer can accept or correct without introducing unsupported facts.
That claim reveals the needed evidence: representative case records, expected source use, forbidden claims, reviewer effort, latency, and the treatment of uncertainty.
If the system takes actions, add claims for tool selection, argument accuracy, authorization, side effects, and recovery.
Build a representative evaluation set
A useful dataset contains more than typical examples. Include:
- frequent, commercially important cases;
- difficult but valid cases;
- incomplete and contradictory inputs;
- requests that should be refused or escalated;
- sensitive data and authorization boundaries;
- tool and dependency failures;
- previously observed production failures.
Version the dataset and document why each case exists. Protect test records with the same discipline as production data. Synthetic cases can expand coverage, but they should not replace expert-curated examples from the actual workflow.
Use a portfolio of measures
No single score captures production fitness. Combine different forms of evidence:
Deterministic checks verify schema, citations, required fields, prohibited content, numeric constraints, and exact tool arguments.
Task measures assess whether the system completed the business job. These are often domain-specific and may require human review or a custom evaluator.
Quality measures can assess relevance, groundedness, coherence, or fluency. A fluent answer can still be wrong; writing quality must never stand in for factual or task correctness.
Safety and security measures test harmful content, prompt attacks, sensitive-data handling, and policy-specific risks.
Agent measures examine task completion, tool-call accuracy, trajectory quality, and whether the system stayed within the allowed operating envelope.
LLM-as-judge evaluators are useful, but they are probabilistic instruments. Calibrate them against human judgments and inspect disagreements.
Make the gate explicit
Define pass criteria before running the final evaluation. A release gate might require:
- zero critical authorization or data-boundary failures;
- all required deterministic checks passing;
- a minimum task-success threshold on priority cases;
- no regression beyond an agreed tolerance;
- acceptable latency and cost under representative load;
- named acceptance of known residual failures.
Report distributions and individual failures, not only an average. A 95% score can hide a catastrophic failure in the five percent that matters most.
Connect traces to evaluation
Distributed traces expose model calls, retrieval, tool invocations, latency, and dependencies across a multi-step flow. Use them to explain why an evaluation failed. The answer may be wrong because retrieval returned the wrong source, the tool received a malformed identifier, or an instruction changed the decision path.
Store the application, prompt, agent, model, tool, and dataset versions with each run. Without version lineage, comparison becomes guesswork.
Continue after release
Production observability should track operational health and quality signals. Sample appropriate traffic for evaluation within privacy and retention boundaries. Alert on meaningful conditions, not every statistical movement.
Add confirmed incidents and recurring failure patterns back into the evaluation set. Re-run the gate when prompts, models, retrieval, tools, policies, or application logic change materially. Model updates are only one source of regression.
Evaluation is a management system
The goal is not a dashboard full of scores. It is a repeatable agreement between business owners, domain experts, engineering, security, and operations about what may be deployed.
Foundry supplies useful evaluation and observability machinery. Production confidence comes from the quality of the cases, thresholds, ownership, and corrective loop wrapped around it.
Official reference
Microsoft outlines evaluation, monitoring, tracing, pre-production testing, and continuous production evaluation in Observability in Generative AI.