Microsoft Copilot Studio - Improvements to agent Evaluations experience

We are enhancing the Evaluations experience to help makers better understand agent quality and behavior in Microsoft Copilot Studio. This feature is now available.

How does this affect me?
The improvements will bring new capabilities that help makers identify quality issues more quickly, understand why evaluations succeed or fail, compare results across runs, and create higher quality evaluation datasets with less manual effort. Additional validation and guidance will help makers resolve configuration issues before running evaluations. New capabilities include the following:

• Richer explanations of evaluation outcomes will help makers understand why evaluations succeed or fail.
• Reasoning traces and cited knowledge sources will help makers understand agent behavior and grounding.
• Evaluation run comparison will help identify improvements and regressions across runs.
• Evaluations will support datasets with up to 100 test cases.
• Makers will be able to use customizable dataset and question generation.
• Makers will be able to generate datasets from configured knowledge sources.
• Single-response datasets will be supported.
• Improved validation, connection prerequisite checks, credit-limit enforcement, and actionable error messages will help makers resolve configuration issues before running evaluations.

These enhancements are designed to reduce manual effort, improve troubleshooting, and help makers evaluate and optimize agents more effectively.

What action do I need to take?
This message is for awareness, and no action is required.