Quality Evaluation Agent

What it does

AI agent that scores customer interactions (conversations, cases, emails) against a supervisor-defined evaluation framework. Replaces manual QA sampling with automated, consistent scoring across all interactions. Feeds quality scores into the Agent Insights dashboard.

Key facts

Three roles: Quality Administrator (full access), Quality Manager (create criteria/plans, complete evaluations), Quality Evaluator (complete evaluations only)
Three prerequisites must all show Ready before the agent works: connection references (Dataverse), Power Automate flows (must be enabled), Copilot Studio agent (must be published)
Requires pay-as-you-go Copilot credits: not included in standard licensing
Requires Enable AI agents toggle in Power Platform admin center (separate from D365 settings)
Scoring cannot be turned off once enabled: set the threshold value deliberately before enabling
Email evaluation is preview only
Conversations: Contact Center standalone only; Cases: Customer Service only
Quality score KPI in Agent Insights dashboard only populates if Quality Evaluation Agent is active
Bulk evaluation available for cases (retroactive assessment of historical data)
Knowledge sources in criteria (preview): a criterion can link one published Dynamics 365 knowledge article, and up to two questions per criterion can be grounded in it. The agent scores against the latest version of the article and cites it in the evaluation summary. It needs Dataverse Copilot indexing and knowledge sources switched on in Copilot Service admin center
Knowledge-grounded criteria work for the case entity and on-demand evaluations only, can't be simulated, and only accept Dynamics 365 knowledge articles
Questions can be marked Not Applicable: they drop out of the score and the total is calculated from the applicable questions only
Regenerate evaluation summary is a separate switch in the quality evaluation settings, alongside scoring and the threshold

When to use / skip

Use on any deployment where the client wants automated QA, consistent scoring, or quality-based KPI tracking. Essential if Agent Insights quality score is in scope. Skip if there's no QA process to digitise, the criteria design requires investment from the QA team.

Configuration decisions

Evaluation criteria and scoring logic: the quality framework must be designed with the client's QA team; generic criteria produce generic insights
Scoring threshold: defines good vs poor quality (out of 100); get client sign-off before enabling, it cannot be changed without disabling
Which record types to enable: Conversation, Case, Email (preview), or combination
Whether to enable bulk evaluation: useful post-go-live to establish a pre-deployment quality baseline on historical cases

Gotchas

Prerequisites checklist is the common failure. All three (connection references, Power Automate flows, Copilot Studio agent) must show Ready. Clients enable the toggle and assume it's done: verify all three before testing.
Scoring is permanent. Discuss threshold with the client first. Enabling with a poor threshold means living with it or complex remediation.
"Enable AI agents" is in Power Platform admin, not D365. If Quality Evaluation Agent won't activate, check this setting: easy to miss.
Knowledge grounding has hard edges. One article per criterion and two questions per criterion, no simulation, cases and on-demand only. Design the criteria around those limits before promising "scored against our policy library".
Not Applicable changes the arithmetic. A call with several N/A answers is scored on fewer questions, so scores aren't directly comparable between interactions unless the QA team understands that.
Compliance obligation: Reps must know interactions may be monitored, recorded, and scored. Legal requirement in many jurisdictions: build into project governance.

Consultant notes

Evaluation criteria design needs to come from the client's QA team, not from you. Generic criteria (was the agent polite, did they resolve the issue) produce generic scores that nobody acts on. Invest time in a criteria workshop before any configuration starts: the framework is the whole point.
Scoring threshold needing a sign-off before enabling is one to document formally. What constitutes "good quality" at this organisation is a business decision, not a technical one. Get written sign-off from the quality lead before enabling: the threshold can't change without remediation.
The prerequisites checklist (three must-show-Ready) is the silent failure. Walk through all three during setup, not just the D365 toggle. The "Enable AI agents" being in Power Platform admin is the one that most commonly gets missed.

Refreshed 2026-10-24 for knowledge sources in evaluation criteria, Not Applicable scoring and summary regeneration. Check this after email evaluation or knowledge-grounded criteria leave preview, or new record types are added.