What it does

Two historical dashboards over the output of the Quality Evaluation Agent — a Summary view of evaluation volume and quality performance, and an Evaluation Criteria view that breaks scores down by criteria and by the sections inside each one. Supervisors reach them from View Historical Analytics on the All evaluations page.

Key facts

• An admin turns it on: Customer Service admin center → Insights → Quality evaluation agent analytics → Manage → Enable Quality Evaluation Agent historical Analytics. Administrator role required.
• Two years of history, refreshed every 24 hours. It is not a near-real-time view.
• Role-based access control applies by business unit — a user only sees evaluations tied to their own business unit.
• Filters: Duration, Channel (case, conversation, email), Owner, Evaluation method (AI agent, AI-assisted, manual), Business unit, Critical questions. Evaluator Status sits in the side filter pane rather than with the rest.
• Shared metrics on both dashboards: Total Evaluations, % of evaluations below threshold, Avg. quality score, % of evaluations with critical question failures.
• A critical question failure fails the whole evaluation regardless of the overall score, and that has its own headline metric.
• Summary charts break volume and average score down by channel, and both of those over time.
• The Evaluation Criteria dashboard has weekly score columns per criteria and per section, so trend is visible at section level rather than just headline score.
• It's a Power BI report underneath — you can edit the dashboard, add pages and build your own visuals. Microsoft documents adding a table over FactEvaluationEntityName and FactEvaluationsExtension to sort records by lowest average score.
• Distinct from the Quality Assurance Agent dashboard, which reports conversation-level quality scoring rather than evaluation plans and criteria.

When to use / skip

Turn it on wherever the Quality Evaluation Agent is running with a criteria framework anyone has agreed to. Without historical analytics, supervisors only get per-evaluation results and have no way to see whether quality is moving. Skip while the criteria are still being drafted — trending a framework that changes every fortnight tells you nothing.

Configuration decisions

• Whether business-unit scoping matches how the client actually supervises. If team leaders sit outside the business unit whose evaluations they own, they'll see an empty dashboard.
• The evaluation threshold, since two of the four headline metrics are expressed relative to it.
• Which questions get marked Critical, because that drives the critical-failure metric and overrides scores entirely.
• Whether supervisors work from the stock dashboards or a customised copy — decide before people start bookmarking things.

Gotchas

• 24-hour refresh. Someone will complete an evaluation, open the dashboard and report the data as missing. It isn't, it's just tomorrow's.
• Evaluation method is a filter, not a caveat — AI agent, AI-assisted and manual evaluations are pooled into the same averages unless someone filters. Mixed methods with different levels of strictness make the headline average close to meaningless.
• The lowest-score view most supervisors actually want isn't in the box; it's the custom table Microsoft documents. Build it during deployment rather than leaving supervisors to find the instructions.
• EvaluationScore has to be aggregated as Average in any custom visual, because one record can carry several evaluations. Get that wrong and the numbers are silently inflated.
• Two years of history sounds generous until you realise it starts when the feature is enabled. Turn it on early, even if nobody's looking at it yet.

Consultant notes

• Enable this at the same time as the Quality Evaluation Agent, not as a phase two. History doesn't backfill, and a client who switches it on six months in has six months of evaluations they can't trend.
• The business-unit access model is the design decision that bites in UAT. Draw the supervision hierarchy against the business unit hierarchy on a whiteboard before configuring anything — in most organisations they don't match, and the fix is a security model change rather than a dashboard setting.
• Critical questions deserve their own conversation with the QA lead. Marking a question critical turns it into a pass/fail gate on the whole evaluation, and the dashboard makes the consequence very visible. Clients tend to mark far too many as critical on the first pass.
• Same warning as every quality dashboard: it's a coaching aid, not a performance-management instrument. Agree with HR up front how the numbers can and can't be used, and put it in writing before supervisors get access.

---

Revisit when Microsoft publishes the underlying data model, or if the refresh interval moves off 24 hours.