Operator QA and coaching dashboard
Review conversation quality trends, coaching areas, manager calibration and operator feedback in one dashboard.
What it is
Operator QA brings together conversation quality scores, trends and coaching detail. You can compare operators, inspect low-scoring conversations and see where managers changed the original AI scores. Operator feedback helps you understand disagreements with scores or individual criteria.
Who uses it and why
- Supervisors with the Manager role investigate quality trends and identify coaching needs.
- Administrators review scoring coverage and scorecard configuration, using Manager access to read the dashboard.
Where to find it
Open Service Overview › Performance › Operator QA with the Manager role and access to Service Overview.
Academy › Operator Feedback offers another feedback view when AI is enabled and at least one scorecard is available. AI Scoring Performance and AI Scoring Agent Performance are separate tables in Reports › AI.
How it works
- Choose the period and available subsite, team and channel filters in Service Overview. A separately opened dashboard offers From, To and Apply.
- Review Avg Score, the score distribution, Scored and Coverage. The dashboard uses score bands below 60, at least 60 and below 80, and 80 or above.
- Inspect Daily trend and Operator leaderboard. Select an Operator to focus the detail cards.
- Use Top 5 Areas of Improvements and Areas of Improvements Breakdown to inspect weaker categories and criteria.
- Review Calibration drift to compare AI avg, Manual avg, override counts and the average difference.
- Open linked chats in Low-scoring chats or Operator feedback to inspect the conversation and its review context.
- Select Print to print the current dashboard through your browser.
Selecting an operator narrows the detail cards but keeps the peer leaderboard for the same date, team, channel and subsite selection. The chosen operator is highlighted within that comparison.
Feedback can apply to the whole conversation or one criterion. Its list includes the chat, operator, score, feedback, scope, scope score, comment and date.
Configure it
Manage scorecards through Settings › AI › Features with the Settings or Developer role. Scorecard criteria, minimum message counts and individual review actions belong to conversation quality scoring.
The dashboard does not change the scorecard or the recorded feedback. Its default local period starts 30 days ago; shared Service Overview filters can supply a different period.
Permissions
| Permission | Lets you |
|---|---|
| Manager | Read the Operator QA dashboard and operator feedback for the website. |
Limits and good to know
- Operators or criteria with fewer than ten scored conversations remain visible with a low sample warning and are not ranked against better-sampled results in the usual way.
- Compare categories and criteria within the same scorecard. Different scorecards can use different criteria.
- Coverage estimates eligibility from closed or resolved conversations that meet the configured minimum message count. It can include conversations whose interaction type does not match a scorecard, so it can understate coverage of truly eligible work.
- The default minimum is four messages when no usable scoring minimum is available. Without a specific scorecard, the lowest positive minimum across enabled scorecards is used.
- Low-scoring chats shows scores below 60 and normally returns up to 50 chats. Feedback normally returns up to 200 entries, and conversation drill-downs up to 500 identifiers. These lists can be smaller than aggregate counts.
- Missing scores appear as missing values. Read score averages alongside sample counts.
- The dashboard combines historical and recent activity. Its coverage rules differ from the scoring tables, so matching dates alone do not guarantee matching percentages.
- Print includes the period and print time and uses a white background, including when your screen uses dark mode.
Depends on and works with
View all connections as a list
This feature uses
- Roles and permissionsConfigured by
The Manager role grants client access to dashboard data.
- Website, channel and subsite accessConfigured by
Reporting access restrictions affect the permitted conversation scope.
- AI feature settings and change historyConfigured by
Enabled scorecards and their minimum message counts affect scoring and coverage.
- AI conversation quality scoringRequires
Conversation scoring, manager reviews and operator feedback supply the dashboard results.
- Service Overview dashboardRequires
Service Overview provides access to the Operator QA dashboard and its shared filters.
- Academy coaching and message feedbackRelated
Academy provides another view of operator feedback on AI scores when available.
- AI conversation scoring reportsRelated
Scoring performance tables provide separate date and operator summaries with different coverage rules.
Used by
- Academy coaching and message feedbackRelated
Operator QA also shows operator feedback on conversation quality scores.
- AI conversation quality scoringRelated
Operator QA combines quality scores, review feedback and coaching detail.
- Service Overview performance comparisonsEnhances
Operator QA adds a quality-scoring and coaching subtab to Service Overview Performance.
- Service Overview dashboardEnhances
Enabled Operator QA provides the quality-review dashboard.
- AI conversation scoring reportsRelated
Operator QA adds coaching and calibration views alongside the separate scoring tables.
- Business Feed and Academy reportsRelated
Operator QA reports quality scores and feedback separately from Academy comment acknowledgements.
FAQ and troubleshooting
Why can I not open Operator QA?
Client access requires Manager, together with access to Service Overview. The Reports role alone grants the reporting tables, rather than this dashboard.
Why does the leaderboard still show everyone after I select an operator?
It keeps the permitted peer comparison for the selected period and scope. The detail cards focus on your chosen operator.
Why is a high-scoring operator listed below other operators?
Check the low sample warning. Fewer than ten scored conversations do not qualify for the normal comparison against better-sampled rows.
Why are fewer chats listed than the total says?
The detail lists have limits. A low-score list or conversation drill-down is not an unlimited export of every counted record.
Why is Operator Feedback missing from Academy?
The view needs Manager access, AI enabled and an available scorecard. After a configuration change, start a new session so the availability check can run again.
What does calibration drift tell me?
It shows how managers' replacement scores differ from the original AI scores. Use the criterion detail to discuss scoring consistency and inspect the conversations before drawing conclusions.