# Operator QA and coaching dashboard

Review conversation quality trends, coaching areas, manager calibration and operator feedback in one dashboard.

*Release 3.3 · admin, supervisor*

**Service Overview › Performance › Operator QA** · **Academy › Operator Feedback** · **Settings › AI › Features**

## What it is

**Operator QA** brings together conversation quality scores, trends and coaching detail. You can compare operators, inspect low-scoring conversations and see where managers changed the original AI scores. Operator feedback helps you understand disagreements with scores or individual criteria.

## Who uses it and why

- Supervisors with the **Manager** role investigate quality trends and identify coaching needs.
- Administrators review scoring coverage and scorecard configuration, using **Manager** access to read the dashboard.

## Where to find it

Open **Service Overview › Performance › Operator QA** with the **Manager** role and access to **Service Overview**.

**Academy › Operator Feedback** offers another feedback view when AI is enabled and at least one scorecard is available. **AI Scoring Performance** and **AI Scoring Agent Performance** are separate tables in **Reports › AI**.

## How it works

1. Choose the period and available subsite, team and channel filters in **Service Overview**. A separately opened dashboard offers **From**, **To** and **Apply**.
2. Review **Avg Score**, the score distribution, **Scored** and **Coverage**. The dashboard uses score bands below 60, at least 60 and below 80, and 80 or above.
3. Inspect **Daily trend** and **Operator leaderboard**. Select an **Operator** to focus the detail cards.
4. Use **Top 5 Areas of Improvements** and **Areas of Improvements Breakdown** to inspect weaker categories and criteria.
5. Review **Calibration drift** to compare **AI avg**, **Manual avg**, override counts and the average difference.
6. Open linked chats in **Low-scoring chats** or **Operator feedback** to inspect the conversation and its review context.
7. Select **Print** to print the current dashboard through your browser.

Selecting an operator narrows the detail cards but keeps the peer leaderboard for the same date, team, channel and subsite selection. The chosen operator is highlighted within that comparison.

Feedback can apply to the whole conversation or one criterion. Its list includes the chat, operator, score, feedback, scope, scope score, comment and date.

## Configure it

Manage scorecards through **Settings › AI › Features** with the **Settings** or **Developer** role. Scorecard criteria, minimum message counts and individual review actions belong to conversation quality scoring.

The dashboard does not change the scorecard or the recorded feedback. Its default local period starts 30 days ago; shared **Service Overview** filters can supply a different period.

### Permissions

| Permission | Lets you |
|---|---|
| Manager | Read the Operator QA dashboard and operator feedback for the website. |

## Limits and good to know

- Operators or criteria with fewer than ten scored conversations remain visible with a **low sample** warning and are not ranked against better-sampled results in the usual way.
- Compare categories and criteria within the same scorecard. Different scorecards can use different criteria.
- Coverage estimates eligibility from closed or resolved conversations that meet the configured minimum message count. It can include conversations whose interaction type does not match a scorecard, so it can understate coverage of truly eligible work.
- The default minimum is four messages when no usable scoring minimum is available. Without a specific scorecard, the lowest positive minimum across enabled scorecards is used.
- **Low-scoring chats** shows scores below 60 and normally returns up to 50 chats. Feedback normally returns up to 200 entries, and conversation drill-downs up to 500 identifiers. These lists can be smaller than aggregate counts.
- Missing scores appear as missing values. Read score averages alongside sample counts.
- The dashboard combines historical and recent activity. Its coverage rules differ from the scoring tables, so matching dates alone do not guarantee matching percentages.
- **Print** includes the period and print time and uses a white background, including when your screen uses dark mode.

## Depends on and works with

### This feature uses

- **Requires** [AI conversation quality scoring](AI-029.md): Conversation scoring, manager reviews and operator feedback supply the dashboard results.
- **Requires** [Service Overview dashboard](REP-016.md): Service Overview provides access to the Operator QA dashboard and its shared filters.
- **Configured in** [Roles and permissions](ADM-008.md): The Manager role grants client access to dashboard data.
- **Configured in** [Website, channel and subsite access](ADM-009.md): Reporting access restrictions affect the permitted conversation scope.
- **Configured in** [AI feature settings and change history](AI-001.md): Enabled scorecards and their minimum message counts affect scoring and coverage.
- **Related** [Academy coaching and message feedback](CON-030.md): Academy provides another view of operator feedback on AI scores when available.
- **Related** [AI conversation scoring reports](REP-025.md): Scoring performance tables provide separate date and operator summaries with different coverage rules.

### Used by

- **Enhances** [Service Overview performance comparisons](REP-015.md): Operator QA adds a quality-scoring and coaching subtab to Service Overview Performance.
- **Enhances** [Service Overview dashboard](REP-016.md): Enabled Operator QA provides the quality-review dashboard.
- **Related** [Academy coaching and message feedback](CON-030.md): Operator QA also shows operator feedback on conversation quality scores.
- **Related** [AI conversation quality scoring](AI-029.md): Operator QA combines quality scores, review feedback and coaching detail.
- **Related** [AI conversation scoring reports](REP-025.md): Operator QA adds coaching and calibration views alongside the separate scoring tables.
- **Related** [Business Feed and Academy reports](REP-028.md): Operator QA reports quality scores and feedback separately from Academy comment acknowledgements.

## FAQ and troubleshooting

### Why can I not open Operator QA?

Client access requires **Manager**, together with access to **Service Overview**. The **Reports** role alone grants the reporting tables, rather than this dashboard.

### Why does the leaderboard still show everyone after I select an operator?

It keeps the permitted peer comparison for the selected period and scope. The detail cards focus on your chosen operator.

### Why is a high-scoring operator listed below other operators?

Check the **low sample** warning. Fewer than ten scored conversations do not qualify for the normal comparison against better-sampled rows.

### Why are fewer chats listed than the total says?

The detail lists have limits. A low-score list or conversation drill-down is not an unlimited export of every counted record.

### Why is Operator Feedback missing from Academy?

The view needs **Manager** access, AI enabled and an available scorecard. After a configuration change, start a new session so the availability check can run again.

### What does calibration drift tell me?

It shows how managers' replacement scores differ from the original AI scores. Use the criterion detail to discuss scoring consistency and inspect the conversations before drawing conclusions.