We're excited to introduce two improvements to the Rating view! You can now see the AI reasoning behind each rating and collect end-user feedback directly through a built-in feedback form. This makes ratings more transparent and actionable, while giving us a better way to capture feedback from your QA team.
Impact on Your Workflow ✨
- Transparency: Understand exactly why the AI passed or failed each criterion — no more guessing. 🔍
- Faster Reviews: Skip the full ticket read for criteria where AI reasoning confirms the decision is clear. ⏱️(Available with AutoQA only)
- Continuous Improvement: Your feedback feeds directly into AI improvements, making future ratings more accurate. 📈
AI Reasoning — How Does It Work? 🛠️
When you open a rated ticket in the QA House → Rate tab, you'll now see an AI reasoning block below each criterion.
What's included in the reasoning:
- The AI's decision — pass, fail, or uncertain (when confidence is low)
- The specific evidence from the conversation that led to the decision
- References to relevant policy or prompt instructions that were applied
💡 Pro-tip: The AI reasoning is visible only to QA Experts and Admins — agents see only the coaching notes, not the internal reasoning.
💡 Pro-tip x2: If the AI references a specific message in the conversation, look for clickable links in the reasoning that will take you directly to that part of the ticket.
🚨 Important: AI reasoning is available with AutoQA only.
Feedback Form — How Does It Work? 🛠️
Next to each criterion's AI reasoning, you'll find a feedback option to rate the AI's decision.
- Open a rated ticket from the QA House → Rate tab.
- Review the AI reasoning for each criterion.
- Click +Add Feedback option to add a feedback.
- If you agree or disagree with the AI's decision, add the feedback and click anywhere outside text box, it will be saved.
- Click on Submit IQS to finalise with feedback.
🚨 Important: Your feedback is collected at the criterion level, not the overall rating level. This means you can agree with the AI on 4 out of 5 criteria and flag only the one that needs work.
Why Your Feedback Matters 💡
Every piece of feedback is used to improve AI accuracy over time. Here's how:
- Disagreements are reviewed by our AI team and used to refine the prompts and instructions that drive each criterion.
- Agreements help us confirm which criteria are performing well and don't need changes.
- Over time, this creates a continuous improvement loop — the more feedback you provide, the more accurate the AI becomes for your specific scorecards.
Where to Find It 🔍
- AI Reasoning: Available in every rated ticket view with AutoQA, below each criterion in the scorecard.
- Feedback Form: Available in every rated ticket view, next to each criterion's AI reasoning.
- Both features work with AutoPilot ratings and manually triggered Auto QA ratings.
Who Can Use It? 👥
- QA Experts: Can view AI reasoning and submit feedback on any rating they review.
- QA Admins: Can view AI reasoning, submit feedback, and access aggregated feedback data.
- Agents: Cannot see AI reasoning. Agents continue to see only their coaching notes and QA score as before.
Getting Started 🚀
No setup required — the feature is already live in your account. Simply open any rated ticket and you'll see the AI reasoning and feedback form.
🧐 Tip: We recommend starting with your most recent AutoPilot ratings. Review the AI reasoning for a few tickets and submit feedback where you disagree. This helps our AI team prioritize the right improvements for your scorecards.
Questions or feedback? Reach out to us at support@kaizo.com or contact your Customer Success Manager.
Comments
0 comments
Please sign in to leave a comment.