🤖 Auto QA: Setup

🚨To get started with the Auto QA feature, make sure you've completed the following steps 🚨

1. Ensure that the External Samurai is enabled. You can find instructions on enabling the External Samurai features here.
   
   
   
2. Confirm that you have the correct role and permissions assigned to you.
   🚨 By default, any roles capable of creating the QA Scorecard and/or QA-rating agent's tickets can set up and use the Auto QA feature.
   
   💡 If the QA Admin wants to test the Auto QA feature privately before making it visible to other users, please create a temporary Custom Role and unselect the "Samurai rating" permission for your QA Experts. Instructions on how to create a Custom role in Kaizo can be found here. 
   
   
3. Set up Auto QA Instructions and make sure that they will work in Settings -> Quality Assurance. You can find a step-by-step guide below 👇🏽

❗️Before we get started, here's something important to note❗️

• As you set up and test Auto QA, your QA raters and agents won't be impacted.
• Only when you switch on the "Enable Auto QA" toggle will your QA raters be able to use the automated criteria.
• Feel free to tweak the instructions or hold off on publishing until you're satisfied with the test results.

1. Choose the QA Scorecard to automate 

To begin, either choose one of your existing Quality Scorecards or generate a new one by navigating to Settings -> Quality Assurance -> QA Scorecards.

Select a criterion from the QA Scorecard and click on "Set up Auto QA."

🤩 What criteria should I start with?

• Try automating Language-related criteria, such as Grammar, Spelling, and usage of forbidden words
• Take it to the next level and automate the process-related criteria. You can describe the company's procedures right inside the Instructions

⭐️ Want to create a new QA Scorecard, but not sure where to start? No problem—check out this Help Center article for guidance!

2. Write Instructions for Samurai 

Feeling a bit intimidated? Fear not! It's remarkably straightforward! 💪

Clearly explain when Samurai should select this criterion as if you were instructing a new QA Expert during onboarding!

🤫 Psst, we also provide an example of how a typical instruction should be phrased. For further insights into crafting effective instructions for the Auto QA, refer to this article!

🤩 Feeling confident in your instructions?! Hit the "Test" button and see how your prompt would work on the previously QA-rated tickets

3. Testing of the Instructions

It's crucial to assess how the instructions you provided align with QA ratings conducted by your QA experts.

👇🏽💡After testing is complete, the random sample of QA-rated tickets will be divided into 2 columns:

• A column with tickets, where the criterion was selected by the QA Expert during the rating
• A column with tickets, where the criterion was not selected by the QA Expert during the rating

👆🏽 You also will receive a % match indicating the expected success of these instructions with future ratings. In simpler terms: "How often Samurai will correctly evaluate this criterion."

🧐 How do we know that? We ask Samurai to evaluate the same QA ratings previously assessed by your QA experts, based on the instructions you provided.

• • If AI produces the same result for the QA rating (criterion selected/not selected) as the QA Expert, the ticket will not be highlighted.
  • If AI disagrees with the decision made by the QA Expert, the ticket will appear in red and is recommended for review.
  • There are cases where AI is not sure how to evaluate the specific case. The reasoning will also be provided to explain why Samurai is "confused".

4. I've Got the Results - What's Next?

Next, we should understand how to improve the percentage match (unless a satisfactory result is achieved right away! 🏆).

Several reasons can contribute to a lower-than-expected match percentage:

• Most common - the Instructions were not clear enough.
  ❗️Remember, while it may be clear to the QA expert that the agent won't be marked for every small grammar mistake, the machine doesn't know that. If you instruct Samurai to select the "Grammar mistake" criterion for every grammar mistake, this might result in a low match limit, as the machine may detect more mistakes than humans.
  
  💡Get more insights on how to improve your Instructions here. 
  
  
• You forgot the "edge cases" - by reviewing the tickets, you will remember that some criteria are not applied for some of the tickets/contact reasons. Samurai was not aware of this information as it was not mentioned in the Instructions. 
  🎯Add this information to the prompt and hit "Test" again to see if this change helped to get a higher % of Match! 
  
  
• QA Expert was not correct in marking the criterion - this can happen as well! In this case, you still can let Samurai know how it actually should have been QA-rated by dragging the ticket into the correct column
  
  

⭐️ Pro-tip: Click on the ticket to open it inside Kaizo. When a ticket is opened you can see the QA rating to which Samurai was comparing the results, including QA Expert's comments and notes. 

⭐️ Pro-tip x2: Samurai, in the testing phase, will also share with you the reasoning behind selected/not selected criteria. This information should help you when setting up the Instructions for AI.

❗️Don't forget to Save all your changes before leaving the Auto QA Settings. 

5. As soon as you are satisfied with the Testing results, Enable Auto QA! 

Whenever you are happy with the results (the % match is high enough) you can proceed and allow your QA team to use assistance during ticket evaluations.

It's straightforward - just switch the toggle to Enable Auto QA and save the QA Scorecard.

🧐 Who will get access to it? Everyone with the permission to QA rate (QA Experts, Team Leads, Admins, QA Admins).

🔥Interested in learning how to use Auto QA during QA ratings? Visit this link!