Best Practices for Custom Questions in AI Quality Management
Custom questions help you measure and guide your agents’ behaviors and performance more precisely—but only if they are clearly written and pass AI Quality Management’s validation checks.
Follow these best practices to create effective, valid custom questions:
1. Use Clear and Grammatically Correct Language
Write questions that are concise, straightforward, and focus on a single behavior or action. Avoid questions that combine multiple requirements or use complex logic.
- “Did the agent ask clarifying questions?”
- “Did the agent follow a logical process to resolve the issue?”
- “Did the agent ask clarifying questions, troubleshoot, and pursue a logical course for resolution politely and efficiently?”
2. Focus on What’s Communicated, Not Unobservable Actions
AI Quality Management analyzes transcripts—it cannot evaluate tone, pitch, or other vocal qualities. Frame questions so an answer can be clearly determined from the text transcript. Avoid questions that require analyzing how something was said, or actions that can’t be seen in the transcript.
For industry-specific needs, start your question with “In the context of...” to help the AI validate the question appropriately.
- “Did the agent indicate they were entering notes in the system?”
- “Did the agent explain the refund policy to the customer?”
- "In the context of a pet supply store, did the agent ask the caller for their pet's name?"
- “Was the volume of the agent's voice too high?”
- “Did the agent enter notes in the system?” (unless the agent states it)
- "Did the agent ask the caller for their pet's name?"
3. Use Positive, Definite Yes/No Structure
Write questions so that a “yes” always indicates the desired, best-practice behavior—even for compliance or undesirable behaviors. This ensures the scoring will not be incorrect.
- “Did the agent avoid using inappropriate language?”
- “Did the agent follow privacy procedures?”
- “Did the agent use curse words?”
- “Did the agent violate policy?”
4. Script Questions
If you want to verify whether an agent used a specific phrase or script in their communication, use the following format for your custom question:
Format: Did the agent say: [Exact phrase, starting with a capital letter, no quotation marks, ending in a question mark?]
For best results, write your script questions clearly, but understand the agent's answer does not have to match the script word-for-word for the AI to count it as correct.
- Did the agent say: Thank you for calling GoTo, how can I help you today?
- Did the agent say: Is there anything else I can assist you with?
Best Practices:
- Start the scripted phrase with a capital letter.
- Do not use quotation marks around the phrase.
- End the question with a question mark.
5. Be Ethical and Service-Oriented
Avoid questions that could be perceived as unethical or that encourage undesirable behaviors. Don’t frame questions around suppressing customer feedback, discouraging complaints, or blocking escalation.
If your organization is in a regulated industry or handles sensitive topics, only collect responses to questions that are appropriate for your business context.
- “Did the agent handle the customer’s concerns in a way that resolved the issue without the need for further escalation?”
- “Did the agent prevent the customer from filing a formal complaint?”
6. Sensitive Topics
Questions about sensitive topics may trigger a system warning but can be overridden if discussing these topics is a normal part of your operation (e.g., medical, legal, or other contexts).
- Discussing medical procedures in healthcare
- Industry-specific terms (even those that may be flagged out of context)
7. Validation Reminders
- All custom questions must pass required validations before they are added.
- Ensure clarity, appropriate language, and ethical intent in all questions.
Summary Checklist
- Is your question clear, focused, and grammatically correct?
- Would the answer be observable in a text transcript?
- Does “Yes” correlate to the desired outcome?
- Is the question ethical and in line with best practices?
- Have you checked for flagged/sensitive content appropriate to your operation?
- Does your question pass validation?
If you need more help, select Contact support within this article.
