Scorecards
Every conversation gets a grade — auto-score 100% of calls on accuracy, compliance, empathy, and resolution
Every Conversation Gets a Grade
Manual QA covers less than 5% of conversations. Scorecards let you score 100% of them — consistently and instantly. You define what "good" looks like (accuracy, compliance, empathy, resolution), and Chanl grades every conversation against those standards. No sampling bias, no inconsistency, no waiting.
A scorecard is a grading rubric. It has categories (like "Policy Compliance" or "Empathy"), each with weighted criteria (like "Did the agent verify identity?" at 10 points, marked critical). Chanl evaluates every conversation against these criteria and produces a score, per-criteria breakdowns, and actionable improvement suggestions.
How Scorecards Work
- You create a scorecard with categories, criteria, and weights
- You attach it to test scenarios or run it against call logs
- Chanl produces an evaluation — a score for every criterion, an overall grade, and a pass/fail result
- You see exactly where each conversation excelled or fell short
Creating a Scorecard
curl -X POST https://api.chanl.ai/v1/scorecards \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Customer Service Quality",
"description": "Grades agent performance on support calls",
"categories": [
{
"name": "Communication",
"weight": 30,
"criteria": [
{
"name": "Empathy",
"description": "Agent acknowledges customer emotions and shows understanding"
},
{
"name": "Clarity",
"description": "Explanations are clear and easy to understand"
},
{
"name": "Professionalism",
"description": "Maintains professional tone throughout"
}
]
},
{
"name": "Problem Resolution",
"weight": 50,
"criteria": [
{
"name": "Issue Identified",
"description": "Correctly understands the customer problem"
},
{
"name": "Solution Provided",
"description": "Offers appropriate solution"
},
{
"name": "Follow-up",
"description": "Confirms resolution and offers additional help"
}
]
},
{
"name": "Compliance",
"weight": 20,
"criteria": [
{
"name": "Required Disclosures",
"description": "Provides all legally required information"
},
{
"name": "Data Privacy",
"description": "Handles customer information appropriately"
}
]
}
]
}'import Chanl from "@chanl/sdk";
const chanl = new Chanl({ apiKey: "YOUR_API_KEY" });
const scorecard = await chanl.scorecards.create({
name: "Customer Service Quality",
description: "Grades agent performance on support calls",
categories: [
{
name: "Communication",
weight: 30,
criteria: [
{
name: "Empathy",
description: "Agent acknowledges customer emotions and shows understanding",
},
{
name: "Clarity",
description: "Explanations are clear and easy to understand",
},
{
name: "Professionalism",
description: "Maintains professional tone throughout",
},
],
},
{
name: "Problem Resolution",
weight: 50,
criteria: [
{
name: "Issue Identified",
description: "Correctly understands the customer problem",
},
{
name: "Solution Provided",
description: "Offers appropriate solution",
},
{
name: "Follow-up",
description: "Confirms resolution and offers additional help",
},
],
},
{
name: "Compliance",
weight: 20,
criteria: [
{
name: "Required Disclosures",
description: "Provides all legally required information",
},
{
name: "Data Privacy",
description: "Handles customer information appropriately",
},
],
},
],
});
console.log(`Created scorecard: ${scorecard.id}`);from chanl import Chanl
chanl = Chanl(api_key="YOUR_API_KEY")
scorecard = chanl.scorecards.create(
name="Customer Service Quality",
description="Grades agent performance on support calls",
categories=[
{
"name": "Communication",
"weight": 30,
"criteria": [
{"name": "Empathy", "description": "Agent acknowledges customer emotions and shows understanding"},
{"name": "Clarity", "description": "Explanations are clear and easy to understand"},
{"name": "Professionalism", "description": "Maintains professional tone throughout"}
]
},
{
"name": "Problem Resolution",
"weight": 50,
"criteria": [
{"name": "Issue Identified", "description": "Correctly understands the customer problem"},
{"name": "Solution Provided", "description": "Offers appropriate solution"},
{"name": "Follow-up", "description": "Confirms resolution and offers additional help"}
]
},
{
"name": "Compliance",
"weight": 20,
"criteria": [
{"name": "Required Disclosures", "description": "Provides all legally required information"},
{"name": "Data Privacy", "description": "Handles customer information appropriately"}
]
}
]
)
print(f"Created scorecard: {scorecard.id}")Scoring Dimensions
These are the four dimensions most teams start with. You can customize, rename, and reweight them for your use case.
Accuracy
Did the agent give correct information? Wrong answers erode trust faster than slow answers. Accuracy criteria check whether the agent's statements match your knowledge base, policies, and facts on the ground.
Compliance
Did the agent follow the rules? This covers required disclosures, policy adherence, data privacy, and anything your legal team says must happen on every call. Compliance criteria are often marked as "critical" — one failure drops the entire score.
Empathy
Did the agent acknowledge the customer's situation? Customers remember how they were treated more than what was said. Empathy criteria evaluate whether the agent recognizes frustration, validates concerns, and responds with appropriate emotional tone.
Resolution
Did the agent actually solve the problem? A polite, compliant agent that doesn't fix anything is still failing. Resolution criteria check whether the issue was identified, an appropriate solution was provided, and the customer confirmed satisfaction.
Scorecard Templates
You don't have to start from scratch. Here are templates for common use cases.
Customer Service
Sales
Technical Support
Using Scorecards
In Test Scenarios
Attach a scorecard when you create a scenario. Every simulation from that scenario gets graded automatically.
curl -X POST https://api.chanl.ai/v1/scenarios \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Refund Request Test",
"prompt": "Customer wants refund for defective product",
"personas": ["frustrated", "analytical"],
"agents": ["agent-v1"],
"scorecard": "customer-service-quality"
}'On Historical Calls
Score calls that weren't evaluated when they happened:
curl -X POST https://api.chanl.ai/v1/call-logs/call_abc123/analyze \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"scorecard": "customer-service-quality"
}'Batch Scoring
Score an entire week's worth of calls in one go:
const results = await chanl.callLogs.batchAnalyze({
filters: {
dateRange: { start: "2024-01-08", end: "2024-01-15" },
},
scorecard: "customer-service-quality",
});
console.log(`Analyzed ${results.count} calls`);
console.log(`Average score: ${results.avgScore}`);
console.log(`Weakest category: ${results.lowestCategory}`);results = chanl.call_logs.batch_analyze(
filters={"date_range": {"start": "2024-01-08", "end": "2024-01-15"}},
scorecard="customer-service-quality"
)
print(f"Analyzed {results.count} calls")
print(f"Average score: {results.avg_score}")
print(f"Weakest category: {results.lowest_category}")Reading Evaluation Results
An evaluation is one scoring run — a conversation graded against a scorecard. Here's what you get back:
{
"overallScore": 87,
"categories": [
{
"name": "Communication",
"weight": 30,
"score": 92,
"contributionToTotal": 27.6,
"criteria": [
{
"name": "Empathy",
"score": 95,
"passed": true,
"notes": "Excellent acknowledgment of customer frustration"
},
{
"name": "Clarity",
"score": 88,
"passed": true,
"notes": "Clear explanations with minor jargon"
}
]
},
{
"name": "Problem Resolution",
"weight": 50,
"score": 85,
"contributionToTotal": 42.5,
"criteria": [
{
"name": "Solution Provided",
"score": 80,
"passed": true,
"notes": "Solution worked but took longer than ideal"
}
]
}
]
}How to read this:
- Overall score (87) is solid but there's room to improve
- Communication (92) is a strength — the agent handles empathy well
- Problem Resolution (85) is the drag, despite carrying 50% of the weight
- The specific weakness is "Solution Provided" — the fix worked but was slow
- Focus your next prompt iteration on faster resolution paths
AI-Powered Improvement Suggestions
Chanl doesn't just grade conversations — it tells you what to fix. After an evaluation, you'll get specific, actionable suggestions based on the patterns it finds:
- "Agents score 15% lower on 'Issue Identification' when customers describe problems using non-technical language. Add examples of common customer phrasing to your prompt."
- "Compliance scores drop when calls exceed 5 minutes. The required disclosure gets buried in longer conversations — consider repeating it at the end."
- "Resolution scores improved 12% after adding the order lookup tool. Consider adding a similar tool for subscription management."
Comparing Agents
Put two agents side by side to see exactly where one beats the other:
curl https://api.chanl.ai/v1/analytics/scorecard-comparison \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"scorecard": "customer-service-quality",
"agents": ["agent-v1", "agent-v2"],
"timeRange": "30d"
}'const comparison = await chanl.analytics.scorecardComparison({
scorecard: "customer-service-quality",
agents: ["agent-v1", "agent-v2"],
timeRange: "30d",
});
for (const insight of comparison.insights) {
console.log(insight);
}comparison = chanl.analytics.scorecard_comparison(
scorecard="customer-service-quality",
agents=["agent-v1", "agent-v2"],
time_range="30d"
)
for insight in comparison.insights:
print(insight){
"scorecard": "customer-service-quality",
"comparison": {
"agent-v1": {
"overallScore": 82,
"categoryScores": {
"Communication": 88,
"Problem Resolution": 78,
"Compliance": 84
}
},
"agent-v2": {
"overallScore": 91,
"categoryScores": {
"Communication": 93,
"Problem Resolution": 92,
"Compliance": 87
}
}
},
"insights": [
"Agent V2 outperforms across all categories",
"Biggest gap is in Problem Resolution (+14 points)",
"Both agents are strong on Communication"
]
}Writing Good Criteria
The quality of your scorecard depends on the quality of your criteria. Here's what separates useful criteria from useless ones.
Be Specific, Not Vague
Bad: "Agent was good"
Good:
{
"name": "Empathy Demonstrated",
"description": "Agent acknowledges customer frustration with phrases like 'I understand that's frustrating' or 'I can see why you're concerned' within the first 30 seconds"
}Include Examples of Pass and Fail
{
"name": "Professional Tone",
"description": "Maintains friendly, professional language throughout",
"examples": {
"good": [
"I'd be happy to help you with that",
"Let me look into this for you"
],
"bad": [
"That's not my problem",
"You should have read the instructions"
]
}
}Make Criteria Actionable
If an agent fails a criterion, the team should know exactly what to change:
{
"name": "Clear Next Steps",
"description": "Agent ends the call by summarizing what will happen next and when the customer can expect resolution",
"passingExample": "I've processed your refund, and you'll see the credit in 3-5 business days. Is there anything else I can help with today?"
}Setting Weights
Weights determine how much each category impacts the overall score. They must add up to 100%.
{
"categories": [
{ "name": "Communication", "weight": 30 },
{ "name": "Problem Resolution", "weight": 50 },
{ "name": "Compliance", "weight": 20 }
]
}When to weight a category higher:
- Compliance — when regulatory consequences are severe
- Problem Resolution — when solving issues is the primary goal
- Communication — when brand perception is critical