What this automation does
Instead of managers reviewing a few calls a month chosen at random, every call gets a first-pass review, and managers spend their time on the calls that teach the most.
- Collect calls. Recordings and transcripts arrive from Zoom, your phone system or a dialer after each call, tagged with rep, call type (discovery, demo, renewal) and CRM deal.
- Prepare the transcript. If the source lacks speaker labels, transcribe with diarization and identify which speaker is the rep using the rep’s known voice channel or name.
- Compute talk metrics. Code calculates talk ratio, longest monologue, questions asked by the rep, and silence gaps from timestamps.
- Score against the scorecard. The model receives the transcript and the scorecard for that call type, and returns met, partly met, or not met for each criterion, with a quoted line and timestamp as evidence.
- Flag coaching moments. Calls with a missed critical criterion, a competitor mention, or a strong example of a skill are queued for manager review with links to the exact moment.
- Roll up weekly. Each rep and manager gets a short weekly view: trends by criterion, two calls worth reviewing, and one example of good practice from a teammate.
When it is worth building
It is worth building when calls are recorded, a written sales methodology exists, and managers cannot listen to enough calls to coach consistently. It is also worth it when you must verify scripted disclosures across every call, not a sample.
It is not worth a custom build if you have no scorecard yet; write one first and test it by hand on ten calls. If your team already pays for a conversation intelligence product and managers use it, improve adoption before building something parallel. Our custom agent vs off-the-shelf comparison lays out that decision more broadly.
Architecture
| Layer | What we use | Why |
|---|---|---|
| Call sources | Zoom cloud recordings, Aircall or RingCentral via their APIs | Pull calls from wherever reps actually talk to buyers |
| Transcription | Source transcripts, or a speech-to-text API with diarization and word timestamps | Accurate speaker turns are the foundation of every metric |
| Talk metrics | Code over timestamps and speaker labels | Exact, cheap and consistent call to call |
| Scoring | Claude Sonnet 5.5 or OpenAI’s GPT models with a per-criterion schema | Judgment criteria with quoted evidence a manager can verify |
| Storage | PostgreSQL with calls, scores, evidence and calibration tables | Trend analysis and an audit trail of every score |
| CRM link | HubSpot or Salesforce deal and owner lookup | Ties coaching to deal outcomes |
| Manager view | A simple internal dashboard or weekly email and Slack digest | Managers need a short list, not another tool to learn |
How to build it
- Write the scorecard per call type. Six to ten criteria each, phrased as observable behaviors (“asked about current process before demoing”), with examples of met and not met. Mark which are critical.
- Score twenty calls by hand. Have two managers score the same calls independently. Where they disagree, the criterion is unclear. Rewrite it before any AI is involved.
- Build ingestion and transcription. Pull calls with metadata, ensure speaker labels, and drop calls below a minimum length or with poor audio quality.
- Implement metrics in code. Talk ratio, monologues and questions from timestamps. Validate on a few calls by listening.
- Prompt per criterion with evidence. Require a quote and timestamp for any met or not met rating, and “insufficient evidence” when the call did not reach that stage.
- Calibrate. Each month, managers score a sample blind and you compare with AI scores. Track agreement per criterion and adjust prompts or criteria where agreement is low.
- Launch transparently. Explain to reps what is scored, why, and how scores are used. Start with coaching only, and give reps access to their own calls and evidence.
What breaks in production
- Scores without evidence. A “not met” with no quote is unreviewable and erodes trust. Reject any score that lacks a valid timestamp.
- Gaming. Reps learn to say the phrases the scorecard looks for. Phrase criteria around the prospect’s response (for example, “prospect described their current process”) rather than the rep’s words.
- Wrong speaker assignment. If the rep and prospect are swapped, every metric inverts. Verify rep identity per call and discard calls where it is uncertain.
- Scoring calls that never reached a stage. A 6-minute reschedule call fails every discovery criterion. Classify call type and stage before scoring.
- Model or prompt drift. A model upgrade quietly shifts scores up or down across the board. Pin model versions and re-run a fixed calibration set before switching.
- Trust collapse. If scores appear in a performance review before reps understand them, adoption stops. Keep the first quarter coaching-only.
Timeline and cost
A first version with one call source, two call types and a weekly digest typically takes four to six weeks, including scorecard calibration. Adding more sources, a dashboard and compliance checks takes six to eight. Lumeor builds sales call coaching systems in the range of $12,000 to $35,000, within our AI integration pricing. Running costs are transcription billed per audio minute and model usage per scored call, which together scale directly with call volume; estimate them from your monthly recorded minutes before committing.
How to measure success
- Share of calls receiving a first-pass review (versus the small sample managers reach by hand)
- AI and manager agreement rate per criterion on the monthly calibration sample
- Manager hours spent finding calls to review, before and after
- Trend in critical-criterion misses per rep over a quarter
- Ramp time for new reps to reach target scorecard performance