How to score a sales roleplay: scorecards, weights and pass marks that actually predict performance
Score a roleplay on five or six named behaviours, each with a written description of what a 3 and a 5 look like, weight them by role, and set the pass mark before anyone practises. A single 1 to 5 overall grade records how the manager felt, which is why two managers watching the same call disagree. Here is the scorecard, the weights and the pass marks we use, with a free spreadsheet template at the end.
What a scorecard is, and why a 1 to 5 overall grade fails
A roleplay scorecard is a short list of behaviours, each with a scale and a weight, that adds up to one number. The scale is the part teams skip. A 1 to 5 with no description of each point is an impression scale: the rater decides what a 4 means, and the next rater decides something else. A 1992 study in Evaluation Review (Stoskopf, Glik, Baker, Ciesla and Cover) tested a 27-item behaviourally anchored rating scale on 757 nursing assistants across 44 care facilities. Agreement between independent observers and the assistants' own supervisors averaged 0.78, and the authors recommended anchored scales because they reduce subjectivity and personal bias. Look at the range, though: agreement ran from 0.34 to 0.96 across items. Even with anchors, some behaviours are far harder to rate consistently than others, and you should expect the same from your own card.
The hiring research points the same way. Schmidt and Hunter's 1998 meta-analysis, as summarised on Test Partnership's structured interview page, put the validity of structured interviews at 0.51 against 0.38 for unstructured ones, and the Sackett et al. 2022 re-analysis widened that gap to 0.42 against 0.19. Structure is what makes a judgement predict anything.
One honest complication. A 2012 paper in Creative Education by Malau-Aduli and colleagues, with Cees van der Vleuten, compared checklist scores with global ratings in medical OSCE exams and found trained examiners agreed more on the global rating: candidate ability explained 90.2% of the variance in global ratings against 68.3% for the checklist. Read that before you build a twenty-item tick list. The agreement came from trained examiners using a described scale. Anchors plus rater training is the combination that works. Boxes on their own, scored by whoever is free, give you the noise of the impression scale with extra admin.
The criteria that predict outcomes
Keep to the five or six moments where a sale or a service outcome is actually decided. In our cards those are the opening, discovery, presenting price, objection handling and the next step, plus compliance disclosures wherever the call is regulated (financing, insurance, telecom contracts, anything with a cooling-off period). The sales roleplay evaluation form used in the State Farm program at the University of Central Missouri covers almost the same ground in seven sections: Rapport, Value Statements, Identify Needs, Presentation of the quote, Responding to Customer Concerns, Closing and Communication Strategy, each scored 0 to 10 on bands from Not Present to Excellent. It is a decent card. Its weakness is the one most cards share: the bands are labels (Poor, Fair, Good) rather than behaviours, so Good at identifying needs is still the rater's call.
Write each anchor as something you could point to in the transcript. Below is the card we start from for a residential sales rep. A 1 means the behaviour is absent, a 3 is competent, a 5 is what your best rep does without thinking about it.
Weights total 100 so the score reads as a percentage. Compliance is scored 0 or 10 with nothing in between, and a miss fails the attempt whatever the total. More on that under pass marks.
| Criterion | What a 3 looks like | What a 5 looks like | Weight |
|---|---|---|---|
| Opening | States name, company and reason for the visit and asks permission to proceed | Sets an agenda with a time check, confirms who makes the decision and gets a yes before moving on | 10 |
| Discovery | Asks at least three open questions about the problem and the current setup | Uncovers the trigger event, budget range, timeline and what happens if nothing changes, then repeats it back for confirmation | 25 |
| Presenting price | States the price clearly and ties it to one need the customer named | Presents options (good, better, best), builds the value before the number, then stays quiet after the price | 20 |
| Objection handling | Acknowledges the objection and gives one relevant answer | Isolates it (if we solve that, is there anything else?), answers with proof, confirms it is resolved before moving on | 20 |
| Next step | Proposes a follow-up | Books a dated, timed next step with the decision maker and confirms it in writing before leaving | 15 |
| Compliance disclosures | Required disclosures delivered, possibly prompted or out of order | All required disclosures delivered unprompted, in order, in plain words the customer confirms they understood | 10 (0 or 10, no partial) |
Setting weights by role
The criteria barely change between roles. The weights change a lot, and this is where teams go wrong by copying a sales card onto a service queue. For a sales rep, discovery and price carry the result. For a service agent in a call center, the outcome is a resolved issue and a retained customer, so verification, diagnosis and the resolution offer dominate, and presenting price becomes presenting the fix or the retention offer. For a field technician the money is in the recommendation at the kitchen table after the diagnosis, and the risk sits in the safety and compliance items that can cost a licence. Second Nature's FAQ describes its default split as 70% knowledge and 30% style (pace, clarity, energy, filler words), which is a fair reminder that delivery deserves a weight, just a small one.
Two rules when you set your own. No single criterion above 30, or the card measures one skill and calls it readiness. And change weights only between cohorts, never during one, or week 1 and week 6 can no longer be compared.
| Criterion | Sales rep | Service agent | Field technician |
|---|---|---|---|
| Opening and verification | 10 | 15 | 10 |
| Discovery or diagnosis | 25 | 25 | 25 |
| Presenting price, fix or offer | 20 | 15 | 25 |
| Objection handling | 20 | 20 | 15 |
| Next step or resolution confirmed | 15 | 15 | 10 |
| Compliance and safety disclosures | 10 | 10 | 15 |
Setting a pass mark, and why certification needs one
Certification means somebody has decided a rep can talk to a real customer. That decision needs a line, and the line has to exist before the first attempt, or the manager ends up moving it to fit the rep standing in front of them. We use three numbers. 70 is practice-ready: the rep can run the scenario unsupervised. 80 is certified for live calls on that scenario. 90 is the bar for trainers and for anyone who will score others. The compliance item sits outside the total: a missed disclosure fails the attempt at any score. Second Nature's FAQ says organisations "can define specific evaluation criteria and passing thresholds" and that managers can "add manual evaluations for certification purposes", which is the same shape: an automated score up to the line, a human sign-off on top of it.
Pick the line from data rather than taste. Score ten recorded calls from reps you already trust and ten from reps you would not put in front of a customer. The pass mark sits where the two groups separate. If they do not separate, the card is measuring the wrong things and no threshold will rescue it. Then re-check the line every quarter against real outcomes by rep: close rate, average ticket, CSAT, callback rate. If reps who scored 85 close at the same rate as reps who scored 65, the card is telling you something and you should listen.
Require the pass on two attempts against different customer personas. One lucky run with an agreeable buyer proves nothing.
Who should score: manager, peer or AI
Managers know the product and the customer, and they are also the least consistent raters you have. They score between other jobs, they remember the rep's last month, and they soften a 2 into a 3 for someone they like. Peers are cheaper and often harsher, and peer scoring teaches the scorer as much as the scored. AI scores every attempt the same way at two in the morning and has no opinion about the rep. Each has a failure mode: managers drift, peers go easy on each other, and AI is exactly as good as the anchors you wrote for it.
The consistency problem is measurable, so measure it. Run a calibration session: five managers score the same three recorded roleplays blind, then compare. The spread you see is the noise inside every score your team has ever given. The ATD article "Sales Training Needs Role Play and Coaching for True Sales Readiness" (February 2020) argues for "systematic and structured coaching forms" and for coaching "at regular intervals"; a calibration meeting once a quarter is the cheapest version of that. Hyperbound's site shows each yes/no scorecard verdict with a "Why were you scored this way?" explanation that cites lines from the transcript, and Second Nature lets a rep request a score review and lets a manager overrule the AI with their own scoring. Whoever scores, the rep must be able to see which line earned the mark. A number with no evidence attached is an opinion with a decimal point.
Our position: AI for every attempt, a manager for the certification call, and peers for the weekly practice review. Spot-check 10% of AI scores against a manager's score each month and keep a log of the gap. If it widens, rewrite the anchors before you blame the model.
Six mistakes we keep seeing, and a free template that avoids them
Scoring the outcome instead of the behaviour. The simulated customer said yes, so the rep gets a 5. Simulated customers say yes to terrible calls. Score what the rep did.
Twenty criteria. The UCMO form promises 100 total points, yet seven sections at 10 points each come to 70, and the form never explains the difference. That kind of arithmetic drift is what happens when a card grows by committee. Six items, weights that total 100, and nothing else.
Anchors written as adjectives. Confident, engaging and professional are not things a rater can point to. Rewrite every anchor until it names an action a stranger could find in the transcript.
Changing the card mid-cohort. You lose every trend line you were building.
No fail-on-compliance rule. A rep who scores 92 and skips the financing disclosure has failed, and a card that averages that away will eventually put you in front of a regulator.
Scoring once and filing it. The score exists to pick the next scenario. A rep at 2 on objection handling should practise objections tomorrow instead of sitting through another full call.
The scorecard from this page is available as a free spreadsheet template at the end of this page (an Excel file that opens in Google Sheets and Numbers). It has four tabs, counting a one-page how-to. The first is the sales rep card with the 1, 3 and 5 anchors written out in full for every criterion, and a weighted total that calculates itself. The second is a weights tab with the three role presets above and a check cell that turns red when the weights stop adding to 100. The third is a calibration tab where up to five raters enter scores on the same recorded call and the sheet shows the spread per criterion, so you can see which behaviour your managers disagree about most. Copy it, rename the criteria into the language your team uses, and keep the anchors as sentences.
How RepLab does it
RepLab scores every roleplay attempt 0 to 100 against the scorecard you upload, using your criteria and your weights rather than a generic rubric. The scenarios and scoring page describes the scorecard as one of three inputs to a scenario, alongside the customer and the objection, and teams can either write the criteria themselves or have RepLab draft them from an existing card. Each attempt comes back with time-stamped notes, line-by-line feedback and a suggested line to try next time, so the rep can see which moment earned which mark.
Pass marks are set as certification thresholds per scenario, and reps retry until they reach the bar the team set. Managers see who has practised, who has passed and which skill is slipping, by rep, team and location, with a skill heatmap across Opening, Discovery, Price, Financing and Close over the last 30 days. Scores, readiness and certifications sync to Salesforce, HubSpot, Pipedrive, ServiceTitan, Housecall Pro and Jobber, and results export to an LMS or BI tool for the quarterly check against real outcomes described above.
RepLab works for sales and service teams in stores, branches, call centers and the field, in English, Spanish and French. It is SOC 2 Type I and GDPR compliant, hosted on AWS with SSO and MFA, and conversations never train outside AI models. Plans are Team, Growth and Enterprise, priced per rep seat with managers and admins included, with a 10-seat minimum; a quote follows a 30-minute demo, and there is a free roleplay to try first.
Frequently asked questions
Should a sales roleplay scorecard use a 1 to 5 or a 1 to 10 scale?
Use 1 to 5 with a written anchor for each point, or at least for 1, 3 and 5. A 1 to 10 scale without anchors gives raters more room to disagree, and the behaviourally anchored scale research (Stoskopf et al., 1992) shows that the anchors are what cut rater bias, whatever the number of points.
What pass mark should I set for roleplay certification?
Start at 80 out of 100 for live-call certification, 70 for practice-ready and 90 for trainers, with any missed compliance disclosure failing the attempt outright. Then validate the line by scoring reps you trust against reps you would not put in front of a customer, and move it to where the two groups separate.
How many criteria should a sales roleplay evaluation form have?
Five or six, with weights that total 100. Opening, discovery, presenting price, objection handling and next step cover the moments that decide a sale; add compliance disclosures where the call is regulated. Longer forms drift in arithmetic and get scored by impression anyway.
Can AI score a sales roleplay reliably?
It scores consistently, which is a different claim from scoring correctly. AI applies the same anchors to every attempt, so its reliability depends on how well the anchors are written. Spot-check a sample of AI scores against a manager each month and rewrite the anchors when the gap widens.
Who should score roleplays, the manager or the rep's peers?
Both, for different jobs. Peers score weekly practice and learn by scoring; the manager scores the certification attempt. Run a quarterly calibration where several managers score the same recorded calls blind, because the spread between them is the noise in every score you have given.
Written by the RepLab team. Updated November 2026. RepLab is an AI roleplay platform for sales and service teams in stores, branches, call centers and the field, scored on your own scorecard.