Back to the blog
Insight4 min read

Completion Is Not Comprehension

A finished course tells you someone sat through it. It doesn't tell you they can do the job. Here's what to measure instead.

Cover for “Completion Is Not Comprehension”, Insight on the Xebit blog
On this page
  1. 01What a completion rate actually measures
  2. 02The forgetting curve makes it worse
  3. 03What to measure instead
  4. 04What this looks like in practice
  5. 05Start with one program

Every training program ends with a number. Ninety-four percent completion. An average quiz score of 88. Certificates issued. The numbers look like proof, and that is the problem: they prove attendance, not ability.

What a completion rate actually measures

A completion rate counts the people who reached the last screen. It says nothing about whether they can do the task the training was for. Someone can click through a module on refund rules in eleven minutes, pass the quiz by elimination, and route the next $640 refund to the wrong queue on Monday.

Quizzes don't close the gap. Most are written to be passed: four options, one obviously wrong, one obviously right. They test recognition – can you spot the answer when you see it – when the job asks for recall and judgment, under pressure, inside a real tool.

A certificate is a record that someone was in the room. It is not a record of what they can do.

The forgetting curve makes it worse

Even when people do learn, most of it is gone within a week unless they use it. Training is usually delivered all at once, long before the moment it's needed, and then never revisited. The completion rate stays at 94% while the knowledge behind it quietly drains away.

What to measure instead

If the goal is competence, measure competence. That means watching people do the work, not asking them about it. Three shifts make the difference:

  • Tasks, not questions. Ask a new support hire to triage a real queue, not to name the triage rules. Doing it shows you whether they understood.
  • Per person, per skill. An average hides the one person who can't escalate a refund. A score for each skill shows you exactly who needs what.
  • A gap, then a retest. When someone misses a step, the useful output isn't a fail. It's the step they missed, a short retrain on it, and a retest until they pass.

What this looks like in practice

In Xebit, a new hire is taught live on the software they'll actually use, by an AI trainer they can interrupt and question. Then they do the tasks – create the ticket, update the deal, handle the refund – in a simulated copy of that software, while Xebit watches what they do.

The report doesn't say “completed”. It says, for example, that a new hire scored 92% on ticket triage and 58% on escalation rules, because she routed a refund over $500 to the standard queue. She gets a short retrain on that one rule, then a retest, and her manager sees both results.

Start with one program

You don't need to rebuild your whole curriculum to measure comprehension. Pick one program where mistakes are expensive – a compliance rule, a new-hire workflow, an escalation path – and measure what people do instead of what they finished. The difference between the two numbers is the gap your training has been hiding.

Share
All posts →

See it instead of reading about it.

The fastest way to understand Xebit is to watch it run a live session on real software. Give it a minute, or book a pilot on one of your own programs.