An analyst opens a draft summary of a supplier contract, produced by an AI assistant in seconds. It reads well. It names a termination clause and a payment term, and it is confident throughout. The analyst used to spend a morning reading the contract to find those things. Now the question is different: is this summary right, and what would they need to check before signing their name under it?
This is an illustrative scenario, not a documented customer case. It captures what is changing in many jobs. Getting to an answer is faster. Deciding whether the answer can be trusted, and who is accountable if it cannot, still sits with the person.
This guide is for heads of L&D and instructional designers planning AI workforce training. It shows how to find the tasks where AI changes the human job, how to build practice around outputs that are sometimes wrong, and how to measure responsible use without promising productivity gains.
Quick facts
| Question | Short answer |
|---|---|
| What changes when people use AI at work? | Producing a first draft or answer gets easier. Verifying it, judging it and owning the outcome do not go away. |
| What should AI training teach? | The checking and judgement a task now needs, plus the domain knowledge required to do it. |
| Can instructions alone fix over-trust? | No. Research on automation bias found it occurred in novices and experts and was not prevented by training or instructions alone. |
| What to measure? | Observed verification on real tasks, with approved boundaries for AI use. |
1. Map how AI changes the job task
Start with one real task, not with a tool. Describe it as it was done before AI and as it is done now, and compare the two column by column.
For the contract example:
- Before: read the full contract, extract the key terms, flag anything unusual, write the summary.
- Now: review an AI-generated summary, check it against the source, decide whether to rely on it, flag anything unusual, take responsibility for the result.
Some steps shrink. Extracting terms may take seconds. Others appear or grow: checking the output against the source, noticing what the summary left out, and deciding when to ask a specialist. The skill demanded has moved from producing to evaluating. In Bloom's terms, which describe the thinking a task asks for and not how it is delivered, the weight shifts towards analyse and evaluate. Our guide to matching learning activities to Bloom's Taxonomy explains how to align the objective, activity and assessment once you know the level.
This comparison is the heart of an honest training needs analysis for AI. It also stops a common mistake: training everyone on a tool when the real change is a new checking duty.
2. Identify verification and judgement demands
Ask three questions about each step in the "now" column.
- What would a wrong output look like here? A fabricated clause, a missed exception, a plausible but outdated figure.
- What evidence would let me check it? The source document, a second system, a colleague, a policy.
- When must I stop and ask someone else? The boundary between "I can verify this" and "this needs an expert".
Researchers have studied what happens when people work with imperfect automated aids for decades. Parasuraman and Manzey's review of that research describes automation bias, where people make omission and commission errors because they lean on a decision aid that is not always right. It found the effect in both novice and expert participants and concluded that it could not be prevented by training or instructions alone (Parasuraman and Manzey, 2010, Human Factors). That research predates today's AI assistants, so treat it as a caution about human behaviour, not as a measurement of any current tool.
The practical lesson for AI training for employees is that telling people to "check the output" is not a design. The conditions around the person matter too: time to check, access to the source, and a workplace that does not penalise them for slowing down.
3. Build practice around unreliable outputs
If the task now involves judging an output, practise judging outputs. Build a small set of cases where the AI response is:
- Correct and complete. The right move is to accept it, and to say why.
- Incomplete. True as far as it goes, but it leaves out something that matters.
- Misleading. Confident, well written, and wrong in a way that is easy to miss.
Learners get each case, a source to check it against, and a choice: accept, correct, or escalate. They must also give the reason. Assess the reasons for agreement or challenge, not confidence alone. A learner who accepts the right answer without checking has not shown the skill, and a learner who challenges everything has not either.
Interactive formats suit this well. Each person commits to a decision and a reason before the group discusses, so over-trust and over-caution both become visible. Self-paced delivery covers the first pass, and a live debrief handles the ambiguous cases where two defensible readings exist. AhaSlides supports interactive learning and moving between live and self-paced delivery, so one case set can run both ways. Check any specific authoring, feedback or reporting function against your own workflow before you plan around it.
4. Preserve prerequisite knowledge
You cannot evaluate an output you do not understand. A person who has never learned how payment terms interact with termination rights will not notice when a summary gets them wrong. Access to AI does not remove the need for that understanding. It changes where it is used.
Decide which domain knowledge each verification task depends on, and check it before practice, with a short self-paced diagnostic. A recall check like this confirms prerequisites only. It does not show that anyone can verify an output. Carnegie Mellon's Eberly Center makes the same design point more generally: assessments, objectives and instructional activities should line up, and when an assessment measures recall but the objectives call for analysis, it fails to measure what learners were meant to learn, and the misalignment can undermine both motivation and learning (Eberly Center, alignment of objectives, assessments and instruction).
5. Measure responsible use on real tasks
Choose a verification task people really do, and set the approved boundaries for AI use before you measure: what may be delegated to the tool, what must be checked, and what must be escalated.
The US Centers for Disease Control and Prevention separates evaluating learning from evaluating how well people apply it back at work, and recommends planning for both (CDC, evaluate training: measuring effectiveness). It is evaluation guidance, not a forecast of results. For AI tasks that means two measures:
- Learning: performance on a new, unseen set of cases, compared with a baseline.
- Transfer: a supervised work task or a sample of real work, reviewed against the same rubric after people have had time to use what they learned.
Define who is included, the denominator and the timing. Report efficiency only from measured evidence. If a team gets faster after AI training, check what else changed in the same period, such as a new template, a staffing change or a different mix of work, before crediting the training.
For a wider view of risk, the National Institute of Standards and Technology publishes the AI Risk Management Framework, released in January 2023, which organisations use to think about AI risk. It is a starting point for the risk conversation, not a training standard or a certification.
A worked example: the contract summary
Illustrative. A practitioner who does this work should validate the case and the answer criteria before use.
The case. The learner receives a fictional supplier contract and an AI-written summary. The summary states a 60-day termination notice. The contract says 60 days for termination without cause and 30 days for breach. The summary also omits a payment-term exception in an appendix.
What the learner is asked to do. Explain the decision (accept, correct or escalate), cite the evidence from the contract, and say what further information or support they would need.
A weak response. "The summary looks accurate and covers the main terms. Accept." It cites nothing and shows no check.
A stronger response. "I would not accept it as it stands. Clause 14 gives 60 days for termination without cause but 30 days for breach, and the summary only mentions 60. Appendix B has a payment exception the summary does not mention. I would correct the termination line and add the exception. I am not sure whether the appendix applies to this supplier, so I would ask the contracts lead."
The retry. The learner gets a second contract where the AI summary is correct. The right move now is to accept and say what was checked. This guards against training people into reflexive suspicion.
An AI task analysis worksheet
Use one row per step in the "now" column for a real task.
| Task step | What AI now does | What the person must still do | Evidence needed to check it | Escalate to | Prerequisite knowledge |
|---|---|---|---|---|---|
| Extract key terms | Drafts the list | Compare with the source | The contract itself | Contracts lead | Contract structure and clause types |
| Flag unusual terms | Suggests candidates | Decide what is unusual in this context | Past contracts, policy | Legal | Standard terms for this supplier type |
| Sign off the summary | Nothing | Own the result | The checked summary | Line manager | Approval limits |
A rubric you can adapt
Score each criterion 0 to 2. This is a proposal to adapt with a specialist in the role, not a validated readiness standard.
| Criterion | Observable evidence |
|---|---|
| Task execution | Identifies where AI changes the verification and judgement task. Records the action and the evidence behind it. |
| Reasoning | Explains the constraints, alternatives and uncertainty. Reasons for accepting or challenging an output are tied to the source. |
| Boundaries | Uses approved procedures for AI use and asks for help when information or authority is not enough. |
Anchors: 0 = absent or unsupported. 1 = partial, with relevant omissions. 2 = complete and justified against agreed criteria. Define critical errors separately, for example signing off an output that contradicts the source, or sharing confidential data with an unapproved tool. A total score must not hide a critical failure.
Constraints that may remain
Better judgement in a training case does not remove every constraint:
- Time pressure. If volume targets leave no room to check, people will skip checking whatever they were taught.
- Unclear accountability. If nobody is named as the owner of a checked output, checking is nobody's job.
- Tool limits. Some tools do not let people see the source behind an answer, which makes verification slow.
- Changing tools. An AI system that changes behaviour after an update can break a verification habit you built last quarter.
Plan separate fixes for these, and say so to sponsors. Training can build the skill. It cannot create the time or the authority.
Frequently asked questions
What should AI training for employees cover?
Start with the tasks AI changes in their role. Cover how to check outputs against evidence, when to escalate, the approved boundaries for AI use, and the domain knowledge needed to judge an answer. Tool tips come second.
Does AI remove the need for domain knowledge?
No. People need enough understanding to notice when an output is wrong or incomplete. What changes is how that knowledge is used: less producing, more evaluating.
How do you assess human judgement in AI-assisted work?
Give learners realistic cases with correct, incomplete and misleading outputs, and score their reasons for accepting, correcting or escalating. Then observe a supervised real task after a suitable delay.
Where to start
Pick one task that people in your organisation now do with AI help. Fill in the worksheet above with someone who does the job, and note the three places where a wrong output would do most harm. Build your first practice case around one of them.
To see how a self-paced check, an interactive case practice and a live debrief could fit together for that task, explore AhaSlides.
