A pilot ends. Twelve people took part, nearly all of them said it was useful and the facilitator is pleased. Then the head of operations asks a fair question: "Did anyone do their job differently?" Nobody measured how people worked before the pilot, nobody compared them with anyone, and the team changed its process in the same month. The pilot produced goodwill, not evidence.
This is an illustrative scenario, not a documented customer case. It shows a common way that evaluating training programs goes wrong: the pilot is run, but it was never designed to answer a question.
This guide is for heads of L&D, instructional designers and training managers. It covers five steps: define the pilot question, choose participants and baseline measures, record the learning journey, assess new task performance and transfer, and agree how results will affect rollout. It ends with a template you can adapt and the constraints that remain.
What is a training pilot?
A training pilot is a small, time-limited run of a learning design before you commit to it at scale. Its job is to find out whether the design is feasible and what it is worth pursuing, not to prove it works everywhere.
That distinction matters. The US National Center for Complementary and Integrative Health describes a pilot study as a small-scale test of the methods and procedures to be used on a larger scale, and warns that a pilot is not powered to answer questions about efficacy, so any effect it appears to show is unstable (NCCIH, Pilot Studies: Common Uses and Misuses). That guidance is written for health research, but the logic carries over. A pilot of twelve people can tell you whether you can recruit, deliver, measure and follow up. It cannot tell you, with confidence, how large the effect will be across a whole organisation.
Quick facts
| Question | Short answer |
|---|---|
| What is a pilot for? | Testing learning and delivery assumptions before scale. |
| What do you need before it starts? | A testable question, a task, a baseline and decision criteria. |
| What counts as transfer evidence? | Performance on a new task after work exposure, not satisfaction. |
| Do you need a control group? | Not always. A comparison group or phased rollout helps when it is feasible and fair. |
| When do you set the decision criteria? | Before you look at any results. |
| What can the pilot not tell you? | How large the effect will be at scale. |
1. Define the pilot question
Start with a question that can be answered yes or no, or by a number, and name the decision it will inform.
A weak question: "Does the new onboarding programme work?" A testable one: "After the redesigned onboarding, can new starters handle a standard customer escalation without help, and is the pilot practical to deliver with the coaching time we have?"
Write down three things:
- The task. What will people be able to do that they could not do, or did not do well, before?
- The decision. Will you revise the design, run another pilot or roll out more widely?
- The learning and delivery assumptions. What are you assuming about prerequisites, time, coaching capacity and tools that the pilot should test?
If you cannot name the decision, you are not ready to run the pilot. The result will arrive with no one waiting to act on it.
2. Select participants and baseline measures
A baseline is evidence of how people perform the task before the training. Without it, "improved" has nothing to be compared with.
Choose participants deliberately. Pick people who do the real task and resemble the group you plan to roll out to. A pilot with only your most enthusiastic volunteers will flatter the design.
Capture the baseline. Use the same kind of measure you will use afterwards, so the two can be compared. Options include a short work sample, a structured observation, a review of recent outputs or a scored case. Record who was included, how many were eligible and when you measured.
Think about a comparison. The strongest design compares people who took the pilot with similar people who did not. That is not always possible or fair, especially when the training is needed for safety or compliance. A phased rollout is a practical alternative: one group starts now, a similar group starts later, and the later group acts as a comparison for the first. Whatever you choose, say in the plan why it is fair to those involved.
3. Deliver and record the learning journey
Evidence is only useful if you know what was delivered. Keep a plain log of what happened, because the cost and effort of delivery are part of what the pilot tests.
Record:
- Preparation. What did people do before the sessions, and how long did it take?
- Practice. Which cases or exercises did they work through, and what feedback did they get?
- Coaching. Who coached, for how long, and was it live or asynchronous?
- Operational changes. Anything else that shifted during the pilot, such as a new tool, a policy change, a seasonal rush or a change of manager.
- Product configuration and costs. The actual setup you used, and the time and money it took to deliver.
The last two matter most for the rollout decision. A design that works only with a level of coaching you cannot afford at scale has taught you something valuable, and the log is what shows it.
4. Assess new task performance and transfer
Training evaluation is often described with the four levels of the Kirkpatrick model: reaction, learning, behaviour and results (Kirkpatrick Partners). A pilot should reach beyond the first level. Reaction, whether people found it useful, is quick to collect but says little about whether they will perform differently.
Test learning on a new task. Give participants a case they have not practised, and score it with the same criteria you used at baseline. A new task shows whether people can apply what they learned, not whether they remember an example.
Follow up after work exposure. Transfer means applying what was learned on the job, and keeping it up over time. Baldwin and Ford's often-cited 1988 review defines transfer as both the generalisation of learned material to the job and its maintenance over time, and looks at how training design, trainee characteristics and the work environment affect it (Baldwin and Ford, 1988). So plan a follow-up after people have had real work to do, for example a few weeks later, and measure the same task again.
Record barriers. If someone does not use the new approach at work, ask why. The answer might be a missing tool, a manager who expects the old way or too little time. These are findings about the work environment, not failures of the learner.
Do not infer transfer from satisfaction. A good feedback score is worth recording. It does not show that behaviour changed.
5. Decide how results affect rollout
Write the decision criteria before you look at the results. Otherwise it is easy to find a reason to proceed with whatever you see.
A simple structure:
| If the pilot shows | Then |
|---|---|
| Task performance improved, transfer held, delivery was feasible | Plan a wider rollout and keep measuring. |
| Task performance improved, but transfer was blocked by the work environment | Fix the barrier, then run a second pilot. |
| Little change on the new task | Revise the design before any further run. |
| Delivery cost or coaching time was not workable | Redesign how it is delivered, then pilot again. |
Be honest about what a small group can support. Treat the numbers as indicative, report the group size and the denominator, and avoid effect sizes or guarantees. State which other changes happened at the same time before you credit the training.
A pilot measurement template
Illustrative. A practitioner who does this work should review it before use.
| Section | What to write down |
|---|---|
| Pilot question | The testable question and the decision it informs. |
| Task | The new task people should perform, and how it will be scored. |
| Participants | Who is included, how many were eligible and any comparison group. |
| Baseline | The measure, the date and the scores before training. |
| Learning journey | Preparation, practice, coaching, time and cost. |
| Concurrent changes | Anything else that changed during the pilot. |
| New-task result | Scores on a task not practised during the pilot. |
| Follow-up | Timing, who was reached, and scores after work exposure. |
| Barriers | What stopped people applying the learning. |
| Decision criteria | The thresholds you set before reviewing results. |
| Outcome | Revise, pilot again or roll out, and why. |
Where interactive delivery helps
Interactive formats suit the practice stage of a pilot because each person commits to an answer before seeing the group's, which gives you a record of reasoning rather than only attendance. A facilitator can run a case live, and the same case can be used self-paced for preparation or follow-up. AhaSlides supports interactive learning and moving between live and self-paced delivery. Check any specific authoring, feedback or reporting function you plan to rely on against your own workflow, and treat AI, simulation and integration features as external or unconfirmed until you have seen them work. To choose between formats, see when to use adaptive, interactive and live learning. To match activities to the thinking you want to measure, see our guide to matching learning activities to Bloom's Taxonomy.
Constraints that may remain
Even a well-designed pilot leaves some things unresolved:
- Small numbers. A pilot shows feasibility and direction, not how big the effect will be at scale.
- Work environment. People can perform well on the new task and still not use it at work if their manager, tools or incentives point the other way.
- Timing. A follow-up after a few weeks does not show what remains after six months.
- Concurrent change. Other changes in the same period can explain some of the improvement.
- Selection. People who volunteered may differ from the people you roll out to.
Plan separate fixes for these, and tell sponsors which questions the pilot did not settle.
Frequently asked questions
How big should a training pilot be?
Big enough to test whether the design is practical to deliver and measure, and small enough to change cheaply. There is no single right number. Report the group size and treat any result as indicative.
Do I need a control group to evaluate a training programme?
A comparison group makes the evidence stronger, but it is not always feasible or fair. A baseline plus a phased rollout, where a similar group starts later, is a practical alternative.
When should I measure transfer?
After people have had time and opportunity to apply the learning at work, so give it a few weeks at least. Measure the same task again, and ask about the barriers that stopped anyone using it.
Is satisfaction a useful measure?
It is useful for improving the experience, but it is not evidence of transfer. Pair it with performance on a new task.
Where to start
Pick one task that matters and write the decision the pilot will inform. Capture how a handful of people perform it today, using a short work sample or scored case. Then write your decision criteria before the pilot begins, and book the follow-up date in the diary.
To see how a practice case could run live with your own team, and be reused self-paced afterwards, explore AhaSlides.
