A credit risk model has behaved well for three years. Its back-test is clean, its outputs sit comfortably inside tolerance and the business trusts it. In a training lab, an analyst changes one approved input assumption, the expected recovery rate on a class of loans, to a less comfortable value. The headline output barely moves. Two segments move a great deal. The analyst has to decide what that means, what to tell the model owner and what the model should not be used for.
This is an illustrative scenario, not a documented customer case. It shows the skill that model risk training exercises have to build: testing assumptions, explaining limitations and justifying a decision when a model looks stable. This guide is for heads of risk training and for the compliance and model risk leads who own the answers. It covers training design only. A training exercise does not validate a live financial model, and a model risk specialist should review every case before use.
What are model risk training exercises?
Model risk training exercises are structured practice tasks in which analysts probe a model, usually a simplified or synthetic one, by changing assumptions and inputs, then interpret what happens and explain the limits of what the model can support. A stress-testing lab is the hands-on form of this: a safe model, a set of approved changes and a requirement to justify conclusions.
Quick facts
| Question | Short answer |
|---|---|
| What is the task? | Test assumptions and explain a model's limitations. |
| What do you score? | The quality of the stress tests and the interpretation of what failed. |
| Does the lab validate a live model? | No. It teaches a method on a safe or synthetic model. |
| Who approves the case? | A model risk specialist, not the trainer. |
| Live or self-paced? | Self-paced for concepts and tools, live or facilitated for interpretation. |
| What can training not fix? | Weak governance, no time to challenge and unclear ownership. |
| How do you know it worked? | A new model case or supervised review after a gap, scored against the rubric. |
1. Describe the performance problem
Start with the decision. What must an analyst do when a model looks fine? Write it as one verb-led sentence: "Change an approved assumption, observe what moves, explain the limitation this exposes and say what the model should and should not be used for."
Supervisors name this skill. The US Federal Reserve and the Office of the Comptroller of the Currency issued SR 11-7, Supervisory Guidance on Model Risk Management, on 4 April 2011. It describes model risk as the potential for adverse consequences from decisions based on incorrect or misused model outputs, and it calls "effective challenge" a guiding principle: critical analysis by objective, informed parties who can identify model limitations and assumptions and produce appropriate changes. That is US guidance, not a rule for every jurisdiction, so check the requirements that apply to you. It is a close description of the performance the training should build.
2. Diagnose the learning and context gaps
Compare what the role demands with what your current training practises and assesses.
- What do analysts practise now? If training is a lecture on model governance plus a documentation checklist, they have learned the vocabulary. They have not practised breaking a model on purpose.
- What is the cognitive demand? In Bloom's revised taxonomy, running a test is application, interpreting why the model failed is analysis, judging whether the model is fit for a use is evaluation, and designing a new stress test is creation. Our guide to matching learning activities to Bloom's taxonomy explains how to read these demands.
- Is something other than skill in the way? Check staffing, tools and procedures first. If analysts have no time to challenge, no access to the model's assumptions or no route to raise a limitation, a lab will not change what happens on Monday.
- Is challenge welcome? If challenging a model means criticising its owner, people will stay quiet. Agree how challenge is received before you train for it.
3. Design purposeful preparation and practice
Give each stage its own job.
Preparation: check the prerequisites. A short self-paced diagnostic on the model's purpose, its main inputs and the statistical ideas the lab needs. A recall check like this confirms prerequisites only. Do not report it as capability.
Practice: model stress-testing labs. Build a small, safe model, or use an approved synthetic one, and give learners a menu of approved changes. Three exercises in rising order of demand:
- Change one assumption. The learner moves one approved input and records which outputs move, by how much and which do not.
- Find the segment it breaks. The learner looks below the headline number and finds where the model is least stable, then states what they would not use it for.
- Design the next test. The learner proposes a further stress test, says what result would change their view and what it would cost to run.
Feedback linked to decisions. "You reported the headline change and did not check the two segments that moved most" is more useful than a score. Name the decision, the evidence used or missed and the approved criterion.
Live instructor input or asynchronous review? Live input adds most value when a model risk specialist explains why a limitation matters, or why a test was weak. Asynchronous review is enough for the concepts, for checking test results against a key and for written limitation statements a specialist can read later. A common mix is self-paced preparation, a facilitated lab, then supervised challenge of a real model by a more experienced colleague.
4. Work through a realistic example
Illustrative. A model risk specialist should validate the case and answer criteria before use.
The case. A simplified loan loss model uses an expected recovery rate of 60 per cent. Back-testing is clean. The approved stress changes the rate to 40 per cent for unsecured loans. The portfolio loss estimate rises by 4 per cent overall. Two segments, both small business unsecured, rise by more than 30 per cent.
The task. The learner states what the test shows, explains the limitation and recommends a decision.
A first attempt. "The overall change is only 4 per cent, so the model is robust to this assumption."
Feedback and retry. The facilitator asks which segments moved most, how large they are in the portfolio and what the overall figure hides. The retry asks for the interpretation, the limitation and the recommendation.
A stronger attempt. "The headline figure moves little because the affected segments are a small share of the portfolio. Within them, the estimate rises by more than 30 per cent, so the model is highly sensitive to the recovery assumption for small business unsecured lending. I would not rely on the single 60 per cent figure for those segments. I would recommend segment level recovery assumptions and a documented sensitivity range, and I would tell the model owner I have only tested one input. I have not tested correlation or data quality."
Why each response is defensible or incomplete. Judge each against the approved criteria, not the facilitator's preference. The first attempt reads the aggregate and stops. The stronger attempt finds where the model is sensitive, states a limitation tied to a use, recommends a bounded action and says what remains untested. It does not claim the model is wrong.
5. Assess capability and plan transfer
Evidence of capability is a response to a new model case the learner has not seen. Score a short rubric from 0 to 2 per criterion, defined with your specialist. Our guide to assessing judgement and problem solving in training covers anchors and calibrating assessors.
| Criterion | Observable evidence |
|---|---|
| Task execution | Runs approved stress tests and records the action and the evidence behind it. |
| Reasoning | Explains what failed, the limitation it shows and what is untested, including the interpretation of failure. |
| Boundaries | Uses approved procedures and seeks help when information or authority is missing. |
Anchors: 0 means absent or unsupported, 1 means partial with relevant omissions, and 2 means complete and justified against agreed criteria. Define critical errors separately, for example presenting a stress test result as validation of a live model, so a good overall score cannot hide one. This rubric is a proposed starting point, not a validated readiness standard.
Workplace follow-up. After a suitable period, give a new model case or review a supervised challenge on a live model, and compare it with the baseline on the same two measures: quality of stress tests and interpretation of failure. Define the population, the denominator and the timing in advance, and record which cases people saw in training so you can tell recall from transfer.
What evidence would show the training worked?
Look for analysts who, on a model they have not seen, change an assumption on purpose, look below the headline, state a limitation tied to a use and say what they have not tested. Add what a review of their challenge notes shows: specific limitations rather than generic caveats.
Be careful with outcomes such as the number of findings raised. A new model inventory, a regulatory exam or a change in who reviews the work could move them in the same period. More findings could mean sharper challenge or a changed threshold, so check before crediting the training.
Where interactive delivery helps
Interactive formats help where everyone should commit to a prediction before the group sees the result. In a live session, a facilitator can describe a change, ask each analyst to predict which outputs will move and then reveal the result, which exposes how far intuition and the model differ. The same cases can run self-paced as preparation. AhaSlides supports interactive learning and moving between live and self-paced delivery. Check any authoring, feedback or reporting function you plan to rely on against your own workflow. Treat simulators, virtual reality, AI and system integrations as external or unconfirmed until you have seen them work. A lab on a safe model is practice, not a substitute for the validation of a live one.
What training will not fix
- Weak governance. If nobody owns the follow-up to a challenge, findings will lapse. Agree owners first.
- No time to challenge. People who are measured on throughput will not run extra tests.
- Closed models. If analysts cannot see the assumptions, they cannot test them.
- Evidence limits. A score on a set of cases does not describe every challenge an analyst will make, and no training exercise validates a live financial model.
Name these to sponsors early and agree who fixes each one.
Frequently asked questions
What are model risk training exercises?
Practice tasks where analysts change assumptions in a safe model, interpret what moves and explain the model's limitations and appropriate uses.
Can a training lab validate a live model?
No. The lab teaches a method on a safe or synthetic model. Validation of a live model follows your own governance and the relevant guidance.
Do analysts need to code to run these labs?
Not necessarily. A spreadsheet model or an approved tool with a menu of stress changes is enough. The skill being taught is interpretation, not programming.
How do you assess challenge skills fairly?
Use a new case, a short rubric and agreed anchors, and have a model risk specialist calibrate the scoring.
Where to start
Ask a model risk specialist for one small, safe model and three approved assumption changes. Draft the three-criterion rubric, run the "change one assumption" exercise with one team, then score a new case before you roll it out.
To see how a predict-then-reveal lab like this could run live and be reused self-paced afterwards, explore AhaSlides.
