A credit risk model has behaved well for three years. Its back-test is clean, its outputs sit comfortably inside tolerance and the business trusts it. In a training lab, an analyst changes one approved input assumption, the expected recovery rate on a class of loans, to a less comfortable value. The headline output barely moves. Two segments move a great deal. The analyst has to decide what that means, what to tell the model owner and what the model should not be used for.
This is an illustrative scenario, not a documented customer case. It shows the skill that model risk training exercises have to build: testing assumptions, explaining limitations and justifying a decision when a model looks stable. This guide is for heads of risk training and for the compliance and model risk leads who own the answers. It covers training design only. A training exercise does not validate a live financial model, and a model risk specialist should review every case before use.
What are model risk training exercises?
Model risk training exercises are structured practice tasks in which analysts probe a model, usually a simplified or synthetic one, by changing assumptions and inputs, then interpret what happens and explain the limits of what the model can support. A stress-testing lab is the hands-on form of this: a safe model, a set of approved changes and a requirement to justify conclusions.
I fatti in breve
| La Domanda | Risposta breve |
|---|---|
| Qual è il compito? | Test assumptions and explain a model's limitations. |
| Che punteggio ottieni? | The quality of the stress tests and the interpretation of what failed. |
| Does the lab validate a live model? | No. It teaches a method on a safe or synthetic model. |
| Who approves the case? | A model risk specialist, not the trainer. |
| Live or self-paced? | Self-paced for concepts and tools, live or facilitated for interpretation. |
| Cosa non si può risolvere con l'allenamento? | Weak governance, no time to challenge and unclear ownership. |
| Come fai a sapere che ha funzionato? | A new model case or supervised review after a gap, scored against the rubric. |
1. Descrivi il problema di prestazioni
Start with the decision. What must an analyst do when a model looks fine? Write it as one verb-led sentence: "Change an approved assumption, observe what moves, explain the limitation this exposes and say what the model should and should not be used for."
Supervisors name this skill. The US Federal Reserve and the Office of the Comptroller of the Currency issued SR 11-7, Supervisory Guidance on Model Risk Management, on 4 April 2011. It describes model risk as the potential for adverse consequences from decisions based on incorrect or misused model outputs, and it calls "effective challenge" a guiding principle: critical analysis by objective, informed parties who can identify model limitations and assumptions and produce appropriate changes. That is US guidance, not a rule for every jurisdiction, so check the requirements that apply to you. It is a close description of the performance the training should build.
2. Diagnosticare le lacune di apprendimento e di contesto
Compare what the role demands with what your current training practises and assesses.
- What do analysts practise now? If training is a lecture on model governance plus a documentation checklist, they have learned the vocabulary. They have not practised breaking a model on purpose.
- Qual è il carico cognitivo richiesto? In Bloom's revised taxonomy, running a test is application, interpreting why the model failed is analysis, judging whether the model is fit for a use is evaluation, and designing a new stress test is creation. Our guide to abbinamento delle attività di apprendimento alla tassonomia di Bloom spiega come interpretare queste richieste.
- C'è qualcos'altro, oltre all'abilità, che ostacola il cammino? Check staffing, tools and procedures first. If analysts have no time to challenge, no access to the model's assumptions or no route to raise a limitation, a lab will not change what happens on Monday.
- Is challenge welcome? If challenging a model means criticising its owner, people will stay quiet. Agree how challenge is received before you train for it.
3. Progettare una preparazione e una pratica mirate
Give each stage its own job.
Preparation: check the prerequisites. A short self-paced diagnostic on the model's purpose, its main inputs and the statistical ideas the lab needs. A recall check like this confirms prerequisites only. Do not report it as capability.
Practice: model stress-testing labs. Build a small, safe model, or use an approved synthetic one, and give learners a menu of approved changes. Three exercises in rising order of demand:
- Change one assumption. The learner moves one approved input and records which outputs move, by how much and which do not.
- Find the segment it breaks. The learner looks below the headline number and finds where the model is least stable, then states what they would not use it for.
- Design the next test. The learner proposes a further stress test, says what result would change their view and what it would cost to run.
Feedback linked to decisions. "You reported the headline change and did not check the two segments that moved most" is more useful than a score. Name the decision, the evidence used or missed and the approved criterion.
Intervento in diretta del docente o revisione asincrona? Live input adds most value when a model risk specialist explains why a limitation matters, or why a test was weak. Asynchronous review is enough for the concepts, for checking test results against a key and for written limitation statements a specialist can read later. A common mix is self-paced preparation, a facilitated lab, then supervised challenge of a real model by a more experienced colleague.
4. Analizza un esempio realistico
Illustrative. A model risk specialist should validate the case and answer criteria before use.
Il caso. A simplified loan loss model uses an expected recovery rate of 60 per cent. Back-testing is clean. The approved stress changes the rate to 40 per cent for unsecured loans. The portfolio loss estimate rises by 4 per cent overall. Two segments, both small business unsecured, rise by more than 30 per cent.
Il compito. The learner states what the test shows, explains the limitation and recommends a decision.
Un primo tentativo. "The overall change is only 4 per cent, so the model is robust to this assumption."
Invia feedback e riprova. The facilitator asks which segments moved most, how large they are in the portfolio and what the overall figure hides. The retry asks for the interpretation, the limitation and the recommendation.
Un tentativo più deciso. "The headline figure moves little because the affected segments are a small share of the portfolio. Within them, the estimate rises by more than 30 per cent, so the model is highly sensitive to the recovery assumption for small business unsecured lending. I would not rely on the single 60 per cent figure for those segments. I would recommend segment level recovery assumptions and a documented sensitivity range, and I would tell the model owner I have only tested one input. I have not tested correlation or data quality."
Perché ciascuna risposta è difendibile o incompleta. Judge each against the approved criteria, not the facilitator's preference. The first attempt reads the aggregate and stops. The stronger attempt finds where the model is sensitive, states a limitation tied to a use, recommends a bounded action and says what remains untested. It does not claim the model is wrong.
5. Valutare le capacità e pianificare il trasferimento
Evidence of capability is a response to a new model case the learner has not seen. Score a short rubric from 0 to 2 per criterion, defined with your specialist. Our guide to valutazione del giudizio e della capacità di risoluzione dei problemi nella formazione covers anchors and calibrating assessors.
| Criterio | Prove osservabili |
|---|---|
| Esecuzione dell'attività | Runs approved stress tests and records the action and the evidence behind it. |
| Ragionamento | Explains what failed, the limitation it shows and what is untested, including the interpretation of failure. |
| Confini | Utilizza procedure approvate e chiede aiuto quando mancano informazioni o l'autorità necessaria. |
Anchors: 0 means absent or unsupported, 1 means partial with relevant omissions, and 2 means complete and justified against agreed criteria. Define critical errors separately, for example presenting a stress test result as validation of a live model, so a good overall score cannot hide one. This rubric is a proposed starting point, not a validated readiness standard.
Monitoraggio sul luogo di lavoro. After a suitable period, give a new model case or review a supervised challenge on a live model, and compare it with the baseline on the same two measures: quality of stress tests and interpretation of failure. Define the population, the denominator and the timing in advance, and record which cases people saw in training so you can tell recall from transfer.
What evidence would show the training worked?
Look for analysts who, on a model they have not seen, change an assumption on purpose, look below the headline, state a limitation tied to a use and say what they have not tested. Add what a review of their challenge notes shows: specific limitations rather than generic caveats.
Be careful with outcomes such as the number of findings raised. A new model inventory, a regulatory exam or a change in who reviews the work could move them in the same period. More findings could mean sharper challenge or a changed threshold, so check before crediting the training.
Dove la consegna interattiva aiuta
Interactive formats help where everyone should commit to a prediction before the group sees the result. In a live session, a facilitator can describe a change, ask each analyst to predict which outputs will move and then reveal the result, which exposes how far intuition and the model differ. The same cases can run self-paced as preparation. AhaSlides supports interactive learning and moving between live and self-paced delivery. Check any authoring, feedback or reporting function you plan to rely on against your own workflow. Treat simulators, virtual reality, AI and system integrations as external or unconfirmed until you have seen them work. A lab on a safe model is practice, not a substitute for the validation of a live one.
Cosa non risolverà l'allenamento
- Governance debole. If nobody owns the follow-up to a challenge, findings will lapse. Agree owners first.
- No time to challenge. People who are measured on throughput will not run extra tests.
- Closed models. If analysts cannot see the assumptions, they cannot test them.
- Limiti delle prove. A score on a set of cases does not describe every challenge an analyst will make, and no training exercise validates a live financial model.
Comunicate questi problemi agli sponsor fin da subito e concordate chi si occuperà di risolverli.
Domande frequenti
What are model risk training exercises?
Practice tasks where analysts change assumptions in a safe model, interpret what moves and explain the model's limitations and appropriate uses.
Can a training lab validate a live model?
No. The lab teaches a method on a safe or synthetic model. Validation of a live model follows your own governance and the relevant guidance.
Do analysts need to code to run these labs?
Not necessarily. A spreadsheet model or an approved tool with a menu of stress changes is enough. The skill being taught is interpretation, not programming.
How do you assess challenge skills fairly?
Use a new case, a short rubric and agreed anchors, and have a model risk specialist calibrate the scoring.
Dove iniziare
Ask a model risk specialist for one small, safe model and three approved assumption changes. Draft the three-criterion rubric, run the "change one assumption" exercise with one team, then score a new case before you roll it out.
To see how a predict-then-reveal lab like this could run live and be reused self-paced afterwards, explore AhaSlides.
