Surgical competency assessment: how to determine when a surgical fellow has reached independent competency

No items found.
Blog thumbnail image

Readiness for independent practice is a judgement built from several observed performances, not one score. A fellow who does well on one simulated case and inconsistently on another has told you something, but not enough. This guide is for directors of medical education and residency programme leads. It covers five decisions: the observable task being assessed, an unfamiliar and representative case, observable criteria with anchors, how to calibrate assessors, and how to use the results. It is about assessment design, not clinical guidance. Authorisation for independent practice stays with your institution's own processes, and a specialist from your programme should review the case and rubric before use.

What the research says, and does not say

Two published methods are worth knowing. In 2005, ten Cate proposed entrustable professional activities: units of work that a supervisor decides a trainee can do unsupervised, based on observed competence. In 1997, Martin and colleagues described the Objective Structured Assessment of Technical Skill (OSATS), which pairs a task-specific checklist with a global rating scale for surgical residents.

Both describe how to structure judgement. Neither shows that a particular fellow is ready, and neither replaces specialist oversight. Use them to shape your criteria, then test the criteria with your own assessors.

Quick facts

QuestionShort answer
What are you judging?Readiness, from multiple observed performances.
What is one simulator score?One data point. It is not authorisation for independent practice.
What makes criteria usable?Observable behaviours with examples of weak, developing and proficient work.
What makes scores trustworthy?Assessors scoring independently, then comparing evidence.
What should you avoid scoring?Confidence or speed alone.
What can it not decide?Authorisation. That sits with your institution's own process.

1. Define the performance being assessed

State the observable task: judging readiness from multiple observed performances. Specify the role, the context and the authorisation boundaries before you define a score. "Independent competency" means something different for a fellow in one procedure, one setting and one level of supervision than for another. Write down what the fellow is being assessed to do, with whom, and what happens if they are judged ready. Our guide to matching learning activities to Bloom's taxonomy helps you separate tasks that need analysis from those that need judgement.

2. Select an unfamiliar, representative task

Choose a case the fellow has not rehearsed, such as a second simulated case that differs from the first in the way real cases do. Explain how it samples the actual work, and what it cannot establish. A single unfamiliar case can show whether skills transfer, but it cannot cover every variation, every team or every complication. That is why a decision rests on several observed performances, spread across cases and time, and not on one.

3. Write observable criteria and anchors

Build criteria around two things: performance consistency across cases and assessor agreement. Write what each score looks like in behaviour. For one criterion, you might describe:

  • Weak (0). Needs repeated prompting, misses a step or cannot explain a choice.
  • Developing (1). Completes the task with omissions or inconsistent quality across cases.
  • Proficient (2). Completes the task, explains choices and handles a variation in the case.

Avoid scores based only on confidence or speed. A confident fellow can be wrong, and a slow one can be careful. Both traits can sit beside a score, but should not replace evidence of what was done. Use the structure Martin and colleagues describe, a task-specific checklist beside a global rating, if it helps your assessors separate steps from overall quality.

4. Calibrate assessors and run a pilot

Calibration is where scores become comparable:

  1. Score independently. Have two or more assessors score the same recorded or observed example without talking.
  2. Compare the evidence. Ask each to say what they saw that led to the score. Disagreement usually sits in an ambiguous criterion.
  3. Revise the criterion. Rewrite the wording or add an example, then score a new example.
  4. Pilot on a small group. Run the assessment with a few fellows before using it for decisions, and check whether assessors still disagree.

Describe the specialist oversight: who reviews borderline results and who owns the final decision.

A worked case with feedback and a retry

Illustrative. The residency programme lead and a surgical specialist should validate the case and the answer criteria before use.

The case. A fellow performs well on one simulated case but inconsistently on another.

The task. Judge readiness. Explain your decision, cite the evidence and say what further information you need.

A first attempt. "The fellow scored highly on the first case, so they are ready." A reviewer asks what happened on the second case, how many performances the judgement rests on, and whether the two assessors agreed. The learner has no answer to any of them.

Feedback and retry. The reviewer notes that one strong score was treated as a verdict and the inconsistency was ignored. The learner is asked to reconsider.

A stronger attempt. The learner states that two cases are not enough, notes the inconsistent second case and asks what differed in it. They propose a further observed performance on a third unfamiliar case, an independent second assessor, and targeted preparation on the step that went wrong. They say they would not recommend a readiness decision yet, and what they still need: the assessors' evidence, the oversight process and the programme's definition of independent practice.

Why each response is defensible or incomplete. Judge against approved criteria, not the reviewer's preference. The first attempt treats one score as authorisation, which is a critical error. The second uses several performances, separates evidence from inference and states the limits of the decision.

5. Use the results for follow-up

Choose the next step from the observed gap:

  • Targeted preparation when a concept or step was missing.
  • Interactive practice when the skill is there but inconsistent.
  • Supervised coaching when the fellow needs feedback in real work.

Plan reassessment, and say what the result cannot do: a readiness judgement describes performance on the cases observed, with the assessors involved, at one point in time.

Criterion0: absent or unsupported1: partial, with omissions2: complete and justified
Task executionJudges readiness from one score.Uses several scores without comparing them.Judges from several observed performances and records the evidence.
ReasoningGives no reason.Gives a reason without naming inconsistency.Explains alternatives, constraints and uncertainty.
BoundariesTreats a score as authorisation.Notes limits without a plan.Uses the approved process and seeks specialist input when evidence is insufficient.

This rubric is a proposed starting point, not a validated readiness standard. Define critical errors separately, for example presenting one simulator score as authorisation for independent practice. An overall score must never hide a critical failure.

To check the process, compare baseline and unseen follow-up tasks. Measure performance consistency and assessor agreement. Define the population, denominator and timing before you start, and look for concurrent changes, such as a new case mix or a change of assessors, before you credit the training.

Where interactive delivery helps

Interactive formats help in the calibration step. In a live session, you can show an example and ask each assessor to submit a score privately, then display the spread before discussing the evidence. Self-paced delivery suits the criteria review and prerequisite checks beforehand. AhaSlides, an all-in-one engagement platform for teams and classrooms, supports interactive learning and both live and self-paced delivery. Check how you would run the calibration in your own workflow before you rely on it.

What a competency assessment will not fix

  • A vague definition of independence. Without one, scores have nothing to be compared with.
  • Too few observations. One or two cases cannot show consistency.
  • Assessors who never calibrate. Agreement does not appear by itself.
  • Evidence limits. The two sources above describe methods, not outcomes for your fellows.

Frequently asked questions

What is surgical competency assessment?

Judging, from observed performance against defined criteria, whether a trainee can do a defined surgical task to the standard your programme requires.

Is one simulator score enough to decide readiness?

No. It is one data point. A readiness decision rests on several observed performances and your institution's approved process.

What is OSATS?

The Objective Structured Assessment of Technical Skill, a method that combines a task-specific checklist with a global rating scale.

How do you check that assessors agree?

Have them score the same example independently, compare their evidence and revise any criterion where they differ.

Who should review the case and rubric?

A surgical specialist and the residency programme lead, before anyone is scored.

Where to start

Pick one procedure and write the observable task. Choose two unfamiliar cases, have two assessors score each independently, and compare their evidence. Revise the criteria where they disagree, then pilot with a few fellows before using the scores for any decision.

To see how a calibration session could run live or self-paced, explore AhaSlides.

Subscribe for tips, insights and strategies to boost audience engagement.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Check out other posts

AhaSlides is used by Forbes America's top 500 companies. Experience the power of engagement today.

Get started free
© 2026 AhaSlides Pte Ltd