National Selection to T&O ST3 posts in England 2020 & 2021

By Mark AA Crowther
Consultant T&O Surgeon, North Bristol NHS Trust
Chairman Selection Design Group & National Selection Lead, T&O SAC

In my role as National Selection Lead on the T&O Specialty Advisory Committee (SAC), I chair the Selection Design Group. This group is made up of senior consultants, current and previous Training Programme Directors, SAC members and representatives of the British Orthopaedic Trainees Association (BOTA) all of whom are experienced in many aspects of training and education with a collective desire to ensure the best potential registrars are chosen to enter higher surgical training. We convene several times a year to scrutinise the most recent process using data released by Medical & Dental Recruitment and Selection (MDRS) group, a body responsible to the Statutory Education Bodies (SEBs), and to improve the design of the following year’s process. We then implement delivery of the process used to select T&O ST3s and have done so on a national basis since 2013. During this time, we have introduced a number of changes and improvements to the large scale interview process undertaken with between 300 and 500 applicants over three or four days at Elland Road football stadium in Leeds. We have removed the infamous ‘killer’ station, have developed the role of simulation with actors and have successfully implemented the use of iPads to allow real time scoring which has facilitated the early recognition of scoring errors. I took over the Chair of the SDG at the wash-up meeting of the 2019 process and felt at the time that we had a fair and robust process which ran smoothly and in which trainees, programme directors and trainers had confidence, this being the case in no small part because of the hard work of my predecessors David Large and James Hunter.

When the COVID-19 pandemic struck in early 2020 the MDRS acted swiftly in declaring that there would be no face-to-face interviews for any medical specialty. We found ourselves in a strong position compared to other specialties given the robust nature of our historical process; our self-assessment form was well established, albeit as part of the portfolio station, and we were able to have confidence in its validity given our knowledge about how well it correlated with the overall score. We made a strong case for not only the use of the self-assessment score further validated with portfolio review, as a sole selection tool given the restrictions imposed on the process by MDRS, but also for the appointment to LAT posts rather than to substantive posts given the extraordinary circumstances surrounding the 2020 process. Neither proposal was adopted however due to the desire for uniformity across all medical specialties and the uncertainty of the rapidly changing global pandemic. The decision of MDRS was to appoint to substantive training posts using unvalidated self-assessment scores.

We looked closely at our previous year’s data to make a careful decision regarding the ‘appointability’ score, the maximum possible being 32. The decision was a score of 21 which we felt carried a reassuringly low risk of appointing an ‘unappointable’ applicant. With a threshold score of 20, the risk would have increased and at 19 that risk would have increased considerably. We predicted that we would have 105 appointable applicants. The 12 questions contained within the Self-Assessment form were also ranked to be used in a tie-break situation and the ranking chosen prioritised clinical over academic achievement.

Applications were up in 2020 with a total after long-listing of 444. These were ranked according to their unvalidated Self-Assessment score and after application of the tie-breakers, adjudication was only required on a single pair of tied applications. On offers-day there was an unfortunate and unintended administrative error leading to a smaller than expected number of offers, but this was recognised quickly, acknowledged by HEE to be related to a data processing issue, and rapidly rectified resulting in 107 offers being made. This number may have been greater, but the Training Programme Directors (TPDs) were rightly conservative in their declaration of ST3 vacancies knowing that they may have ST8s whose post-CCT fellowship plans could be disrupted resulting in late requests for training extensions as 'periods of grace'. Ultimately 96 offers of ST3 posts were accepted.

In autumn 2020 the SDG met and agreed on what should be submitted to MDRS as our Plans A & B (ideal and fall-back scenarios) given the decision that yet again there would be no in person interviews for the 2021 ST3 recruitment process. The self-assessment form was adjusted adding 12 months to the banding for maximum scores in question 1 (total time since foundation programme or equivalent) and question 2 (time spent in T&O posts) to ensure that no applicant was disadvantaged by the events of the pandemic, for example the trainee previously unsuccessful at application who was in a post labelled as T&O, but which in practice provided little T&O experience. We were relieved that our Plan A, with validated self-assessment and a 30-minute video interview, was agreed. Having written questions that could be delivered without repetition over five days, and being confident that we would have assessor panels that could be used over such a period using three question per interview station, we wrote to the HEE Selection Lead expressing our desire to interview all applicants in accordance with the normal process in Leeds. For logistical reasons HEE were only able to support 90 interviews over four days hence the use of a short-list with a maximum of 360 applicants. Following several years of declining numbers, 604 applications were submitted in 2021, the most since the advent of National Selection. After a small number of withdrawals and some long-listing rejections, we still had 571 applications.

A source of considerable confusion about the self-assessment form is the concept and use of the number 'N'. Questions 1 & 2 offer ranges of months into which applicants allocate themselves with respect to total time since Foundation Programme (or equivalent) for Q1 and experience in T&O for Q2. Several years ago the academic brains of the SDG suggested the concept of N which we use in the self-assessment score to ensure that applicants have maintained a productive academic trajectory relating to publications, presentations and audits and QI projects throughout their career. The times that determined N were altered to 0-39 months for N to be 1 in Q1, and to 10-42 months for maximum of 8 points in Q2. By doing this we tried to ensure that Core trainees were not disadvantaged in open competition against the cohort of other doctors who have been failing to progress despite many years of ‘experience’. Whilst this is publicised to all the applicants, many others and indeed many of the applicants, fail to realise that the overall scoring is capped as follows (Q5a is first author publications, Q5b other author publications, Q6 presentations and Q7 audits):

Please note: the value attributed to Q1, ’N’, does not contribute to the overall score. Q1, ’N’, is used to weight Q5a, Q5b, Q6, and Q7.

The values attributed for Q5a, Q5b, Q6, and Q7 are divided by ’N’. A maximum score is also applied to these questions. The overall maximum Self-Assessment score is 32.

Please see the table below for the score per item, weighting applied, and maximum score:

For example: ’N’ for an applicant who responded to Q1 with 40 to 51 months was 2. If they had 10 first author publications, 3 other author publications, 1 presentation and 3 audits the scores awarded for their validated responses to the Self-Assessment forms would be calculated as follows:

Some applicant’s perception that an extra year in unconventional posts may have put them at significant disadvantage is mis-founded provided they have continued to be active and productive in paper writing, presentations and audit and QI projects. The concept that 'N' is in some way a punishment must be dismissed.For example: ’N’ for an applicant who responded to Q1 with 40 to 51 months was 2. If they had 10 first author publications, 3 other author publications, 1 presentation and 3 audits the the scores awarded for their validated responses to the Self-Assessment forms would be calculated as follows:

In February 2021 the SDG, with additional help from some SAC members, met remotely over two days to validate the evidence uploaded by each applicant relating to the 12 questions on the Self-Assessment form. For quality assurance purposes the first session of the process was performed in pairs before breaking out individually. There was real-time discussion of queries, seeking clarity on inconsistencies and consensus decisions made over the course of the meeting. Evidence relating to 6,852 questions was validated with some scores adjusted up or down and justification notes were recorded. This was an impressive undertaking by dedicated T&O surgeons in their own time. Ten days later, a smaller group reassembled remotely for another day to address the 430 appealed questions (6.3% of those possible) with 54% being deemed unsuccessful. A further dozen complaints were considered and addressed. This left us with the final ranked order of the 571 eligible applicants, of which the top 360 were offered video interviews. These were to comprise three ten-minute stations akin to three of the five stations traditionally used in Leeds. The ‘Technical Skills’ and ‘Communication’ (with Actors) stations were felt impractical to deliver virtually and we were mindful that aspects of the latter would naturally be assessed in the remaining Portfolio, Clinical and Prioritisation stations.

The constrains of COVID-19 this year meant that each applicant could only be seen by three interviewers in a half hour interview slot with one asking the questions which were independently scored by their two colleagues from their own home or office. We tried to ensure that the process was as fair as possible with Quality Assurance involving lay reps and SDG members as 'flies on the virtual wall' to confirm the delivery of a consistent process. For each of the four days of interviews we ran ten panels, each with three interviewers, each overseen by an allocated HEE administrative staff member and each having intermittent QA presence, this allowing each panel to perform nine interviews per day. The administrative staff and QA members provided me with feedback and other senior SDG members, also sitting at home in front of screens, were able to see the scores being submitted in real time. This allowed us to address issues with the interviewers immediately by relaying messages back to them via the administrative staff. These related for example to technical errors in submitting scores, and to challenges to decisions when significant discrepancies in marking were identified. There was never any intention to persuade the assessors to change their scores, merely to ensure that they could justify their scores and that this was recorded. Our interview panels comprised experienced assessors who were well versed in the selection process and have received extensive training, for example relating to equality and diversity, so discrepancies were rare, but occasionally scores were modified in accordance with the marking descriptors. 

Over four full days we delivered successful (with minimal logistical and IT glitches) remote interviews for 360 applicants and of those only 20 failed to reach the appointability threshold. This is not a concept peculiar to T&O, HEE requiring every medical specialty, however popular, to draw a line somewhere and ours is an average score of 3.25 out of 5 for each question plus an average Self-Assessment score. This left us with 340 appointable applicants to fill 177 posts – a roughly 50% chance of success if interviewed – but even then the successful applicants are asked to rank regions and sometimes those who have specific geographical preferences may still be disappointed. The standard has been extremely high and whilst the process has not worked out for some it has worked for most. Regarding geographic preferences, we have made it very clear this year that barring unexpected and compelling reasons, we expect all trainees to receive their training in their accepted Programme.

The SDG is acutely aware of the potential for unconscious bias to influence the outcome of the selection process. Under normal circumstances an applicant can emerge from a station in which they have performed poorly, dust themselves down and start afresh in the next station with a new pair of interviewers who have not seen them before. An applicant could see at least ten assessors in the process, and we feel this is preferable to this year’s compromise. In addition, as a craft specialty, the public and future employers understandably assume that some assessment of technical skills is undertaken when selecting the consultant surgeons of the future and whilst Core training does involve such assessment, we strongly believe in the face validity of a technical skills assessment as part of our ST3 interview process. For these reasons we will make a strong case to HEE that we should return in the future to face-to-face interviews, assuming government rules allow.