The impact of the Covid-19 pandemic on ARCP outcomes issued to trainees in trauma and orthopaedic surgery

By Charlie Barter​​​​​​^a, David Humes^b, Deepa Bose^c and Jon Lund^a

^aDivision of Medical Sciences and Graduate Entry Medicine, University of Nottingham
^bDivision of Epidemiology and Public Health, and Nottingham Digestive Diseases Centre, University of Nottingham
^cDepartment of Trauma and Orthopaedics. Queen Elizabeth Hospital, University Hospitals Birmingham NHS Foundation Trust, Birmingham

 

Introduction

Since being declared a pandemic in March 2020¹, Covid-19 has disrupted the provision of surgical healthcare worldwide. Repercussions can be seen in the provision of surgical training, particularly in specialties with a heavy elective component such as trauma and orthopaedic surgery^2,3. The literature describes the various aspects of these negative effects⁴, from reduction in operating room time^5,6, reduction in other clinical environment exposures^7,8, detriments to trainee mental health^9,10, and impacting the holding of exams and other necessary learning events^3,11.

UK Surgical training follows outcome based curricula written by the Joint Committee on Surgical Training (JCST)¹² and approved by the General Medical Council (GMC). Records of each trainees’ training and assessment are kept on a training management platform: The Intercollegiate Surgical Curriculum Programme (ISCP)¹³. Trainee operative cases are recorded in the Surgical eLogbook¹⁴.  The succession from one training stage to another is recorded at an Annual Review of Competency Progression (ARCP) on ISCP by a panel of surgical educators, against benchmarks described in the surgical curriculum. Several classes of outcome can be awarded at ARCP, each denoting the panels advice on the trainee’s progression for the next year (or less if their continued development is deemed to require further input). Possible outcomes are shown in Table 1.

In recognition of the significant challenges to progression imposed by Covid-19 two new 'non-standard' outcomes were introduced in 2020 as derogations to the eighth edition of the 'Gold Guide'^15,16 (numbered 10.1 and 10.2, Table 1). These two new outcomes indicated 'no fault' on the part of the trainee, i.e. lack of progression was a fault of the training environment. A trainee awarded one would not be prevented from sitting postgraduate examinations, or taking out of programme leave, as they would be with other non-standard outcomes. 

The awarding of non-standard outcomes has been linked in previous studies to some of the protected personal characteristics of the trainee¹⁷, a situation the surgical community is committed to addressing. Orthopaedic surgery remains a male-dominated area of practice, with only 28.5% of doctors in training, and 7.4% of consultants, female as of 31st May 2021¹⁸. As such it is a specialty where attention to differential attainment is incredibly important.

The primary aim of this study was to assess the impact of the pandemic on trainee progression through higher specialty training in trauma and orthopaedic surgery in the UK. Secondary aims include the investigation of use of the new Covid-19 specific ARCP outcomes, and differential effects on non-standard ARCP outcome awarding by demographic factors, working pattern, and region.

Table 1: Possible ARCP Outcomes; adapted from the Gold Guide (Conference of Post Graduate Medical Deans of the UK 2019, 2020)^15,16 and C Hope¹⁷. Outcomes were grouped as 'Standard' (1 and 6) and 'Non-Standard' (2, 3 and 10).

Methods

Study design

This was a longitudinal cohort study using anonymised trainee data taken from the ISCP training management system. It was performed in line with 'Strengthening Reporting in Observational Studies in Epidemiology' (STROBE) guidelines¹⁹.

Cohort derivation

The cohort consisted of all UK higher speciality trainees in trauma and orthopaedic surgery with an ARCP outcome recorded between January 2017 and December 2022.  All ARCP outcomes with complete data were included.  The smaller populations of trainees who did not disclose their sex (19 trainees, 69 ARCP outcomes), and those who had left training without achieving CCT (45 trainees, 171 ARCP outcomes) were also excluded as their size precludes reliable statistical inferences being drawn from them. Excluded records are detailed in Figure 1.

Figure 1: Cohort Derivation Flowchart. ARCPO: Annual Review of Competency Progression Outcome, ISCP: Intercollegiate Surgical Curriculum Programme, T&O: Trauma and Orthopaedic Surgery, Excluded Training Groups include Fixed term posts, locum appointed for training, those who left training without CCT, and post certificate of completion of training. Excluded outcomes include 'Released from Training Programme' (outcome 4), incomplete evidence provided (outcome 5), and 'Out of Programme' (outcome 8).

Exposure variables

Exposure variables included year of ARCP, trainee sex and age group, trainee working pattern, and stage of training.  Age was grouped into those under 30 on starting their higher specialty training (HST), and those over 30. For working pattern groupings ARCPs were divided into those issued to fulltime trainees, and those issued to trainees who were either currently less than fulltime (LTFT) or had worked LTFT previously. A period of LTFT working was assumed to affect more than just the chronologically closest ARCP. Training status at point of data extraction (February 2023, provided as 'still in training' and 'achieved CCT' groups) was used as an indicator of stage of training during the Covid-19 effected years. Those in the 'still training' group were assumed to be earlier in their training during the pandemic, with those having 'achieved CCT' by February 2023 likely in the late stages of training at this point).

Outcome measure

ARCP outcome was the primary outcome measure. Outcomes were grouped into 'standard' (outcomes 1 and 6), 'non-standard without extra training time' (outcomes 2 and 10.1), and 'non-standard with extra time' (outcomes 3 and 10.2). Outcome 5s were excluded from analysis, as their use is as a stay when a trainee needs to provide further evidence of their progress before a formal outcome can be awarded. Outcome 8s; 'out of programme' were also excluded as during periods of 'OOP' trainees are not expected to contribute evidence of in programme progression. 'Removal from the training programme' and 'fixed term placement' (outcomes 4 and 7) were infrequently used (8 and 47 respectively), and thus excluded. Ungrouped outcomes were used to examine the use of Covid-19 outcomes (10.1 and 10.2).

Statistical analysis

Univariable logistic regression analysis was used to investigate the association between potential risk factors and non-standard outcome at ARCP.  Variables producing a statistically significant (p <0.05) result were carried forward to build a multivariate logistic regression model to assess the strength of associations. Trainee gender and age group were considered a priori confounders. Individuals providing incomplete responses were excluded from the analyses to maintain a stable denominator. All statistics were performed using R Studio Version 2023.06.2+561²⁰.

When comparing the effects of year on ARCP outcome 2017 was used as the comparator year. On analysing by region London, as the region with the largest trainee population, was used as the comparator.

Results

This analysis included 7,675 ARCP outcomes issued to 1,417 HSTs in trauma and orthopaedic surgery between January 2017 and December 2022, (Table 2). Female trainees accounted for 21.5% (387) of the population. The population was evenly divided between older and younger trainees, with 50.6% (912) of the group under 30 on starting HST. Fifty-seven-point five percent (1,038) of our cohort had achieved CCT by February 2023 (hence would have been in the later stages of HST during the pandemic).

Table 2: Demographics of Trauma and orthopaedic trainees registered on ISCP, & ARCP outcomes issued by demographic group. Totals do not include missing data for each group.

Whole cohort statististis

Prior to the pandemic standard outcomes ranged between 87.8% and 89.1% (Figure 2). Standard outcomes reduced significantly in 2020 and had not fully recovered to pre-pandemic levels by the end of 2022 (Figure 2). Non-standard outcomes advising extra time in training (Outcomes 3 and, once available, 10.2) rose steadily after 2020, from a pre-pandemic average of 3.2% to 10.0% of outcomes issued in 2022.

Following adjustment for sex, age group, stage of training, and deanery, the Odds Ratio (AOR) of receiving a non-standard outcome was found to peak at 5.47 (95% CI: 4.39-6.81, P <0.001) in 2021 (Table 3).

Figure 2: Trends in Grouped ARCP outcomes received by Trauma and orthopaedic surgery trainees by year. Total outcomes by year; 2017 = 1,247; 2018 = 1,320; 2019 = 1,274; 2020 = 1,306; 2021 = 1,257; 2022 = 1,229.

Trainee protected characteristics

The percentage of non-standard outcomes issued to female trainees was 27.2% (455 outcomes) compared to 24.9% (1,493 outcomes) in the male trainee population. Female trainees did not have an increased odds ratio of receiving a non-standard outcome at ARCP compared to their male colleagues (Adjusted OR 1.10 (0.97 – 1.26), P = 0.149).

Trainees who were older on starting their higher specialty training were more likely to receive non-standard outcomes than their younger colleagues (27.1% of 1,039 outcomes vs. 23.7% of 909 outcomes). After adjusting for other factors older trainees were 1.4 times more likely to receive a non-standard outcome (AOR: 1.39, 95% CI: 1.24-1.56, P<0.001) as shown in Table 3.

186 trainees in this cohort had worked at least one LTFT placement, and a total of 573 LTFT/post LTFT ARCP outcomes were analysed.  Having worked a less than fulltime (LTFT) placement was not associated with an increased likelihood of receiving a non-standard outcome (unadjusted OR 1.17, 95% CI: 0.97-1.42, P = 0.100, as per Table 3).

Table 3: Trauma and orthopaedic surgery: Use of non-standard ARCP outcomes: Univariate and multiple logistic regression. Total n = 3,809.

Stage of training

Those remaining 'In Training' in February 2023 were taken as the baseline group for analysing effects of point in training on ARCP outcome (Table 3). Compared to these, trainees who had completed the training programme by February 2023 (CCT group) were significantly less likely to receive a non-standard ARCP outcome with an adjusted OR of 0.60 (95% CI:0.53-0.67, P<0.001)

Deanery

The adjusted odds of receiving a non-standard outcome based on training region are shown in Figure 3. Five areas were found to have statistically significantly different AORs for non-standard ARCP outcome from the reference region (London). From lowest to highest adjusted OR they were: the East Midlands; (AOR 1.61, 95% CI:1.23-2.10), Yorkshire and the Humber (AOR 2.00, 95% CI:1.56-2.57), the West Midlands (AOR 2.23, 95% CI: 78-2.70), Scotland (AOR 2.24, 95% CI:1.77-2.83), and North-West England (AOR 2.73, 95% CI:2.19-3.40).

Figure 3: Orthopaedic surgery higher specialty trainees’ ARCP outcomes: Adjusted odds ratios of receiving a Non-standard outcome by region relative to London.

Figure 4: Orthopaedic surgery higher specialty trainees’ ARCP outcomes: Adjusted odds ratios of receiving any Covid-19 outcome (10.1-10.2) by deanery, relative to London.

Covid-19 outcomes

Following their introduction, Covid-19 outcomes (10.1 and 10.2) made up 34.3% (437) and 38.3% (500) of all outcomes issued in 2020 and 2021 (Table 4). Their use dropped in 2022 to a proportion of 21.5% (264 outcomes). The adjusted OR of receiving a Covid-19 outcome in 2022 was 0.39 (compared to 2017 ,95% CI: 0.32-0.47, P <0.001).

Neither trainee sex nor age were associated with an increased OR of receiving a Covid-19 outcome (Table 4). Working an LTFT placement was protective against a Covid-19 outcome, as was being later in training during the pandemic (CCT by February 2023 group), with adjusted ORs of 0.74 (95% CI: 0.57-0.96, P= 0.024) and 0.48 (95% CI: 0.41-0.56, P <0.001) respectively (Table 4).

Significant variation in geographical use of Covid outcome use was demonstrated, even following adjustment for other variables (Figure 4). Trainees in only one deanery carried a lower AOR of receiving a Covid-19 outcome than those in London (Kent, Surrey, and Sussex Deanery at 0.35, 95% C 0.22- 0.57). Two deaneries showed no significant difference in AOR for Covid-19 outcome use compared to London; the east of England (AOR 0.91, 95% CI: 0.63- 1.33) and Wales (AOR 1.32, 95% CI: 0.85-2.06). The remaining 10 deaneries all demonstrated a significant increase in AOR of receiving a Covid-19 outcome.