Factors Predictive of Performance on USMLE Step 1: Do Commercial Study Aids Improve Scores?

Introduction

The United States Medical Licensing Examination (USMLE) Step 1 is a required exam for allopathic medical students and is the first of three board exams for physician licensure in the USA. In the 2016 National Resident Matching Program (NRMP) survey asking residency directors what factors are considered for granting applicants residency interviews, USMLE Step 1 was cited by 93% of respondents making it the most cited factor [1]. Studies have shown that medical students with higher USMLE Step 1 scores are more likely to match into more competitive residencies, such as dermatology, surgical subspecialties, and otolaryngology consistent with NRMP-reported Step 1 score means for matched applicants [2–4].

Many studies indicated pre-clinical curriculum performance in medical school was strongly correlated with USMLE Step 1 results [5–8]. Previous studies have shown statistically significant relationships between exam preparation with practice questions and USMLE exam performance [5–7, 9, 10]. Generally, books and preparatory courses have not been consistently shown to have significant association with higher USMLE Step 1 scores [9]; however, a student-initiated preparatory course at the University of Illinois Chicago utilizing reading assignments from First Aid, student run review sessions, and practice questions did demonstrate efficacy in increasing test scores [11]. Additionally, a study done at Wake Forest showed that a seminar series emphasizing process-oriented preparation was helpful in decreasing the anxiety surrounding the USMLE Step 1 exam [12].

The overall purpose of our study was to identify predictors of performance on USMLE Step 1. Unlike other studies focusing on metrics such as undergraduate grade point average or Medical College Admissions Tests [13], we opted for a predictive model based on medical school performance and commercial study aids. We sought to determine the relationship between practice question utilization, practice test utilization, and pre-clinical academic performance in our curriculum with performance on USMLE Step 1 in an effort to provide students with evidenced-based advice for preparation. Additionally, we investigated the impact of preparation time and its relationship to USMLE Step 1 performance.

Methods

Participants in this study were recruited from the medical school classes completing their second year in June of 2016 and 2017. The medical students attended a state-funded, public medical school in the Midwest. Each class matriculated 132 students for a total sample size of 264.

Our medical students take USMLE Step 1 between May and June following successful completion of their second year. Students are required to complete the exam before the junior clerkships that begin in July. The College of Medicine provided a 90-day subscription to UWorld, a question bank containing approximately 2400 practice questions. At the request of the medical students, part of their annual technology fees were used to obtain 90-day subscriptions to UWorld for every student.

The curriculum during the first 2 years followed a traditional format. Year 1 covered normal anatomy, biochemistry, physiology, and neurosciences. The second year focused on pathophysiology of disease organized around organ systems and included pharmacology. Delivery of curricular content was primarily through lectures, with less than 10–20% being problem-based learning, small group, and lab activities depending on individual core classes. For individual courses, students’ grades were determined by final percentage that was weighted on exam performance (approximately 90%) and small group or lab activities (10%). Using the final percentages for courses, an overall 2-year cumulative percentage was calculated. The overall percentage is what we used for this study as it reflected performance across 2 years.

At the time of the required orientation to the junior clerkships, the Office of Medical Education administers a post-Step 1 survey to better understand how students prepare and the resources they used. This survey was locally developed and has been in use for 10 years. It has been revised regularly to reflect current resources students are using as well as to obtain feedback on new initiatives in the curriculum designed to help with USMLE Step 1. In 2016, the survey was modified to include specific questions about target and expected scores, numbers and types of practice exams taken, and completion rates for UWorld (see Appendix 1 for the survey). The survey was administered via SurveyMonkey with two reminders to complete it. All students surveyed in two consecutive classes of second year medical students were included.

The two primary sources of practice examinations included UWorld USMLE STEP 1 Qbank and the National Board of Medical Examiners Comprehensive Basic Science Self-Assessments (CBSSA). UWorld USMLE STEP 1 Qbank also offers a question bank containing 2400 USMLE Step 1 style questions. The overall academic performance percentage based on courses from the first 2 years of medical school as well as USMLE Step 1 scores was obtained and matched to survey responses. First-time USMLE examination date was also collected to calculate the amount of dedicated study time that was used to prepare for the exam. Data was de-identified for analysis. The institutional review board deemed this an exempt study.

Data was analyzed using descriptive statistics. Correlational and regression analyses were used for continuous variables. Categorical items were analyzed using analysis of variance. IBM SPSS v24 was used to analyze results.

Results

In total, 243 medical students were invited to complete the survey (2016: n = 118; 2017: n = 125). A total of 170 surveys (2016: n = 74; 2017: n = 96) were completed (response rate 69.96%). The mean Step 1 score for those surveyed was 229.3 with a standard deviation of 20.6 (Table 1).

To answer our first research question, survey responses for completion of the two study resources students used most often included UWorld USMLE STEP 1 Qbank and First Aid for the USMLE Step 1. As seen in Table 2, 98 (62.4%) of the respondents completed UWorld at least once and 126 (81.8%) completed First Aid at least once.

A one-way between subjects analysis of variance (ANOVA) was conducted to compare the effect of study aid completion on USMLE Step 1 scores. Completion rates were reported as some questions, half, most, completed once, or completed twice. For UWorld, there was a significant effect of completion rates on USMLE Step 1 scores at the p < .05 level (F_4,134 = 10.29, p = .001) (Fig. 1). Post hoc tests revealed that completing UWorld questions at least once was associated with higher USMLE Step 1 scores compared with completing a few questions (t = 27.04, p = .036) or about half (t = 24.80, p = .001). Post hoc tests for students completing UWorld two times had higher scores compared with those completing a few questions (t = 31.73, p = .015) or half (t = 29.49, p = .001). For First Aid, there was no statistically significant effect of completion rates on exam scores (2016: F_3,61 = 2.16, p = .102; 2017: F_2,43 = .46, p = .635).

Our second question sought to determine if the number of practice exams completed impacted performance on USMLE Step 1. Ninety-nine percent of the respondents completed at least one practice exam, either UWorld or CBSSA (Table 1). ANOVA was conducted to determine if there is a statistically significant difference in scores based on the number of practice exams taken. There was no statistically significant effect based on the number of practice exams taken on USMLE Step 1 scores (F_7,51 = .71, p = .663).

Our final question asked if academic performance was associated with USMLE Step 1 performance. In aggregate, USMLE Step 1 scores were significantly correlated with overall academic percentage (r = 0.719, p < 0.01). Further analysis by class also indicated overall academic percentage correlated with USMLE Step 1 scores (2016: r = .72, p < .01; 2017: r = .75, p < 0.01) (Fig. 2). Additionally, a multiple regression model was conducted using overall academic percentage and number of times UWorld was completed. The model was significant (F_2,136 = 96.14, p = .001, R² = .586); overall academic percentage contributed to the model with B = 2.58, t = 10.94, p = .001, and UWorld completion was B = 6.52, t = 5.54, p = .001.

We also measured dedicated study period days from the last day of scheduled class to a student’s test date. Overall, there was a negative correlation in number of dedicated study days with USMLE Step 1 exam scores (r = − .225, p = .003). In the 2016 cohort, there was no correlation between amount of days in the dedicated study period and performance (r = − .04, p = .759). With the 2017 cohort, a negative correlation was demonstrated (r = − 0.304, p < 0.01), with mean scores decreasing past 6 weeks. There was a negative correlation between overall academic percentage and number of days dedicated to studying for the exam (r = − .292, p = .001). Aggregate data is summarized in Fig. 3.

Discussion

The quest to identify the best study aids to achieve a high score on USMLE Step 1 eludes medical students. Our study indicates that overall academic performance in conjunction with completing UWorld practice questions most strongly correlated to USMLE Step 1 performance. Our findings are similar to those of another study indicating that medical students who completed at least 2000 questions performed better on USMLE Step 1 [9, 10].

What is unique with our study is our analysis of practice exam usage to prepare for USMLE Step 1. Both UWorld and the National Board of Medical Examiners offer practice exams. These exams are designed to reflect a realistic experience of taking the actual examination. With our findings that completing UWorld questions was associated with higher scores, we hypothesized that taking more practice exams would also be linked to better performance. However, we found no statistically significant effect based on the number of practice exams taken.

These results should not be taken as discouraging use of practice exams. Testing is a good method of learning material [14]. In fact, taking a practice test before undertaking a plan of study can inform medical students what material they recall and what they need to spend more time studying. They are also useful at providing an experience of actually taking the test, which may result in ameliorating anxiety about the examination experience.

The results of USMLE scores in residency applications are a concern for medical students. Based on other studies, USMLE scores are associated with performance on future board and in-training exams in a variety of specialties [15–23]. In a recent study involving 15 pediatric programs, USMLE Steps 1 and 2 results were positive predictors of performance on the American Board of Pediatrics certifying exam [23]. Although there are associations between USMLE Step examinations and specialty boards, it has also been argued that high scores on specialty boards can be achieved regardless of USMLE scores [22]. Some are now pleading with residency programs to put less weight on this exam in residency applicant selection [24].

Reports have varied as to whether higher scores on USMLE Step 1 correlate with improved faculty evaluations during residency training [3, 22, 25, 26]. A recent study examining predictors of success in first year residents over a variety of specialties found Step 2 CK, National Board of Medical Examiners (NBME) Medicine Exam, and clinical performance evaluations to be significantly correlated with performance, while USMLE Step 1 score was not [27].

Our study has some limitations. This study was conducted at a single institution in the upper Midwest. Most of our students are from our state, which limits the diversity. Some of our questions were vague and unable to provide useful data. We asked students how many hours per week they studied, which yielded ineffective data. We also were limited in that we asked students questions about utilization in loosely quantitative terms, “some,” “half,” “most,” “once,” or “more than once,” which does not allow for precision. However, with 2 years of data, we believe these results can be used by other medical schools to offer guidance about how to effectively prepare for USMLE Step 1.

Conclusions

With the competitiveness of some residency programs, performance on USMLE Step 1 is a concern for medical students and medical educators. For medical schools like ours, using locally developed pre-clinical examinations, providing medical students guidance on which preparatory resources to use, is important. Based on our findings, performance in our curriculum was a better predictor of USMLE Step 1 performance. The commercial products designed to help students are useful, primarily as a means of offering more complex questions, which is useful for medical schools using locally developed examination questions. Based on our findings, medical students’ academic performance was a better predictor of USMLE Step 1 performance, complemented by use of practice exams.

Compliance with Ethical Standards