Predictive validity of pre-admission assessments on medical student performance

Introduction

The medical school admission process is an ever-growing topic on which the literature is voluminous. Scrutiny and filtration of student applicants have become more meticulous than ever before due to the increasing number of applicants, the increasing proportion of outstanding ones, and the continuous strive for maintaining the highest possible standard of students. Studies have questioned the predictive validity of the various pre-admission variables on college academic performance. The necessity of further validation and refinement of pre-admission variables that are invalid or weakly predictive has been established in several studies.^1-3

In Saudi Arabia, medical schools depend on a composite of pre-admission variables in their student selection. The main pre-admission variables currently being used are the cumulative high school average, the National Achievement Test score (NAT, locally known as Tahsili), the General Aptitude Test score (GAT, locally known as Qudrat), in addition to the English language proficiency exam score.

Conventionally, pre-admission assessments aim to cover academic and non-academic cognitive variables in addition to English language proficiency. Firstly, academic achievement assessment tools cover some forms of cognitive skills, mainly the student's knowledge base.⁴ According to the literature, this category is believed to be the most reliable in predicting academic performance in medical school.^5-10 Such tools include high school averages, Medical College Admission Test (MCAT), A-levels, and SAT Subject Tests. In Saudi Arabia, the NAT is used in assessing the cumulative knowledge acquired throughout the final three years of high school in five subjects: biology, chemistry, physics, mathematics and English.^{11, 12}

Secondly, there is a general literature agreement in regards to the significance of cognitive non-academic assessment tools in selecting medical students.^13-16 These cover a wide range of competencies that are essential for success in the field of medicine, particularly in the clinical years, such as problem-solving, critical reasoning and quantitative skills.^10,17 Their predictive validity, however, is uncertain.^1-3 Examples of such tools include the United Kingdom Clinical Aptitude Test (UKCAT) and the Undergraduate Medicine and Health Sciences Admission Test (UMAT). In Saudi Arabia, non-academic assessment is achieved through the GAT, which is composed of two sections: verbal and quantitative. The GAT aims to assess reading and comprehension abilities, understanding logical relationships, problem-solving skills based on mathematical concepts and culturally acquired knowledge.¹⁸

English has become the language of choice at many international colleges, the written language of most of the top leading journals and medical publications¹⁹ and of course international conferences. Without a fundamental command of the English language, students would most likely not be able to fulfill their assignments or study goals and thus not be able to attain their maximal performance potential as the lack of adequate language competence could act as a major limiting factor. Tools that assess English language proficiency cover various aspects of the language domain, including reading comprehension, writing, listening, and speaking. Several studies have shown that English language proficiency affects academic performance among non-natives as with the case in Saudi Arabia and other Middle Eastern countries.^20-23 Moreover, this has also been a concern among natives in medical schools of English speaking countries such as Australia, as reported by Hayes and colleagues.^20-23 Two of the most widely used assessment tools for English proficiency are the Test of English as a Foreign Language (TOEFL) and the International English Language Testing System (IELTS). These are now standard exams used and recognized worldwide by thousands of institutions.^{24, 25} Despite their popularity, these exams have not been used in predicting medical school performance in previous studies.

Predictive validity studies commonly use college performance indicators in examining pre-admission variables.^{6, 26} A widely used college performance indicator is the cumulative Grade Point Average (cGPA) by virtue of its perceived value in reflecting students' in-course college performance. Generally, cGPA is considered a valid and reliable tool in university settings;^{27, 28} however, this has not been confirmed specifically for medical colleges yet. It also bears some flaws that may limit its reliability and validity, like grade inflation and institutional differences in grading.²⁷ Nevertheless, to date, there has not been any better significant indicator of student performance than the cGPA, which henceforth makes it the most extensively used tool in academic research to correlate pre-admission variables with students' college performances.²⁷

Another assessment method that has recently emerged as an indicator of student performance is the progress test (PT). Unlike the cGPA, the PT is a longitudinal assessment method that assesses student achievement and progression.^{29, 30} It was first introduced in Maastricht Faculty of Health in Netherlands as a tool to assess its problem-based learning (PBL) curriculum in the early 1970's.^31,32 Since then, medical educationists have documented several advantages of the PT. For instance, the PT, which focuses on functional knowledge (knowledge that comes with repetition), provides valuable information that can be used for feedback and has research potential that can be used in education.³² This contributed to the development, implementation and rapid spread of various types of PTs in many medical schools worldwide; including in South Africa,³³ Indonesia,³⁴ Germany,³⁵ and the US.³⁶ A national PT is being administered in Saudi Arabia by the college of medicine at Qassim University and is offered to all medical colleges willing to participate.³⁷

The aim of this study is to examine the predictive validity of the Saudi national pre-admission variables and English language proficiency standardized tests on medical students' performance using the PT and cGPA as indicators of student performance.

Methods

Participants, Setting, and Procedure

We conducted a retrospective study at the College of Medicine at Alfaisal University in Saudi Arabia, where we collected data of 737 students from all academic years through the college's Student Information System. Collected data included student demographics such as student academic year and gender; student pre-admission variables, namely: the cumulative high school average, NAT, and GAT; English proficiency test scores, namely: TOEFL and IELTS; and college academic performance indicators, namely: cGPA and PT scores.

The required ethical approval was obtained from the Institutional Review Board of Alfaisal University in Saudi Arabia. To ensure confidentiality, the data was coded by the main investigator in line with institutional ethical guidelines. Following the data coding step, statistical analysis was performed.

Descriptive analysis was performed. Multiple linear regression analyses were used to devise models that assess the predictive relationship of five pre-admission variables (namely, cumulative high school average, NAT, GAT, IELTS and TOEFL) on their cGPA and PT results, in both preclinical (years 1, 2 and 3) and clinical years (years 4, 5 and 6). All the analyses were carried out using IBM Statistical Package for the Social Sciences (SPSS) software, version 20. A p-value of <0.05 was considered significant.

Results

Data was available for a total of 737 students. The demographic features of our study subjects are demonstrated in Table 1. The mean and standard deviation of pre-admission and college performance indicator scores are shown in Table 2.

Factors predicting the cGPA

Two separate models were analyzed to study the ability of the pre-admission variables to predict students' cGPA as seen in Table 3. One multiple linear regression was calculated to predict cGPA in preclinical years based on the five pre-admission variables. A significant regression equation was found (F_(5,5)=10.729, p=0.01), with an R² of 0.951. The regression further revealed that the NAT (p=0.04, B=0.08, t=2.752) and TOEFL (p=0.017, B=0.01, t=3.527) scores were both significant positive predictors of students' cGPA in preclinical years of medical school, with the NAT being a stronger predictor. However, GAT was found to be a significant negative predictor of cGPA in the same group (p=0.048, B=-0.05, t=-2.606). Cumulative high school average and IELTS were not significant predictors of preclinical cGPA (p=0.17, p=0.691 respectively).

A second multiple linear regression was calculated to predict cGPA in clinical years based on the five pre-admission variables. This regression, on the other hand, revealed no significant predictors of cGPA among any of the five pre-admission variables included in the analysis; with an overall regression equation (F_(5,1)=0.654, p=0.729), and with an R² of 0.766.

Factors predicting the PT

Two separate models were analyzed to study the ability of the pre-admission variables to predict students' PT performance, as seen in Table 3.

The preclinical regression model analysis revealed no significant predictors of PT performance among any of the five pre-admission variables included in the analysis; with an overall regression equation (F_(5,4)=0.692, p=0.657), and with an R² of 0.464.

The clinical years' model, however, did not contain sufficient data to run the analysis and is thus not included in the table. This is due to the relatively lower number of students in this phase currently at our institution.

Prediction of PT by cGPA

Another linear regression was calculated to predict PT performance based on cGPA. A significant regression equation was found (F_(1,387)=59.006, p<0.0001), with an R² of 0.132. The regression further revealed that the cGPA was a strongly significant predictor of students' PT performance (p<0.0001, B=10.904, t=7.682). Therefore, on average, one-point increase in cGPA predicts for almost 11 points increase in the PT.

Discussion

Academic assessment

Our findings indicate that the NAT is a significant positive predictor of student performance in preclinical years through the cGPA; however, it was not found to be predictive of performance in clinical years. This could be partially attributable to the test being focused on some of the students' academically acquired knowledge and cognitive abilities such as the ability to infer, comprehend and apply; however, it does not assess other cognitive and non-cognitive skills which influence performance in clinical years.^11,12,38 These results are in line with other national studies conducted in Saudi Arabia. The general literature consensus is that NAT is a significant predictor of college performance in Saudi medical schools, such that in many studies it was even considered as the best predictor of the pre-admission variables.^12,39,40

On the contrary, our results show that the cumulative high school average was not predictive of cGPA at either pre-clinical or clinical years. In the majority of studies, except Al-Rukban and colleagues,³⁹ cumulative high school average was usually reported to be statistically predictive of performance in medical school.^12,40 This difference we report could be attributed to many factors. Firstly, the cumulative high school average represents all subjects taught in high school including those that are not related to medicine.³⁹ Secondly, the schools' teaching medium is mainly Arabic, while the medical college courses are exclusively conducted in English. Finally, the high school assessment system is different from that of the medical college in many ways, such as exam questions' style (multiple choice questions, short answer questions, objective structured clinical evaluation, etc.) and scope of questions (recall, communication, critical thinking, clinical application, etc.). Neither the NAT nor the CHSA was found to be predictive of performance through the PT.

Non-academic cognitive assessments

Our findings suggest that GAT is a negative predictor of the cGPA in preclinical years in addition to not being predictive of performance during clinical years. While initially surprising, this goes hand in hand with internationally published work indicating that measures of aptitude, in general, tend not to correlate with students' university academic performance or may even negatively correlate with it.⁴¹ This could perhaps be attributable to several causes reported by other studies about aptitude measures, including: firstly, failure of aptitude tests in predicting the performance of students from minority groups and diverse ethnic backgrounds could sometimes be expected due to the way such tests are developed.⁴² Secondly, aptitude tests are limited to assessing cognitive variables and cease to cover non-cognitive variables which were shown to be important contributors to academic success.⁴² Studies have shown that cognitive variables constitute only around 25% of the variables affecting academic performance and success.⁴³

These explanations were reported to apply to general aptitude measures, and since GAT is an aptitude test, we hypothesize that these might provide a possible explanation to our findings; however, more studies are needed to explore this area further.

In comparison, several studies have been conducted in Saudi Arabia to assess the predictive validity of GAT on the cGPA with mixed outcomes. Some of these studies reported that GAT is predictive of performance at different levels of medical school.^12,44 Whereas other studies, namely those conducted by Al-Rukban and colleagues, and Murshid reported that GAT is not predictive of the cGPA.^{39, 45}

Moreover, our results show that GAT, similarly to the academic achievement assessments, also does not predict performance through PT.

English language proficiency assessment

Our results found the TOEFL scores to be predictive of cGPA in preclinical years. However, this was not observed in clinical years. There are several possible explanations for this observation. Firstly, during the preclinical phase, the inadequate English proficiency of non-native speakers would significantly impact their performance as it would be strongly reflected in their studies and assessment, which goes in hand with the reported predictability of TOEFL on PBL performance.⁴⁶ Whereas by the time the students are in their clinical years, their English proficiency would improve gradually as they would have had already got used to studying and conversing in an English medium for the period of their preclinical phase.

Secondly, the TOEFL was found to be predictive of preclinical performance possibly due to the fact that the test mainly involves components that require reading and writing skills, which are strongly reflected in preclinical years' assessment. This is in contrast to clinical years that focus more on communication skills and require more verbal proficiency, both of which are not adequately reflected by the test. This limitation in verbal proficiency and communication skills assessment is supported in Roemer's study which highlights that the TOEFL's prediction accuracy is limited to only grammar and reading proficiency.⁴⁷ This also goes in hand with the recommendations of the Educational Testing Service (ETS) to not use rigid cut-off TOEFL scores for student selection.⁴⁸

In contrary, IELTS was not found to be predictive of performance in either preclinical or clinical years. Neither the TOEFL nor the IELTS was found to be predictive of performance through the PT.

College performance indicators: cGPA and PT

Our results show a positive correlation with strong predictive validity. Giving the several additional advantages mentioned earlier for the PT over the cGPA, namely functional knowledge; being able to quantify the cGPA's predictability of the PT provides educators with a valuable tool to better monitor students' progress and performance over time, permitting a more detailed scope for analysis of students' annual performance in specific subject disciplines. This valuable information could be used to improve and inform mentorship programs such that students would be able to receive individualized feedback on their annual performance progression, in a detailed and subject-wise manner.⁴⁹

Despite the wide use of PT and cGPA, the literature investigating the relationship between them is quite deficient. Our literature search identified only one study by Al Alwan and colleagues that touched upon this relation,²⁹ such that a positive correlation between the PT and cGPA was indicated; however, the predictive validity of this correlation was not reported. Our study offers the advantage of bridging this literature gap through quantifying this predictive validity.

Pre-admission variables and PT

While our results show some of the pre-admission variables to be predictive of performance through the cGPA, none of them succeeded in predicting pre-clinical students' performance through the PT. This is mostly attributable to the fact that the predictive validity of the pre-admission variables does not usually include examining functional knowledge. Therefore, upon adding the PT data to our study analysis, pre-admission variables that previously demonstrated to be predictive of student performance with the cGPA appeared to lose their performance predictability.

Limitations

Since we are a relatively new school, the sample size of our students currently in the clinical years was not sufficient to examine the predictability of pre-admission variables on PT in that phase, but as more of our students' progress into the clinical years, we will be able to explore this area further. Another applicable limitation is that the study represents data of a single school in Saudi Arabia. Thus, further studies using data from multiple schools would be needed to validate these findings further.

Conclusions

Our study shows that among all the pre-admission variables currently used in Saudi Arabia, the NAT and TOEFL scores are the only positive predictors of performance in the preclinical years when reflected by students' cGPA. However, when using the PT as a performance indicator, the predictive effects of those variables cease; and it is likely so due to the additional assessment of the longitudinal functional knowledge component by the PT. We henceforth recommend using the PT, in addition to the cGPA, when examining the predictive validity of pre-admission variables at other institutions in Saudi Arabia and around the world. Lastly, we believe that there is a pressing need to recognize or develop more sensitive pre-admission assessment variables that can better predict student performance in the clinical years, and that much further research in this area is yet anticipated to underpin the existing ones.

Acknowledgements

The authors sincerely thank Dr. Mohammed Saleh Alowayed, former Dean of Admission & Registration in the Student Affairs Department at Alfaisal University, for his gracious efforts in facilitating the data collection process from the Student Information System. The authors extend the acknowledgement to Prof. Khaled Al-Kattan, Dean of College of Medicine, Alfaisal University and Dr. Wael Al-Kattan, Vice Dean for Academic and Student Affairs, College of Medicine, Alfaisal University for their continuous care and unlimited support to this study.

Conflict of Interest

The authors declare that they have no conflict of interest.

References

1. Poole P, Shulruf B, Rudland J and Wilkinson T. Comparison of UMAT scores and GPA in prediction of performance in medical school: a national study. Med Educ. 2012; 46: 163-171.
   Full Text PubMed
2. Wilkinson D, Zhang J and Parker M. Predictive validity of the Undergraduate Medicine and Health Sciences Admission Test for medical students' academic performance. Med J Aust. 2011; 194: 341-344.
   PubMed
3. Yates J and James D. The UK Clinical Aptitude Test and clinical course performance at Nottingham: a prospective cohort study. BMC Med Educ. 2013; 13: 32.
   Full Text PubMed
4. Ward A, Stoker HW, Murray-Ward M. Educational measurement: theories and applications. Lanham: University Press of America; 1996.
5. Dunleavy DM, Kroopnick MH, Dowd KW, Searcy CA and Zhao X. The predictive validity of the MCAT exam in relation to academic performance through medical school: a national cohort study of 2001-2004 matriculants. Acad Med. 2013; 88: 666-671.
   Full Text PubMed
6. Ferguson E, James D and Madeley L. Factors associated with success in medical school: systematic review of the literature. BMJ. 2002; 324: 952-957.
   PubMed
7. Frischenschlager O, Haidinger G and Mitterauer L. Factors associated with academic success at Vienna Medical School: prospective survey. Croat Med J. 2005; 46: 58-65.
   PubMed
8. Julian ER. Validity of the Medical College Admission Test for predicting medical school performance. Acad Med. 2005; 80: 910-917.
   PubMed
9. McManus IC, Smithers E, Partridge P, Keeling A and Fleming PR. A levels and intelligence as predictors of medical careers in UK doctors: 20 year prospective study. BMJ. 2003; 327: 139-142.
   Full Text PubMed
10. Mercer A and Puddey IB. Admission selection criteria as predictors of outcomes in an undergraduate medical course: a prospective study. Med Teach. 2011; 33: 997-991004.
    Full Text PubMed
11. Achievement test for Science Colleges. National center for assessment in higher education; 2016 [cited 15 June 2016]; Available from: http://qiyas.sa/Sites/English/Tests/EducationalTests/Pages/Achievement-test-for-Science-Colleges---Males.aspx.
12. Albishri JA, Aly SM and Alnemary Y. Admission criteria to Saudi medical schools. Which is the best predictor for successful achievement? Saudi Med J. 2012; 33: 1222-1226.
    PubMed
13. Siu E and Reiter HI. Overview: what's worked and what hasn't as a guide towards predictive admissions tool development. Adv Health Sci Educ Theory Pract. 2009; 14: 759-775.
    Full Text PubMed
14. Katz S and Vinker S. New non-cognitive procedures for medical applicant selection: a qualitative analysis in one school. BMC Med Educ. 2014; 14: 237.
    Full Text PubMed
15. Urlings-Strop LC, Stegers-Jager KM, Stijnen T and Themmen AP. Academic and non-academic selection criteria in predicting medical school performance. Med Teach. 2013; 35: 497-502.
    Full Text PubMed
16. Mercer A and Puddey IB. Admission selection criteria as predictors of outcomes in an undergraduate medical course: a prospective study. Med Teach. 2011; 33: 997-991004.
    Full Text PubMed
17. McGaghie WC. Assessing readiness for medical education: evolution of the medical college admission test. JAMA. 2002; 288: 1085-1090.
    PubMed
18. General Aptitude Test (GAT). National Center for Assessment in Higher Education; 2016 [cited 15 June 2016]; Available from: http://qiyas.sa/Sites/English/Tests/EducationalTests/Pages/General-Aptitude-Test-(GAT).aspx.
19. Wulff HR. The language of medicine. J R Soc Med. 2004; 97: 187-188.
    Full Text PubMed
20. Ahmed B, Ahmed LB and al-Jouhari MM. Factors determining the performance of medical students of the Faculty of Medicine, University of Kuwait. Med Educ. 1988; 22: 506-508.
    PubMed
21. Alfayez SF, Strand DA and Carline JD. Academic, social and cultural factors influencing medical school grade performance. Med Educ. 1990; 24: 230-238.
    PubMed
22. Al-Mously N, Salem R, Al-Hamdan N. The impact of gender and English language on the academic performance of students: An experience from new Saudi medical school. J Contemp Med Edu. 2013;1:170-6.
23. Hayes SC and Farnill D. Medical training and English language proficiency. Med Educ. 1993; 27: 6-14.
    PubMed
24. About us. International English language testing system; 2016 [cited 14 June 2016]; Available from: http://www.ielts.org/about_us.aspx.
25. TOEFL Home. Educational testing service; 2016 [cited 14 June 2016]; Available from: http://www.ets.org/toefl/.
26. Zhou YX, Zhao ZT, Li L, Wan CS, Peng CH, Yang J and Ou CQ. Predictors of first-year GPA of medical students: a longitudinal study of 1285 matriculates in China. BMC Med Educ. 2014; 14: 87.
    Full Text PubMed
27. Richardson M, Abraham C and Bond R. Psychological correlates of university students' academic performance: a systematic review and meta-analysis. Psychol Bull. 2012; 138: 353-387.
    Full Text PubMed
28. Bacon DR and Bean B. GPA in research studies: an invaluable but neglected opportunity. Journal of Marketing Education. 2016; 28: 35-42.
    Full Text
29. Al Alwan I, Al-Moamary M, Al-Attas N, Al Kushi A, AlBanyan E, Zamakhshary M, Al Kadri HM, Tamim H, Magzoub M, Hajeer A and Schmidt H. The progress test as a diagnostic tool for a new PBL curriculum. Educ Health (Abingdon). 2011; 24: 493.
    PubMed
30. Schuwirth LW, van der Vleuten CP. The use of progress testing. Perspect Med Educ. 2012;1:24-30.
31. Freeman A, Van Der Vleuten C, Nouns Z and Ricketts C. Progress testing internationally. Med Teach. 2010; 32: 451-455.
    Full Text PubMed
32. Vleuten CPMVD, Verwijnen GM and Wijnen WHFW. Fifteen years of experience with progress testing in a problem-based learning curriculum. Medical Teacher. 2009; 18: 103-109.
    Full Text
33. Verhoeven BH, Snellen-Balendong HA, Hay IT, Boon JM, van der Linde MJ, Blitz-Lindeque JJ, Hoogenboom RJ, Verwijnen GM, Wijnen WH, Scherpbier AJ and van der Vleuten CP. The versatility of progress testing assessed in an international context: a start for benchmarking global standardization? Med Teach. 2005; 27: 514-520.
    Full Text PubMed
34. Findyartini A, Werdhani RA, Iryani D, Rini EA, Kusumawati R, Poncorini E and Primaningtyas W. Collaborative progress test (cPT) in three medical schools in Indonesia: the validity, reliability and its use as a curriculum evaluation tool. Med Teach. 2015; 37: 366-373.
    Full Text PubMed
35. Nouns ZM and Georg W. Progress testing in German speaking countries. Med Teach. 2010; 32: 467-470.
    Full Text PubMed
36. Johnson TR, Khalil MK, Peppler RD, Davey DD and Kibble JD. Use of the NBME Comprehensive Basic Science Examination as a progress test in the preclerkship curriculum of a new medical school. Adv Physiol Educ. 2014; 38: 315-320.
    Full Text PubMed
37. Progress test methodology. Qassim University College of Medicine; 2016 [cited 14 June 2016]; Available from: http://med.qu.edu.sa/eng/index.php/links/progress-test-results/test-mehodology.html.
38. Adam J, Bore M, Childs R, Dunn J, Mckendree J, Munro D and Powis D. Predictors of professional behaviour and academic outcomes in a UK medical school: A longitudinal cohort study. Med Teach. 2015; 37: 868-880.
    Full Text PubMed
39. Al-Rukban MO, Munshi FM, Abdulghani HM and Al-Hoqail I. The ability of the pre-admission criteria to predict performance in a Saudi medical school. Saudi Med J. 2010; 31: 560-564.
    PubMed
40. Al Alwan I, Al Kushi M, Tamim H, Magzoub M and Elzubeir M. Health sciences and medical college preadmission criteria and prediction of in-course academic performance: a longitudinal cohort study. Adv Health Sci Educ Theory Pract. 2013; 18: 427-438.
    Full Text PubMed
41. Côté JE, Levine CG. Attitude versus aptitude is intelligence or motivation more important for positive higher-educational outcomes? J Adolesc Res. 2000;15:58-80.
42. Louderback PJ. Comparison of the predictive validity of traditional intellectual measures and sociocultural factors on American Indian student academic achievement [Doctoral Dissertation]. Stillwater: Proquest Dissertations Publishing; 2008.
43. Mouw JT, Khanna RK. Prediction of academic success: a review of the literature and some recommendations. College Student Journal. 1993;27:328-36.
44. Al-Alwan IA. Association between scores in high school, aptitude and achievement exams and early performance in health science college. Saudi J Kidney Dis Transpl. 2009; 20: 448-453.
    PubMed
45. Murshid KR. The predictive value of individual admission criteria on academic performance in a Saudi medical college. Journal of Taibah University Medical Sciences. 2013;8:18-23.
46. Mpofu DJ, Lanphear J, Stewart T, Das M, Ridding P and Dunn E. Facility with the English language and problem-based learning group interaction: findings from an Arabic setting. Med Educ. 1998; 32: 479-485.
    Full Text PubMed
47. Roemer A. A more valid alternative to TOEFL? College and University Journal. 2002;77:13.
48. Tannenbaum RJ, Wylie EC. Linking English-language test scores to the common European framework of reference: an application of standard-setting method-ology. ETS Research Report Series. 2008;2008:i-75.
49. Mohamed AM, Dabbagh MM, Dabaliz AM, Barakat AA, Ahmed EK, Obeidat A. Medical curriculum: students' perspectives. In: Ganguly P, editor. Health and disease: curriculum for the 21st-century medical students. New York: Nova Science Publishers; 2014.