Comparison of web-based and face-to-face interviews for application to an anesthesiology training program: a pilot study

Introduction

Medical education extends beyond completion of a medical degree in many countries.  To become a practicing physician and to specialize within a distinct medical discipline, additional post-graduate clinical training must be completed.  This training is known as "residency" in the United States (US) and Canada, while in the United Kingdom, physicians-in-training complete two years of work-based education in the Foundation Programme before proceeding to specialty / general practitioner training.  In each country, applicants to post-graduate medical training programs often complete both written applications and face-to-face interviews at prospective training program sites as part of the admissions process.^1-3 Applications to US residency programs are coordinated through the National Resident Matching Program (NRMP).¹ Medical students begin the application process during the final year of medical school by submitting standardized written applications to a central application service. After reviewing applications for academic merit, residency programs invite select candidates for face-to-face interviews. 

At the conclusion of the interview period, residency applicants submit a "rank-order list" to the NRMP.  The rank-order list indicates a list of programs where the applicant wishes to train ranked in his/her order of preference.  Each residency program also submits a rank-order list indicating a list of applicants the program wishes to train ranked in the program's order of preference. The process is blinded so that neither applicant nor residency program will see the other's list.  A computer algorithm then assigns each applicant to a residency position using data obtained from the rank-order lists.  The NRMP application process closely mirrors that of the Canadian Resident Matching Service used by Canadian residency programs.²

Anesthesiology is an increasingly competitive medical specialty in the US. As such, 49 senior US medical school students failed to secure a desired residency position in anesthesiology through the NRMP in 2014.⁴ The probability of admission to a residency program in anesthesiology increases with the number of contiguous program ranks made by a specific applicant: US medical school seniors admitted to a post-graduate residency program in anesthesiology through the NRMP in 2014 ranked an average of 14.4 programs on their rank-order lists.⁴ It is clear that applicants to anesthesiology residency positions often apply to multiple training programs to remain competitive in the admissions process.

Traditionally, applying to US anesthesiology residency programs has required applicants to participate in face-to-face interviews with faculty members at each prospective residency program.  Face-to-face interview attendance poses several disadvantages for residency applicants.  Time away from medical studies decreases educational productivity. The residency interview season is abbreviated, and some applicants experience scheduling conflicts between interview dates at different residency programs. Traveling to multiple out-of-town interviews may result in financial hardship.^5,6 Candidates may be unable to accept interview offers they otherwise would have accepted due to these constraints, thus failing to capitalize on important opportunities to market themselves to residency programs.⁷    

Web-based residency interviews have been proposed as an alternative to traditional face-to-face interviews to maximize applicants’ interview opportunities. Videoconferencing software applications have been used to conduct residency interviews in other medical specialties, generally in conjunction with a face-to-face interview on the same day or on a day shortly after the web-based interview.^5,8-10 The impact of web-based interviews on residency program admission results is unclear. A 2014 NRMP survey of US anesthesiology residency program directors cited interactions with faculty during interview and visit, interpersonal skills, and interactions with current residents during interview and visit as the three top factors determining an applicant's position on the residency program rank-order list.¹¹ Some residency applicants may hesitate to select web-based residency interviews due to a belief that not participating during face-to-face residency interview events will negatively impact the chance of admission to their preferred residency program.¹² Prior investigations of web-based interviews focused on cost-effectiveness and post-interview applicant satisfaction, but did not provide residency program admission data.  This pilot study was designed to compare admission rates to a US anesthesiology residency program for applicants completing face-to-face versus web-based interviews as part of the residency admissions process.   

Methods

Study design

We performed a prospective observational study comparing NRMP admission rates to the Loma Linda University anesthesiology residency program between applicants completing face-to-face versus web-based admission interviews.  Ethical approval to complete the study was obtained from the Loma Linda University Institutional Review Board.

Study participants

The 211 applicants invited to interview for admission to the Loma Linda University anesthesiology residency program during the 2014-2015 NRMP application cycle were eligible for study participation.  Applicants chose to complete either a face-to-face or a web-based residency selection interview.  Of the 211 applicants invited to interview, 141 (72.7%) selected face-to-face and 53 (27.7%) selected web-based interviews.  Fourteen (9.9%) face-to-face and 11 (20.8%) web-based interview applicants did not complete interviews. Thus, 127 (75.1%) applicants completed face-to-face and 42 (24.9%) completed web-based interviews.  The 169 applicants who completed either face-to-face or web-based interviews were invited to complete post-interview surveys regarding their interview experiences.  This is illustrated in Figure 1.

Data collection

Applicant demographic data such as age, gender, United States Medical School Licensing Examination Step 1 score, and medical school location were obtained from written application forms completed by all applicants to the residency program. The residency program coordinator maintained a database indicating the type of interview (face-to-face versus web-based) selected and completed by each applicant.  A list of admitted applicants was provided to the residency program by the NRMP after applicant and residency program rank-order lists were analyzed by the NRMP computer algorithm.

A brief Internet survey was developed by the authors and aimed to explore applicants' reasons for choosing either face-to-face or web-based interviews. An invitation to complete this survey was provided to all applicants who completed residency admission interviews after interview completion. Survey participation was voluntary and anonymous. Applicants were informed in writing that participation would not affect the residency application process. 

Figure 1

Distribution of study participants

Procedures

Interview offers were extended to residency applicants via standardized electronic mail that included a list of available interview dates, each designated as a face-to-face interview or a web-based interview day. Applicants selected their preferred interview dates.  The interview invitation included a statement advising residency applicants that face-to-face and web-based interviews would be considered equivalent by the faculty members determining the residency program rank-order list.

Face-to-face interviews

Applicants who selected face-to-face interviews were invited to attend an optional off-campus dinner with current anesthesiology residents the night before their interviews. On the day of the interviews, the residency program director provided a comprehensive program overview via PowerPoint slide presentation. Applicants completed three or four 10-minute interviews with faculty members. These non-structured interviews were not recorded to protect applicant privacy, but faculty made notes and provided summary evaluations of their judgments of applicant suitability for admission to our residency program that were later considered when the final program rank-order list was formulated. The face-to-face applicants toured the medical center, and participated in a lunchtime question and answer session with current residents. 

Web-based interviews

Applicants who selected web-based interviews were provided access to audio and video versions of the comprehensive program overview given to face-to-face interview applicants by the residency program director.  They were also provided access to a video tour of Loma Linda University Medical Center and its surrounding communities.

Web-based interview applicants selected either FaceTime or Skype as their preferred videoconferencing application. Applicants were allowed their choice of electronic device and Internet connection. They were allowed to use headphones and/or external microphones as needed to improve audio quality. All Department of Anesthesiology videoconferencing connections were established using desktop iMac computers located in separate rooms and hard-wired to the high-speed university Ethernet network. Like face-to-face applicants, web-based interview candidates completed three or four 10-minute interviews with faculty members.  The same six faculty members who conducted face-to-face interviews conducted web-based interviews.  The non-structured web-based interviews were not recorded to protect applicant privacy, but faculty made notes and provided summary evaluations of their judgments of applicant suitability for admission to our residency program that were later considered when the final program rank-order list was formulated. Web-based interview applicants were also invited to interact with current anesthesiology residents through audio-visual Google Hangouts chat sessions conducted twice on each web-based interview day.

All applicants (face-to-face and web-based interview) were advised they could schedule an optional on-campus department tour at their convenience on a day subsequent to their interview. However, such visits were not required for applicants to be placed on the program rank-order list.

Post-interview survey

Following interview days, all applicants received e-mail invitations to participate in an anonymous web-based survey focusing on the applicants' reasons for selecting a face-to-face or a web-based interview and their perceptions of the efficacy of each interview type.  No identifying data were collected and results were not analyzed until after NRMP admission results were announced.

Residency program rank-order list formation

The residency program rank-order list was compiled by the six faculty interviewers, none of whom interviewed all applicants, and the department chairman. Factors considered included, but were not limited to, faculty interviewer summary comments, United States Medical Licensing Exam Step 1 scores, medical school evaluations, and letters of recommendation.  The faculty members who were involved in rank-order list formation were given summaries of interviewer impressions and comments that did not include information regarding whether the candidates completed face-to-face or web-based interviews. The finalized rank-order list was then submitted to the NRMP.

Data analysis

The proportion of applicants who were admitted to our anesthesiology residency program through the NRMP was compared between those who completed face-to-face and those who completed web-based interviews. Differences in answers to survey questions were compared between applicants who completed face-to-face and those who completed web-based interviews. Continuous data were analyzed for normality using the Shapiro-Wilk test and expressed as either mean [95% Confidence Interval (CI)] or median [95% CI]. Normally distributed data were analyzed using the t-test. Data that were not normally distributed were analyzed by the Wilcoxon rank-sum test and are expressed as the smoothed empirical likelihood median [95% CI]; the Hodges Lehman method, assuming data symmetry, was used to compare differences. Categorical data were analyzed by Chi-square test, and differences were compared using the Wald Test for two proportions. Statistical significance was taken at p <0.05 (JMP 10.0.0, SAS Institute, Cary, NC, USA).

Results

There were no between group differences in applicant characteristics including United States Medical Licensing Examination Step 1 scores (Table 1). A larger proportion of face-to-face applicants attended a medical school in California, the same state as our residency program (p=0. 000016). However, the overall regional distribution of medical schools was not different from the prior 4 years of applicants to our program. A larger proportion of web-based applicants completed a post-interview campus visit (p=0.0025).

Choice of web-based versus face-to-face interview was not associated with a difference in the proportion of applicants admitted to our residency program through the NRMP (Table 1). An equal proportion of applicants were assigned to the upper half of the program rank-order list, regardless of the type of interview completed. Only one of 3 web-based and one of 14 face-to-face interview candidates who were ultimately admitted to our residency program completed a post-interview visit.

Post-interview surveys were completed by 111 applicants (76 face-to-face and 35 web-based interview applicants).  One hundred three applicants provided answers to questions exploring factors driving applicants to select each interview type. The most common self-reported reasons for selecting web-based interviews were conflict of interview dates between programs, geographic/travel concerns, or financial limitations (Table 2 Applicants largely selected face-to-face interviews due to a desire to interact with current residents, geographic proximity, or a desire to visit the campus/facility.

Of web-based interview survey respondents, 3 (9.4%) reported difficulty in maintaining eye contact; 2 (6.3%) reported sub-optimal video quality; and 1 (3.1%) reported sub-optimal audio quality during their interview. Still, web-based interviews met or exceeded expectations of all submitting survey responses. Similarly, all but 2 face-to-face interview applicants who completed surveys felt their interview experience was good or excellent.

Discussion

Residency admission interviews are viewed as important factors influencing rank-order list formation by both residency programs^13-20 and applicants.^21,22 Residency programs use interviews to assess applicants’ noncognitive skills, such as interpersonal and communication skills, maturity, interest in the field, and honesty.^13,23 Applicants value interactions with current residents as important opportunities to assess resident morale and program weaknesses as perceived by current trainees.^24,25 However, these perceptions that resident interviews are key components of the residency selection process are not always supported by the available literature.  In three studies, interviews changed the rank-order list position of applicants more than 10 positions below or above their preinterview rank.^26-28 However, one study found such a strong correlation between the interview and academic variables that final rank was unchanged whether the interview was added to the statistical model or not.²⁹ Interviewers not blinded to applicants’ academic records may subconsciously base their interview assessments on academic criteria. Interview performance is also not predictive of an applicant’s subsequent clinical performance, as outlined in a review of 34 studies on residency interviews.³⁰ This review, noting the high financial cost and poor predictive power of residency interviews, cited use of web-based residency interviews as an area of future study in hopes of diminishing interview expenses for both residency programs and applicants. 

In our pilot study, we found that completing a web-based versus a face-to-face residency selection interview did not affect the proportion of applicants who were accepted to our anesthesiology residency program through the NRMP.  Web-based interview and face-to-face interview applicants were equally likely to be ranked in the upper or lower half of the program rank-order list. Internal residency program data from the prior 3 years shows that applicants who gained admission to our program in those years were ranked in the upper half of the program rank-order list. Scheduling, geographic/travel, and financial issues were the most common self-reported considerations leading applicants to select web-based interviews. All web-based interview applicants who completed anonymous post-interview surveys perceived web-based interviews to be effective, with a majority indicating a preference for this interview type over face-to-face interviews. While videoconferencing software applications have been used to conduct residency interviews for other medical specialties, our study addresses several questions left unanswered by prior investigations.

First, we address the use of web-based interviews for all candidates considered qualified for admission to our residency program. Ours is a larger cohort of applicants (169 completed interviews) than prior reports that only included up to 48 applicants.⁹ We offered the opportunity to choose face-to-face or web-based interviews to all applicants invited to interview, rather than using web-based interviews as a screening tool for applicants felt to be less qualified for admission to the residency program as previously reported.⁵

Second, our study investigated the impact of an applicant completing either a web-based or a face-to-face interview, rather than both types of interviews. We did not require that applicants completing web-based interviews schedule a post-interview campus visit as in prior comparisons of web-based and face-to-face interviews.^5,10

Third, we report admission rates to our residency through the NRMP. Prior studies of 33 urology residency applicants who completed both face-to-face and web-based interviews¹⁰ and of 16 gastroenterology fellowship applicants who also completed both face-to-face and web-based interviews⁸ focused on applicant satisfaction but did not report admission rates. A study of 48 ophthalmology residency applicants allowed self-selection of face-to-face versus web-based interview.⁹ In that study, 21 (44%) chose a web-based interview and 12 (57%) of those also completed a post-interview campus visit. While there was no significant difference in the number of web-based interview versus face-to-face interview applicants ranked in the top 25 spots on the program rank-order list, final admission data were not presented.  The residency program rank-order list is important to show that the program is not biased, but does not indicate whether web-based candidates will rank a program highly on their rank-order lists. Residency program directors may perceive that web-based interview applicants either lack serious interest in the residency program or could fail to rank the residency program highly because they did not develop interpersonal connections with the program faculty or residents during an in-person visit. The equivalent admission rate between candidates completing web-based and face-to-face interviews suggests that web-based interview applicants were committed candidates who ranked our program highly enough to gain admission.     

Several factors limit generalizing the findings of our study. This was a single-center investigation using one year of NRMP applications and admission results, in which applicants were not randomized to complete a particular interview type. While this allowed us to better characterize the type of applicant desiring web-based interviews, it is unclear whether residency applicants' perceptions of interview efficacy would change if they were not assigned to their first choice of interview type. The characteristics of applicants to our anesthesiology residency program may differ from those applying to other anesthesiology residency programs, limiting external validity. Importantly however, we interviewed candidates from a wide range of medical schools across the United States, and those who completed surveys self-reported application to nearly one-third of United States anesthesiology residency programs. Further, the average United States Medical Licensing Examination Step 1 scores of our applicants (Table 1) were similar to the average score (230) reported from all applicants who were admitted to US anesthesiology programs in 2014.¹ This suggests that the applicants studied here are similar to those applying to other anesthesiology residency programs in the US. Randomized, multi-center studies are needed to determine more definitively whether web-based interviews are a viable alternative to traditional face-to-face interviews for all, or only for a subset of applicants and residency programs.

Conclusions

Completing a web-based versus a face-to-face residency selection interview did not affect the admission rate to our anesthesiology residency program.  Applicants who chose web-based interviews indicated that scheduling, travel or financial concerns were important considerations motivating this choice. Taken together, these findings suggest that for at least some applicants, offering web-based interviews could provide applicant benefit without harming chances of acceptance to a residency program.  Future studies should focus on identifying which applicants for post-graduate medical education would most benefit from web-based interviews as well as on how residency programs can optimally implement web-based interviews.

Acknowledgements

The authors acknowledge the statistical analysis assistance of David Juma, MPH, Consulting Coordinator and Senior Consultant, and Briana Wells, MS, Research Analyst, Loma Linda University Research Consulting Group. We also acknowledge the interview coordination assistance of Alex Serafin, MD, Oliver Small, MD, and Leyla Embree, Anesthesiology Residency Program Coordinator, Loma Linda University Medical Center, Loma Linda, CA, USA.

Conflict of Interest

The authors declare that they have no conflict of interest.

References

1. National Resident Matching Program, Association of American Medical Colleges. Residency application toolkit, 2015 [cited 24 January 2016]; Available from: http://www.nrmp.org/residency/applicant-toolkit/.
2. Canadian Resident Matching Service. An overview of your 2016 R-1 main residency match, 2015 [cited 24 January 2016]; Available from: http://www.carms.ca/wp-content/uploads/2015/09/2016-R-1-Match Overv-iew-CMGs-EN.pdf.
3. The UK Foundation Programme Office. FP/AFP 2016 applicant's handbook, 2015 [cited 24 January 2016]; Available from: http://FP2016_Applicants_Handbook_FINAL_WEB_updated_300915_V1-3.pdf.
4. National Resident Matching Program, Association of American Medical Colleges. Charting outcomes in the match: characteristics of applicants who matched to their preferred specialty in the 2014 main residency match, 5th ed., 2014 [cited 24 January 2016]; Available from: http://www.nrmp.org/wpcontent/uploads/2014/09/Charting-Outcomes-2014- Final.pdf.
5. Edje L, Miller C, Kiefer J and Oram D. Using skype as an alternative for residency selection interviews. J Grad Med Educ. 2013; 5: 503-505.
   Full Text PubMed
6. Kerfoot BP, Asher KP and McCullough DL. Financial and educational costs of the residency interview process for urology applicants. Urology. 2008; 71: 990-994.
   Full Text PubMed
7. Liang M, Curtin LS, Signer MM and Savoia MC. Understanding the Interview and Ranking Behaviors of Unmatched International Medical Students and Graduates in the 2013 Main Residency Match. J Grad Med Educ. 2015; 7: 610-616.
   Full Text PubMed
8. Daram SR, Wu R and Tang SJ. Interview from anywhere: feasibility and utility of web-based videoconference interviews in the gastroenterology fellowship selection process. Am J Gastroenterol. 2014; 109: 155-159.
   Full Text PubMed
9. Pasadhika S, Altenbernd T, Ober RR, Harvey EM and Miller JM. Residency interview video conferencing. Ophthalmology. 2012; 119: 426-426.
   Full Text PubMed
10. Shah SK, Arora S, Skipper B, Kalishman S, Timm TC and Smith AY. Randomized evaluation of a web based interview process for urology resident selection. J Urol. 2012; 187: 1380-1384.
    Full Text PubMed
11. National Resident Matching Program, Association of American Medical Colleges. Results of the 2014 program director survey, 2014 [cited 24 January 2016]; Available from: http://www.nrmp.org/wp-content/uploads/2014/09/PD-Survey-Report-2014.pdf.
12. Schlitzkus LL, Schenarts PJ and Schenarts KD. It was the night before the interview: perceptions of resident applicants about the preinterview reception. J Surg Educ. 2013; 70: 750-757.
    Full Text PubMed
13. Wagoner NE, Suriano JR and Stoner JA. Factors used by program directors to select residents. J Med Educ. 1986; 61: 10-21.
    Full Text PubMed
14. Wagoner NE and Gray GT. Report on a survey of program directors regarding selection factors in graduate medical education. J Med Educ. 1979; 54: 445-452.
    Full Text PubMed
15. Wagoner NE and Suriano JR. Program directors’ responses to a survey on variables used to select residents in a time of change. Academic Medicine. 1999; 74: 51-8.
    Full Text PubMed
16. Hirthler MA, Glick PL, Hassett JM and Cooney DR. Evaluation of the pediatric surgical matching program by the directors of pediatric surgical training programs. J Pediatr Surg. 1994; 29: 1370-1374.
    Full Text PubMed
17. Makdisi G, Takeuchi T, Rodriguez J, Rucinski J and Wise L. How we select our residents--a survey of selection criteria in general surgery residents. J Surg Educ. 2011; 68: 67-72.
    Full Text PubMed
18. DeLisa JA, Jain SS and Campagnolo DI. Factors used by physical medicine and rehabilitation residency training directors to select their residents. Am J Phys Med Rehabil. 1994; 73: 152-156.
    PubMed
19. Galazka SS, Kikano GE and Zyzanski S. Methods of recruiting and selecting residents for U.S. family practice residencies. Academic Medicine. 1994; 69: 304-6.
    Full Text PubMed
20. TAYLOR C, WEINSTEIN L and MAYHEW H. The process of resident selection: A view from the residency director's desk. Obstetrics & Gynecology. 1995; 85: 299-303.
    Full Text PubMed
21. Puscas L, Sharp SR, Schwab B and Lee WT. Qualities of residency applicants: comparison of otolaryngology program criteria with applicant expectations. Arch Otolaryngol Head Neck Surg. 2012; 138: 10-14.
    Full Text PubMed
22. Simmonds AC, Robbins JM, Brinker MR, Rice JC and Kerstein MD. Factors important to students in selecting a residency program. Acad Med. 1990; 65: 640-643.
    Full Text PubMed
23. LaGrasso JR, Kennedy DA, Hoehn JG, Ashruf S and Przybyla AM. Selection criteria for the integrated model of plastic surgery residency. Plast Reconstr Surg. 2008; 121: 121-125.
    Full Text PubMed
24. Aagaard EM, Julian K, Dedier J, Soloman I, Tillisch J, Perez-Stable EJ. Factors affecting medical students’ selection of an internal medicine residency program. J Natl Med Assoc. 2005;97(9):1264-1270.
25. Yousuf SJ, Kwagyan J and Jones LS. Applicants' choice of an ophthalmology residency program. Ophthalmology. 2013; 120: 423-427.
    Full Text PubMed
26. Christakis PG, Christakis TJ, Dziura J and Christakis JT. Role of the interview in admissions at the University of Toronto ophthalmology program. Can J Ophthalmol. 2010; 45: 527-530.
    Full Text PubMed
27. Collins M, Curtis A, Artis K, Staib L and Bokhari J. Comparison of two methods for ranking applicants for residency. J Am Coll Radiol. 2010; 7: 961-966.
    Full Text PubMed
28. Gong H, Parker NH, Apgar FA and Shank C. Influence of the interview on ranking in the residency selection process. Med Educ. 1984; 18: 366-369.
    Full Text PubMed
29. Young MJ, Woolliscroft JO and Holloway JJ. Determining the policies of a residency selection committee. J Med Educ. 1986; 61: 835-837.
    Full Text PubMed
30. Stephenson-Famy A, Houmard BS, Oberoi S, Manyak A, Chiang S and Kim S. Use of the Interview in Resident Candidate Selection: A Review of the Literature. J Grad Med Educ. 2015; 7: 539-548.
    Full Text PubMed