Evidence-based medicine curricula and barriers for physicians in training: a scoping review

Introduction

Evidence-based medicine (EBM) has been defined as the "conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients".¹ EBM involves integrating clinical expertise with the best available evidence in an attempt to bridge the gap between the growing medical literature and patient care while also considering the preferences and values of the patient in the decision-making.² EBM is patient-centered, requires active participation by the learner, acknowledges gaps in knowledge, and bases clinical decisions on evidence rather than authority.³

Understanding the concepts of EBM and mastering the abilities to retrieve, critically appraise, and apply medical literature to patient care are essential skills for clinicians.⁴ The Accreditation Council for Graduate Medical Education states that residents must demonstrate the ability to investigate and evaluate their care of patients, appraise and assimilate scientific evidence, and continuously improve patient care based on constant self-evaluation and life-long learning.⁵ Each step of the EBM process requires different levels of competency.^6,7 Despite these requirements, most physicians in training have limited knowledge of the EBM process and research methodology, including study design and interpretation of the results.⁸ During the training years, learning fulfills all the adult-learning theory requirements, as the physicians in training come to the table with their own set of life experiences and motivations, they can direct their own learning, they learn better by doing and want to apply their learning to concrete situations sooner rather than later.⁹ As the EBM process is very complex, the learning efficacy is enhanced when it is based on the adult learning theory and when constant, longitudinal exposure pertains over time to the EBM concepts.¹⁰

Despite the fact that EBM training is now standard in physician training programs, there is still no consensus on how best to ensure that these skills are taught effectively for life-long learning. The most successful teaching methods for EBM are still unknown, and there is little evidence about which teaching methods lead to improved patient outcomes.^11-16 It is also extremely important to know of potential barriers that might need to be addressed in order to achieve a successful implementation of the EBM curriculum.^17-18

The objectives of this scoping review are 1. Describe the structure of the EBM curricula and its impact on the residents' attitudes, behaviors, skills and knowledge, and 2. Describe the barriers that were identified during the EBM curriculum implementation process.

Methods

The methodology for this scoping review was based on the framework outlined by Arksey and O'Malley.¹⁹ The review included the following five key phases: (1) identifying the research question, (2) identifying relevant studies, (3) study selection, (4) charting the data, and (5) collating, summarizing, and reporting the results. The optional 'consultation exercise' of the framework was not conducted.

For the purpose of this scoping review, the word "resident" was used interchangeably with "physician in training", as the term "resident" is mainly used in North America and not widely recognized.  This review was guided by the question, 'What is the structure of the existent EBM curricula and its impact on the residents' attitudes, behaviors, skills and knowledge and what are the barriers identified during the EBM curriculum implementation process? Given the complexity of the research question, we decided that a scoping review would be the ideal study design to answer our question.

Information sources

The following databases were searched: PubMed, Embase, ERIC, Scopus and Web of Science from the earliest available date for each database until April 25, 2019, and rechecked on September 4, 2019. Gray literature was also searched.  No limits or barriers were applied, but only articles written in English were included. We screened the reference lists of retrieved studies for relevant publications.

Search and eligibility criteria

PubMed search strategy is described: step #1 "Evidence-Based Practice" [Mesh:NoExp] OR "Evidence-Based Medicine" [Mesh] OR "evidence-based health care" OR "evidence-based healthcare" OR "evidence-based medicine" OR "evidence-based practice" OR "evidence-based care" OR ebm OR ebp OR (critical* AND apprais*) OR ebhc; step #2 "Education, Medical, Graduate" [Mesh] OR gme OR (graduate AND medical AND education) OR postgrad* OR "post-grad" OR "post-graduate" OR "post-graduates" OR residency OR residencies OR resident* OR trainee*; step #3 "Teaching" [Mesh] OR teach OR teaching OR learn* OR curriculum OR curricula OR workshop* OR instruct* OR educat* OR train* OR "journal club" OR "journal clubs" OR lectur* OR module* OR course*; step #4: #1 AND #2 AND #3.

The Appendix contains the search strategies for each database. To identify articles relating to teaching EBM in residencies in curricular format and barriers to teach EBM during the curriculum implementation, the following search terms and their word variants were used: resident, evidence-based, education, teaching, lecture, curriculum, barriers. Exclusion criteria were: student, practitioner, private. The word 'curriculum' was not an inclusion criterion word because not all the articles that described an EBM curriculum had the word 'curriculum' included in the title or abstract. Because the assessment is part of the curriculum development process, studies that reported on an EBM curriculum but didn't have an assessment were excluded. Studies reporting on a single EBM education session or single journal club were excluded as they were thought not to qualify for the curriculum description if they did not address all of the EBM steps.

Selection of  sources of evidence

We aimed to identify all randomized, nonrandomized, and before and after studies that reported on: 1. teaching EBM in a curriculum format to residents and on: 2. barriers during curriculum implementation. To be included, studies had to provide details about the format of their EBM curriculum and report on the evaluation of their curricula, as evaluation is the last step in the curriculum development process. We used the following criteria to screen studies for inclusion in our review: 1. Study design: original studies and abstracts presented at national conferences, quantitative, qualitative, and mixed methods studies were included if they reported on EBM curriculum with or without barriers to teaching EBM to residents; abstracts describing an EBM curriculum and its evaluation were included to avoid the risk of reporting bias if they had been excluded. Systematic reviews on the EBM curriculum and barriers were not included. Their references were manually verified against our search, and the articles not initially included were uploaded for screening; 2. Population: residents of any specialty, working under the supervision of a medical specialist. Medical students and attending physicians were not included as their learning environment, patient care exposure, and motivation are significantly different than for the physicians in training; 3. Outcomes: pertaining to curriculum evaluation, the following categories based on Kirkpatrick's training evaluation model²⁰ were used to assign the types of outcomes for included studies related to teaching EBM to residents: level 1 (attitudes toward EBM and participants' satisfaction), level 2 (EBM skills and knowledge), level 3 (behavior, the degree to which participants apply what they learned during training) and level 4 (results, the degree to which targeted outcomes occur as a result of the training, patient-related outcomes); barriers reported during curriculum implementation were collected, and categorization was reported.

Data charting process

A detailed excel spreadsheet with all the data points that were planned to be collected was developed through collaboration between 2 of the authors (AH and BH). The form was calibrated through concomitant (both authors) analysis and data extraction from various studies. Data charting was done independently and in duplicate. Potentially relevant article abstracts were independently reviewed by two authors (AH and BH), who then met and agreed on article selection. For each potentially eligible study, the full paper was read and reviewed by the same two authors (AH and BH) independently. They both assessed whether the study fulfilled the inclusion criteria. Studies about which both reviewers had doubt or disagreement about inclusion were thoroughly discussed during three consensus meetings, and a common decision was reached.

Two authors (TD) and (BH) performed the data extractions from the full-text articles included, which were then fully reviewed by another author (AH) and any disagreements on data extractions were resolved by consensus between AH and BH. Two authors (BH) and (EN) performed the data extraction from the abstracts included. A prespecified data extraction form was used to extract and collect information from the included studies on the author, year of publication, type of study design, country, residency type, number of participants, study period, specific details of the EBM curriculum structure, length, teachers, content, outcome measures, evaluation methods, results, and barriers. Because we expected to find no homogenous studies in this area, no pooling of data was attempted.

Critical appraisal of individual sources of evidence

To assess the risk of bias of individual studies, two authors (BH and TD) independently rated the quality of the studies at the study level. All disagreements were reviewed and resolved by a third author, AH.

Risk of bias assessment of individual studies

We assessed the quality of the pre-post studies using the National Heart, Lung and Blood Institute Quality Assessment Tool for Before-After (Pre-Post) Studies with no Control Group (Table 1).²¹ Maximum points for each study were 12. We added scores for each criterion together and divided them by 12. The risk of bias rating was: low (75-100%), moderate (25-75%), or high (0-25%). For the randomized controlled trials, quality was assessed according to the Cochrane Collaboration recommended methodology.²² Each criterion was assessed as high, low, or unclear risk of bias. The quality of cohort studies included in this review were assessed using the Newcastle-Ottawa Scale (NOS),²³ which assigns a maximum of 9 points to each study. The assessment scale analyzes three broad perspectives of each study: the selection of the study groups, the comparability of the groups, and the ascertainment of either the exposure or outcome of interest for cohort studies. When a study received 3 or 4 pluses in the selection domain and 1 or 2 pluses in the comparability model and 2 or 3 pluses in the outcome/exposure domain, it was considered to be of good quality (or low risk of bias); if the study received 2 pluses in selection domain and 1 or 2 pluses in comparability domain and 2 or 3 pluses in outcome/exposure domain, it was considered of moderate quality; if a study received 0 or 1 plus in selection domain or 0 pluses in comparability domain or 0 or 1 plus in outcome/exposure domain, it was deemed to be of high risk of bias (Table 1).

Results

Selection of sources of evidence

Nine-thousand-forty-two references were imported for screening, and 214 duplicates were removed. Eighteen extra articles were considered for inclusion from a manual review of the references, and after cross-checking with the articles already included in the initial screening, 8 of them were duplicates and were removed. Four extra articles were suggested by experts in the field and were included for screening. A total of 9064 articles were included after automatically removing the duplicates. Further duplicates were removed manually, and 8842 articles were screened against the title and abstract, and 8742 articles were found irrelevant and were excluded. One hundred articles were assessed for full-text eligibility, and 57 articles were excluded because they didn't meet the eligibility criteria. Only one out of the 18 studies manually added after searching the references was added to this review. One of the 4 studies suggested by the experts fulfilled all inclusion criteria. The final review included 43 articles (29 full-text articles and 14 abstracts) describing the structure of the EBM curriculum. Thirteen of these articles reported on barriers to EBM curriculum implementation.

Characteristics of sources of evidence

Of the 43 studies included in the final systematic review, 29 studies were reported as full text as follows: 18 described the results of quantitative pre-post study design, 5 described the results of quantitative surveys, 4 described prospective cohort studies, two randomized controlled trials, and 14 meeting abstracts. Table 2 and Table 3 present a more detailed description of the studies. Most studies included were performed in countries where English is the primary language: 40 in the United States, 2 in Canada, and one in the Netherlands. Full-text articles included between 4 and 181 residents, and the abstracts included between 11 and 108 residents. One full-text article and two abstracts didn't report on the number of residents enrolled. The specialties included were varied; the majority were internal medicine (12 full studies and ten abstracts). Other specialties included were: emergency medicine (3 full studies^15,16,17 and ² abstracts^24,25), family medicine (7 full text studies), pediatrics (4 full text studies^4,12,17,18 and 1 abstract²⁶), 1 neurology^{27, 1} general surgery²⁸ and ² obstetrics and gynecology^29,30 full studies. One abstract didn't specify the residency type.³¹ The study period reported was as short as six weeks³² and as long as five years.^32,33 One study was a single point in time survey.²⁷ One full-text study³⁴ and 3 abstracts^35-37 didn't report on the study period.

Critical appraisal within sources of evidence

The risk of bias was assessed only for the studies included as full text. The abstracts were not assessed for their quality, as it was appreciated that they contained very limited information about the methodology and possible bias. Also, the full-text surveys were not assessed for risk of bias. The overall quality of the 24 studies evaluated was determined to be moderate or low risk of bias (Appendix 1 and Appendix 2).

Eighteen studies had a moderate risk of bias, and six studies had a low risk of bias. After applying the NOS criteria, 2 of the four observational prospective cohort studies were assessed as having a low risk of bias^38,39 and the other two studies were assessed as having a moderate risk of bias.^40,41 From the randomized controlled trials, Kim and colleagues⁴² had a low risk of bias, and Kortekaas and colleagues⁴³ had a moderate risk of bias. From the 18 pre-post studies, only 3 of them were assessed as being at low risk of bias;^44-46 the rest were considered at moderate risk of bias. None of the studies were evaluated as having a high risk of bias.

Synthesis of results

Details regarding the EBM curriculum structure, length, frequency, teachers, content, outcome measures, evaluation method, and results can be found in Table 2 for the full-text articles and in Table 3 for the abstracts.

EBM curriculum structure

The EBM curricular structure was found to be highly variable from study to study. EBM was taught through multiple venues: workshops,^{37,38,40,41,47} journal club format,^{10,29,30,32,33,4} resident-led sessions,^{10,31,40,44,49-51} didactic lectures,^28,39 librarian sessions^10,33,50 EBM teaching rounds,^44,52,53 online sessions,^47,52 hands-on interactive sessions,^33,36,38 small group sessions,^{4,10,26,37,49,50,54} multiple choice questions,²⁸ modules,^24,35 self-paced modules,⁴⁵ one-on-one meetings,^10,55 morning report,⁴⁴ EBM rotation,⁴⁷ point-of-care teaching,³³ block rotation,⁵⁶ and problem based learning.⁵⁷  The curricular structure has been reported to be developed on the adult learning theory principles in 3 articles.^10,46,58 Most curricula contained more than 2 teaching methods, with only 9 curricula reporting on a single intervention repeated for multiple times: interactive workshops,⁵⁹ EBM teaching rounds,⁵³ Journal Club,^29,60 one-on-one meetings,⁵⁵ self-paced modules,⁴⁵ modules,²⁴ and small group sessions.^26,36,54

EBM curriculum length / frequency / teachers

The majority of the EBM curricula was longitudinal with different lengths, from weekly 1-hour sessions for 4 weeks⁴⁹ to 3 year curriculum.³² Some curricula were reported in blocks of 2 weeks^25,56 7 weeks,⁴⁶ 8 weeks,³⁸ two 2-hour sessions,¹³ daily EBM teaching rounds for one month,⁵³ or intensive 30 hours of EBM training during a month.³³ The frequency of the sessions in each curriculum was variable. In the longitudinal curricula, the sessions were as often as weekly,^34,49 monthly,^29,60 or a few months every year.⁴¹ Only two curricula were designed based on the PGY level, reporting different sessions and activities based on the year of training.^4,32 The rest of the curricula were designed for the generalized audience of all physicians in training. There was a multitude of EBM teachers involved in curriculum delivery in most of the studies, faculty physician, faculty librarian, all level residents, learning coach,⁵⁵ research program coordinator,²⁸ nonphysician biostatistician, research assistant³² and pharmacist.⁵⁰ Four out of the 29 full-text articles and 5 out of the 14 abstracts didn't specify their EBM curriculum teachers.

EBM curriculum content

The majority of the reported EBM curricula address all five of the EBM competencies.¹ Bentley and colleagues⁴⁵ Konen and colleagues³² and Schwartz and colleagues⁶¹ had the main focus on critical appraisal skills. Kitchens and colleagues³⁴ used a series of articles as a basis for critical appraisal teaching. Trickey and colleagues²⁸ and Windish and colleagues⁴⁹ had a major focus on teaching basic and advanced biostatistics.

EBM curriculum development process

Kern's six-step approach to design an experiential curriculum in knowledge translation has been a well-established model for the curriculum development process that has been shown to lead to successful implementation and long-term sustainability.^62,63 The included steps are: 1. problem identification and general needs assessment; 2. needs assessment of targeted learners; 3. goals and objectives; 4. educational strategies; 5. implementation; 6. evaluation and feedback. From the 29 full-text articles, only 5 of them (17%) mentioned that they used Kern's six-step approach for curriculum development.^{33,49,50,52,59} Allan and colleagues and Green and colleagues reported on all the steps as well but without mentioning on the approach.^46,48 George and colleagues reported on all the steps except goals and objectives.⁵⁵ Nicholson and colleagues reported on all the steps without the needs assessment.⁵³

Clinical integration of EBM curriculum

We defined clinical integration of the EBM curriculum if the EBM principles, steps, or concepts were taught in the context of patient care. The EBM curriculum was clinically integrated into 29.3% (10 out of 29) of the full-text articles. Ten percent of the EBM curricula were only partially clinically integrated^4,43,44, and the rest, which was the vast majority, were not clinically integrated.

Outcome measures

Most of the EBM curricula reported on level 2 (EBM skills and knowledge), 45% (13 out of the 29 full-text articles) and 57% (8 out of the 14 abstracts). Fewer articles reported on level 1 (attitude toward EBM and participants' satisfaction): 17% (5 out of 29 full-text studies) and 35.7% (5 out of 14 abstracts). A total of 10 articles (9 full texts and one abstract) reported on both level 1 and level 2. We found 2 full-text articles that also reported on level 3 (behavior, the degree to which participants apply what they learned during training when they are back on the job). Kortekaas and colleagues⁴³ reported on information-seeking behavior, and Ross and colleagues⁵⁹ reported on EBM terminology usage on tape-recorded interactions.

Evaluation method

From the 29 full-text articles, only 4 used the validated Fresno^33,42,44,60 and 1 used the validated Berlin questionnaire⁴⁵ to assess outcomes. Out of the 14 abstracts, 5 used the validated Fresno^{25,31,36,56,57} and 1 used the validated Berlin questionnaire.²⁶ Nelson and colleagues⁵⁸ reported the usage of a 20 question previously validated questionnaire. The rest of the studies used unvalidated surveys and questionnaires to assess their outcomes. Eighteen studies out of 43 (41.8%) reported on the usage of surveys to assess the residents' attitudes and comfort with EBM concepts, and 24 out of 43 studies (55.8%) used some questionnaire (multiple choice examinations) to assess the EBM knowledge and skills. Only three studies (7%) also used qualitative analysis (focus group) to assess the residents' reaction to the EBM curriculum implementation.^4,44,55

Barriers to EBM curriculum implementation

Barriers reported during the EBM curriculum implementation were categorized into three main subsets: barriers to teach, barriers to learning, and barriers to practice EBM. (Table 4) Most commonly, EBM curriculum articles reported on barriers to teach, with time and difficulty to recruit teachers being the most frequent barriers reported. Time has been as well reported as the main barrier to practice EBM.

Does competency in EBM improve after EBM curriculum implementation?

The heterogeneity of the studies interventions and outcome assessments precludes any attempt to combine the results quantitatively. All the published studies and abstracts, independent of the EBM curriculum structure and the evaluation method used, found an improvement in the residents' attitudes and/or EBM skills and knowledge. Bentley and colleagues⁴⁵ reported a 33.4% increase in the Berlin questionnaire posttest. George and colleagues⁵⁵ also reported a significant increase of 31% in the post quiz score. Kenefick and colleagues¹³ showed a 58% improvement in the posttest regarding PICO question and 53% improvement in the searching skills. Al Jabout and colleagues²⁵ reported a 23point improvement in the median Fresno posttest, which was statistically significant at p=0.002. Gupta and colleagues³⁵ had 97% of residents satisfied with the EBM curriculum. There was a 37% improvement in the written posttest in the study by Vogel and colleagues⁵¹ and Walkey and colleagues³⁶ had an impressive improvement in the Fresno test score of 55%. None of the studies reported on long term, post-residency training, retention of EBM skills, or knowledge.

Discussion

Summary of evidence

We reviewed a total of 43 articles, from which 29 full-text articles and 14 abstracts reporting on the EBM curriculum. Overall the body of evidence regarding EBM curriculum structure, efficacy, and barriers is small and of limited quality. The studies included were considerably heterogeneous with respect to their EBM curriculum structure, content, length, evaluation method, and outcome assessment. A common finding within the studies was that multiple teaching strategies were implemented simultaneously. All studies showed an improvement in educational outcomes across different EBM curricula. However, no specific curriculum was found to be superior because there was no direct comparison.

The curricular structure in each article included was described in adequate detail that would allow reproduction and generalizability. This is in contrast with what Green and colleagues⁶⁴ noticed in their systematic review of EBM curricula, that many of the reports suffer from the incomplete description of the curricula. The vast majority of the curricula contain multiple strategies to teach EBM in multiple settings. The most common integral session in the EBM curricula reported was journal club, followed by small group sessions, resident-led sessions, hands-on interactive sessions, and workshops. Also, the majority of the EBM curricula reported has been comprehensive, addressing all of the EBM competencies. It is known that offering multiple EBM interventions in multiple learning venues throughout residency training will facilitate the reinforcement and retention of key skills and behaviors.¹⁰ Some suggest adopting a spiral approach to EBM teaching, whereby concepts increase in complexity and are reinforced throughout learning experiences.⁶⁵

Moving EBM instruction out of the classroom and into the clinical arena is crucial for the application.⁴ When residents have the time and opportunity to integrate EBM competencies during real-time patient care, they shift from a paternalistic to a participatory decision-making style.⁵³ This goes along with the adult-learning theory principles and should be considered in future EBM curriculum development. In his review, Coomarasamy and colleagues⁶⁶ reported that only integrating EBM teaching with clinical activities was associated with increased EBM competency across all levels (attitudes, knowledge, skills, behaviors) in medical postgraduates.

Our findings are consistent with Green and colleagues⁵² regarding lack of attention to curriculum development principles, as only less than a third of the full-text articles reported on the curriculum development and implementation steps. Eight of the 9 studies that explained the EBM curriculum development and implementation process were published in the last 20 years, which can suggest the hypothesis that in the previous 20 years, with the progression of science, the introduction of reporting guidelines and also with the wide availability of the internet, scientists improved their knowledge in curriculum development and reporting. The lack of attention to curriculum development principles could be explained by the lack of knowledge and training in curriculum development of the curriculum developers.

Very few studies reported on the Kirkpatrick' evaluation model level 3, behavioral change, and no studies reported on level 4, results/patient outcomes. These findings are similar to previous systematic reviews data on medical education interventions.⁴⁶ This could be due to the fact that assessing level 3 and level 4 has been previously described as difficult to achieve because it is significantly more time-consuming and requires a more demanding evaluation. Level 4 should be viewed as the primary goal of any training program, but it is very challenging to assess whether certain patient outcomes can be linked to the EBM curriculum intervention or not. In a systematic review of 599 research articles published in three major medical education journals, patient outcomes accounted for only 0.7% of all articles.⁶⁷

Ilic and colleagues⁷ describe criticism of the current state of the evidence in this field as being the lack of a uniform, validated assessment tool that can measure all aspects of the EBM competencies. Our findings are similar, with the vast majority of curricula being evaluated by non-validated tools. A 2006 systematic review identified 104 unique instruments with reasonable validity for evaluating some domains of evidence-based practice and may be targeted to different evaluation needs.⁶⁸ The majority (90%) were not high-quality instruments with established inter-rater reliability, objective outcome measures, and three or more types of established validity. To this day, only the Fresno test of competence in evidence-based medicine⁶⁹ and the Berlin Questionnaire⁷⁰ are high-quality instruments identified as evaluating EBM knowledge and skills across at least 3 of the 5 EBM steps.

Time has been mentioned as the most frequent barrier to teach and practice EBM. This finding is consistent with prior literature. Dijk and colleagues¹⁷ reported in his systematic review on barriers to residents' practicing EBM that limited available time has been the most often mentioned and primary barrier for residents. Another important barrier that has been very frequently reported was the lack of EBM experts/teachers. This barrier has significant implications in the successful implementation of any EBM curriculum and constitutes one of the first steps in curriculum development.

Strengths and limitations

The strengths of our scopic review include systematic reproducible search of the literature, explicit methods which limit bias, reliable and accurate conclusions and the ability to bring a vast body of literature to an update review. One of the main limitations of this review is the likely publication bias. Because negative studies are less likely to be written, submitted or published, especially in medical education, the studies' results might be skewed toward reporting the positive effects of the interventions. We also noticed that the published data on EBM curriculum is geographically limited and thought that it might be due to publication bias and might further bias the results by not fully representing the EBM curricula from many other countries. It is also possible that there are many innovative and effective EBM curricula that have never been reported. Another major limitation is the quality of the study design of the full manuscripts included. The heterogeneity of the interventions and outcomes constitutes a limitation in the ability to draw a conclusion based on this data. One other limitation worth noting is the small sample size in the majority of these studies, as a convenience sample was most often used, and the studies were single centre—none of the studies reported on patient care impact assessment.

Conclusions

This is the first scoping review to answer the question regarding the structure of EBM curricula and barriers found during the EBM curriculum implementation process. EBM competencies are necessary for providing high-quality patient care within physicians in training education. The current body of literature available to guide educators in EBM curriculum development and barriers to implement and teach EBM is enough to constitute a strong scaffold for developing any EBM curriculum. Teaching EBM concepts can use any method and can be integrated into almost any setting. Special attention should be paid to the potential barriers during the curriculum implementation, and ways to overcome them should be identified early in the curriculum development process. Given the amount of time and resources needed to develop and implement an EBM curriculum, it is imperative to follow the curriculum development steps and use validated assessment tools. During that development, medical educators should design rigorous evaluation strategies with more meaningful clinical outcomes that would evaluate Kirkpatrick's learning levels 3 and 4. Further studies should focus on studying long term retention of the EBM concepts, skills and knowledge as they remain unknown for now.

Conflict of Interest

The authors declare that they have no conflict of interest.

Supplementary materials

References

1. Guyatt GH, Drummond R. Users' guides to the medical literature: essentials of evidence-based clinical practice. New York: McGraw-Hill; 2002.
2. Price A and Burls A. Increased water intake to reduce headache: learning from a critical appraisal. J Eval Clin Pract. 2015; 21: 1212-1218.
   Full Text PubMed
3. Elnicki DM, Halperin AK, Shockcor WT and Aronoff SC. Multidisciplinary evidence-based medicine journal clubs: curriculum design and participants' reactions. Am J Med Sci. 1999; 317: 243-246.
   Full Text PubMed
4. Chitkara MB, Boykan R and Messina CR. A longitudinal practical evidence-based medicine curriculum for pediatric residents. Acad Pediatr. 2016; 16: 305-307.
   Full Text PubMed
5. ACGME. ACGME common program requirements (residency). Chicago (IL): Accreditation Council for Graduate Medical Education; 2018 [cited 27 January 2020]; Available from: https://acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRResidency2019.pdf.
6. Straus SE, Glasziou P, Richardson WS, Haynes, RB. Evidence-based medicine: how to practice and teach it. 4th ed. New York: Elsevier; 2011.
7. Ilic D and Maloney S. Methods of teaching medical trainees evidence-based medicine: a systematic review. Med Educ. 2014; 48: 124-135.
   Full Text PubMed
8. Windish DM, Huot SJ and Green ML. Medicine residents' understanding of the biostatistics and results in the medical literature. JAMA. 2007; 298: 1010-1022.
   Full Text PubMed
9. Knowles MS. Application in continuing education for the health professions: chapter five of "Andragogy in Action". Mobius. 1985; 5: 80-8100.
   Full Text PubMed
10. Halalau A, Falatko J, Mi M. Application of adult learning theory in teaching evidence based medicine to residents. Journal of Medical Education (Tehran). 2016;15(4):185-93.
11. Hatala R, Keitz SA, Wilson MC and Guyatt G. Beyond journal clubs. Moving toward an integrated evidence-based medicine curriculum. J Gen Intern Med. 2006; 21: 538-541.
    Full Text PubMed
12. Ahmadi N, McKenzie ME, Maclean A, Brown CJ, Mastracci T and McLeod RS. Teaching evidence based medicine to surgery residents-is journal club the best format? A systematic review of the literature. J Surg Educ. 2012; 69: 91-9100.
    Full Text PubMed
13. Kenefick CM, Boykan R and Chitkara M. Partnering with residents for evidence-based practice. Med Ref Serv Q. 2013; 32: 385-395.
    Full Text PubMed
14. Carpenter CR, Kane BG, Carter M, Lucas R, Wilbur LG and Graffeo CS. Incorporating evidence-based medicine into resident education: a CORD survey of faculty and resident expectations. Acad Emerg Med. 2010; 54-61.
    Full Text PubMed
15. Harris J, Kearley K, Heneghan C, Meats E, Roberts N, Perera R and Kearley-Shiers K. Are journal clubs effective in supporting evidence-based decision making? A systematic review. BEME Guide No. 16. Med Teach. 2011; 33: 9-23.
    Full Text PubMed
16. Coomarasamy A, Taylor R and Khan KS. A systematic review of postgraduate teaching in evidence-based medicine and critical appraisal. Med Teach. 2003; 25: 77-81.
    Full Text PubMed
17. van Dijk N, Hooft L and Wieringa-de Waard M. What are the barriers to residents' practicing evidence-based medicine? A systematic review. Acad Med. 2010; 85: 1163-1170.
    Full Text PubMed
18. Maggio LA, ten Cate O, Chen HC, Irby DM and O'Brien BC. Challenges to learning evidence-based medicine and educational approaches to meet these challenges: a qualitative study of selected EBM curricula in U.S. and Canadian medical schools. Acad Med. 2016; 91: 101-106.
    Full Text PubMed
19. Arksey H and O'Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8: 19-32.
    Full Text
20. Kirkpatrick DL. Techniques for evaluating training programs. Journal of the American Society for Training Directors. 1959;13(11):3-9.
21. Quality assessment tool for before-after (pre-post) studies with no control group. Bethesda (MD): National Heart, Lung, and Blood Institute; [cited 27 January 2020]; Available from: https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools.
22. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane handbook for systematic reviews of interventions. Version 6.0. Cochrane; 2019 [cited 27 January 2020]; Available from: https://training.cochrane.org/handbook/archive/v6.
23. Wells GA, Shea B, O'Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottowa scale (NOS) for assessing the quality of nonrandomized studies in meta-analyses. Ottawa (ON): Ottawa Hospital Research Institute; [cited 27 January 2020]; Available from: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp.
24. Moll J. Academic and clinical integration: a novel residency curriculum translating evidence into action. Acad Emerg Med. 2012;19(4 Suppl 1):S403.
25. Al Jalbout N, Regan L, Hinson J, Mc Quown C. A novel, online educational series improves emergency medicine resident knowledge in evidence-based medicine. Acad Emer Med. 2019;26(Suppl 1):S262.
26. Vu T, Alexander B, Skey D and Chen JG. P-value: a novel, resident-driven, interactive evidence-based medicine curriculum increases EBM knowledge and confi-dence (research abstract). Acad Pediatr. 2017; 17: e14.
    Full Text
27. Burneo JG, Jenkins ME and Bussière M. Evaluating a formal evidence-based clinical practice curriculum in a neurology residency program. J Neurol Sci. 2006; 250: 10-19.
    Full Text PubMed
28. Trickey AW, Crosby ME, Singh M and Dort JM. An evidence-based medicine curriculum improves general surgery residents' standardized test scores in research and statistics. J Grad Med Educ. 2014; 6: 664-668.
    Full Text PubMed
29. Lentscher JA, Batig AL. Appraising medical literature: the effect of a structured journal club curriculum using the Lancet handbook of essential concepts in clinical research on resident self-assessment and knowledge in milestone-based competencies. Mil Med. 2017;182(11):e1803-8.
30. Letterie GS and Morgenstern LS. The journal club. Teaching critical evaluation of clinical literature in an evidence-based environment. J Reprod Med. 2000; 45: 299-304.
    PubMed
31. Kluesner N. A dedicated EBM curriculum integrated into journal club increases and sustains EBM competency: an innovation in EBM curriculum. West J Emerg Med. 2015;16(Suppl):S2.
32. Konen JC and Fromm BS. A family practice residency curriculum in critical appraisal of the medical literature. Fam Med. 1990; 22: 284-287.
    PubMed
33. Shaughnessy AF, Gupta PS, Erlich DR and Slawson DC. Ability of an information mastery curriculum to improve residents' skills and attitudes. Fam Med. 2012; 44: 259-264.
    PubMed
34. Kitchens JM and Pfeifer MP. Teaching residents to read the medical literature: a controlled trial of a curriculum in critical appraisal/clinical epidemiology. J Gen Intern Med. 1989; 4: 384-387.
    Full Text PubMed
35. Gupta R, Julliard K, Barrientos J. Interdisciplinary evidence-based medicine course design for NAS milestone assessment through online adaptive learning strategies. J Gen Intern Med. 2014;29(Suppl 1):S524.
36. Walkey B, Fairchild DG. Evidence-based medicine in primary care: a patient-centered curriculum for resident training. J Gen Intern Med. 2006;21(Suppl 4):S172-3.
37. Zipkin D, Roberts R, Feldstein D. A curriculum in evidence-based medicine improves resident knowledge and skills. J Gen Intern Med. 2010;25(Suppl 3):S435-6.
38. Keddis MT, Beckman TJ, Cullen MW, Reed DA, Halvorsen AJ, Wittich CM, West CP and McDonald FS. Senior internal medicine residents' confidence with essential topics in evidence-based medicine taught during internship. J Grad Med Educ. 2011; 3: 490-496.
    Full Text PubMed
39. Kohlwes RJ, Shunk RL, Avins A, Garber J, Bent S and Shlipak MG. The PRIME curriculum. Clinical research training during residency. J Gen Intern Med. 2006; 21: 506-509.
    Full Text PubMed
40. Gehlbach SH, Bobula JA and Dickinson JC. Teaching residents to read the medical literature. J Med Educ. 1980; 55: 362-365.
    Full Text PubMed
41. Luciano GL, Visintainer PF, Kleppel R and Rothberg MB. Ambulatory morning report: a case-based method of teaching EBM through experiential learning. South Med J. 2016; 109: 108-111.
    Full Text PubMed
42. Kim S, Willett LR, Murphy DJ, O'Rourke K, Sharma R and Shea JA. Impact of an evidence-based medicine curriculum on resident use of electronic resources: a randomized controlled study. J Gen Intern Med. 2008; 23: 1804-1808.
    Full Text PubMed
43. Kortekaas MF, Bartelink ME, Zuithoff NP, van der Heijden GJ, de Wit NJ, Hoes AW. Does integrated training in evidence-based medicine (EBM) in the general practice (GP) specialty training improve EBM behaviour in daily clinical practice? A cluster randomised controlled trial. BMJ Open. 2016;6(9):e010537.
44. Aneese AM, Nasr JA and Halalau A. A prospective mixed-methods study evaluating the integration of an evidence based medicine curriculum into an internal medicine residency program. Adv Med Educ Pract. 2019; 10: 533-546.
    Full Text PubMed
45. Bentley S, Slovis BH and Shah K. Introduction of a novel evidence-based medicine curriculum in emergency medicine. MedSciEduc. 2018; 28: 497-501.
    Full Text
46. Green ML and Ellis PJ. Impact of an evidence-based medicine curriculum based on adult learning theory. J Gen Intern Med. 1997; 12: 742-750.
    Full Text PubMed
47. Thom DH, Haugen J, Sommers PS and Lovett P. Description and evaluation of an EBM curriculum using a block rotation. BMC Med Educ. 2004; 4: 19.
    Full Text PubMed
48. Allan GM, Korownyk C, Tan A, Hindle H, Kung L and Manca D. Developing an integrated evidence-based medicine curriculum for family medicine residency at the University of Alberta. Acad Med. 2008; 83: 581-587.
    Full Text PubMed
49. Windish DM. Brief curriculum to teach residents study design and biostatistics. Evid Based Med. 2011; 16: 100-104.
    Full Text PubMed
50. Zeblisky K, Birr RA and Sjursen Guerrero AM. Effecting Change in an evidence-based medicine curriculum: librarians' role in a pediatric residency program. Med Ref Serv Q. 2015; 34: 370-381.
    Full Text PubMed
51. Vogel EW. Evaluation of a black curriculum in evidence-based medicine for internal medicine residents. J Gen Intern Med. 2000;15(Suppl 1):S48.
52. Friedman S, Sayers B, Lazio M, Friedman S and Gisondi MA. Curriculum design of a case-based knowledge translation shift for emergency medicine residents. Acad Emerg Med. 2010; 42-48.
    Full Text PubMed
53. Nicholson LJ and Shieh LY. Teaching evidence-based medicine on a busy hospitalist service: residents rate a pilot curriculum. Acad Med. 2005; 80: 607-609.
    Full Text PubMed
54. Vom Eigen KA, Nestler EO, Allen SR. Teaching evidence-based medicine to primary care internal medicine residents: subjective and objective outcomes. J Gen Intern Med. 2002;17(Suppl 1):S237.
55. George P, Reis S and Nothnagle M. Using a learning coach to teach residents evidence-based medicine. Fam Med. 2012; 44: 351-355.
    PubMed
56. Kao RL, Justice EM, Easterby-Gannett S, Belleh E, Elliott DJ. Faculty-librarian collaboration on an evidence based medicine block curriculum in internal medicine residency. J Gen Intern Med. 2012;27(Suppl 2):S193-4.
57. Mroz M, Carroll T. Building a needs-based EBM curriculum within ambulatory block education. J Gen Intern Med. 2019;34(Suppl 2):S791.
58. Nelson B, Ingard C and Nelson D. Teaching trainees how to critically evaluate the literature - a crossover study at two pediatric residency programs. Int J Med Educ. 2017; 8: 137-141.
    Full Text PubMed
59. Ross R and Verdieck A. Introducing an evidence-based medicine curriculum into a family practice residency--is it effective? Acad Med. 2003; 78: 412-417.
    Full Text PubMed
60. Mohr NM, Stoltze AJ, Harland KK, Van Heukelom JN, Hogrefe CP and Ahmed A. An evidence-based medicine curriculum implemented in journal club improves resident performance on the Fresno test. J Emerg Med. 2015; 48: 222-229.
    Full Text PubMed
61. Schwartz MD, Kalet AL, Mukohara K. Beyond clever nihilism: teaching residents evidence-based medicine (EBM). J Gen Intern Med. 2000;15(Suppl 1):S43.
62. Kern DE. Curriculum development for medical education: a six-step approach. Baltimore: Johns Hopkins University Press; 1998.
63. Windish DM, Reed DA, Boonyasai RT, Chakraborti C and Bass EB. Methodological rigor of quality improvement curricula for physician trainees: a systematic review and recommendations for change. Acad Med. 2009; 84: 1677-1692.
    Full Text PubMed
64. Green ML. Graduate medical education training in clinical epidemiology, critical appraisal, and evidence-based medicine: a critical review of curricula. Acad Med. 1999; 74: 686-694.
    Full Text PubMed
65. Harden RM. What is a spiral curriculum? Med Teach. 1999; 21: 141-143.
    Full Text PubMed
66. Coomarasamy A and Khan KS. What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review. BMJ. 2004; 329: 1017.
    Full Text PubMed
67. Chen FM, Bauchner H and Burstin H. A call for outcomes research in medical education. Acad Med. 2004; 79: 955-960.
    Full Text PubMed
68. Shaneyfelt T, Baum KD, Bell D, Feldstein D, Houston TK, Kaatz S, Whelan C and Green M. Instruments for evaluating education in evidence-based practice: a systematic review. JAMA. 2006; 296: 1116-1127.
    Full Text PubMed
69. Ramos KD, Schafer S and Tracz SM. Validation of the Fresno test of competence in evidence based medicine. BMJ. 2003; 326: 319-321.
    Full Text PubMed
70. Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH and Kunz R. Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ. 2002; 325: 1338-1341.
    Full Text PubMed