Relationship between web-based illness scripts and the performance of medical students in orthopedic surgery placements

Introduction

Several educational strategies are used to promote medical knowledge and clinical reasoning to enhance students' skills in making an accurate diagnosis.¹ One of these strategies used for teaching in the clinical setting is illness scripts.² The illness script is based on the script theory, which proposes a high-level conceptual knowledge structure kept in long-term memory, which can be retrieved from the storage to integrate the patients' information into an appropriate clinical context.^2-4The illness script has three main components, including epidemiological factors or enabling conditions, pathophysiological processes, and the consequences such as signs and symptoms of the disease.^3,5 Management can also be included in an 'extended' illness script.^6,7 The concept of illness scripts is to provide a cognitive framework, rather than pure memorization, for diseases and help the student to organize knowledge structures to interconnect information in the memory to apply to either an examination or real-life patients.^4,5,8 In other words, these scripts are designed to help guide the medical students to link various kinds of knowledge in their pre-clinical years and in clinical settings. The goal of these scripts is to assist medical students to form a disease framework and to facilitate diagnostic problem-solving.^9,10

 After repeated practicing with the illness scripts, the medical student should be able to recognize a pattern, discard unnecessary info, and provide a diagnosis from the given information.^6,10,11 In case scenario problems, the medical student should be able to distinguish relevant from irrelevant information from a case presentation and then gather the vital features from each main component of the scripts, such as peak incidence of the disease in adult/elderly patients (epidemiological factors), cause of disease from degenerative change/infection, or characteristics of pain (signs and symptoms of the disease).¹² To distil this information into a differential diagnosis and identify the most likely diagnosis, they would integrate these screened clinical details with their stored knowledge. Developing a list of possible diseases and practicing diagnostic skills, leading to further appropriate investigation and management, contribute to their future medical expertise.¹³

 Various traditional script-based teaching methods, including problem-based learning, concept mapping, and self-explanations, have been used with medical students.2 However, nowadays, medical education is being gradually converted from traditional teaching to electronic or online learning.¹⁴ Web-based illness scripts as a part of online learning can be beneficial in providing diagnostic experiences to solve actual clinical problems.⁹ The advantage of online learning is that it can provide students a variety of experiences leading to a greater quantity of knowledge.¹⁵ Many factors may influence online learning success or failure, such as cultural resistance and already overworked teachers or students. ^16-18 Shifting to an entire online learning medical curriculum is challenging and must be done carefully with appropriate and timely feedback loops to ensure a successful transition.¹⁴ Creating additional online educational courses containing illness scripts apart from traditional education may be preferable. Therefore, in our institution, we generated a series of case-based illness scripts as part of the online course for our students to increase their performance and integrate pre-clinical knowledge of orthopedics into a clinical setting.

This study aimed to determine whether web-based illness scripts correlated with the performance of fifth-year medical students in orthopedic surgery placements. We hypothesized that the illness scripts would be positively correlated with the students' performance. The results from this study will help improve the adaptation and improvement of illness scripts in future online orthopedic courses.

Methods

Study design and participants

We conducted a retrospective study of fifth-year medical students studying at the Orthopedics Department of the Faculty of Medicine at Prince of Songkla University in Thailand, following their rehabilitation and orthopedic surgery placements in the academic year of 2020. We enrolled all medical students who took part in the online educational course, except students who missed the examination in the last week of their rotation or failed the overall academic year. This resulted in 83 students in the study. All of them were Thai and had never failed any summative assessments. This study was approved by the Institutional Review Board, Faculty of Medicine, Songklanagarind Hospital, Prince of Songkla University.

Illness scripts and student performance

In our curriculum, the fifth-year medical students are required to attend an orthopedic surgery placement for one month to gain experience in orthopedics. Due to the short orthopedics and rehabilitation rotation period, we have developed an additional online educational course in the learning management system (LMS). This online course provides learning resources, learning assignments, electronic lectures, online quizzes, and illness scripts. On the first day of the rotation, the fifth-year medical students are instructed to do as many optional assignments as possible, including illness scripts.

The web-based illness script exercises, especially for the fifth-year medical students, were generated by a group of orthopedic surgeons with more than five years of experience in undergraduate medical education and prepared for fifth-year medical students only. These special illness scripts were all web-based, case-based practical problems, covering the ten necessary domains the fifth-year students needed to learn. The illness scripts included 8 cases of elbow and hand problems, 10 cases of shoulder problems, 10 cases of foot and ankle problems, 8 cases of hip problems, 12 cases of knee problems, 12 cases of spine problems, 6 cases of pediatric orthopedic problems, 3 cases of orthopedic oncology problems, and 5 cases of orthopedic trauma problems. Each case scenario began with a stem that described the clinical presentation of a patient followed by two sections of questions, beginning with a multiple-choice question (MCQ) for diagnosis and followed by four short-answer questions linked to the first question with further exploration of signs and symptoms, epidemiology, and pathophysiology of the disease. The students could do any individual script only one time during their one-month placement. After they finished and submitted their answers, the correct answers were displayed, which they could review at any time. The completed illness scripts were registered after they answered the questions, each script had a total of five questions, and one point was awarded for each correct answer: one point per question with a total possible score of 5 points per script. The number of completed scripts was automatically recorded in our database. Only approved teachers were able to access the database.

In the last week of the rotation, each student's performance was assessed by examination to ensure they had achieved the learning objectives. Our summative assessment comprised practical and theoretical examinations evaluated by objective structured clinical examination and MCQs, respectively. We selected only the MCQ scores for data analysis as our MCQs are mostly comprehensive questions asking about possible diagnoses, consistent with the illness scripts questions, and do not rely on any subjective differences among the examiners. Every month, 40 questions were randomly selected from a question bank containing 200 questions. All MCQs were in the English language and began with a stem describing a particular clinical problem followed by five options, from which there was one best answer. The students needed to choose the correct answer to get a score. The examination results were recorded in both paper and electronic documents.

Data collection

The demographic data of the included medical students were obtained from the Faculty of Medicine database with permission. We recorded these data in electronic documents, including gender, race, marital status, and cumulative grade point average (GPA), calculated by dividing total grade points from all academic years by the number of total credit hours.

We retrieved one-year data from our LMS database. The completed scripts and scores of each case scenario of the illness scripts from the included medical students were collected at the end of the 2020 academic year. The total number of completed scripts of each medical student was calculated and recorded individually. The scores from the web-based illness scripts in each of the ten orthopedic problem domains were summarized and recorded electronically. Additionally, we asked for permission from the undergraduate education division of the Orthopedics Department to access the entire summative assessment results of the fifth-year medical students throughout this academic year. We recruited the MCQ scores obtained from the total year records and documented them in record forms to apply in data analysis.

Data analysis

Before performing the comparative and correlation analyses, we separated the students into two groups based on how many of the web-based illness scripts each student had completed. The cutoff point of 60% completion of the illness scripts was used since we regarded this as an acceptable response rate,¹⁹ which was approved by the Orthopedic Undergraduate Education Committee. The students completing 60% or more of the scripts were assigned to the high-response group, while students completing fewer than 60% were assigned to the low-response group.

The characteristics of the students, completion of the illness scripts, and examination scores in summative assessment were compared between the two groups. Categorical variables are presented in absolute numbers with percentages and compared between the two groups using the Chi-square test. Continuous variables are presented in means and standard deviations and compared between the two groups using a two-sample t-test or Mann-Whitney U-test, as appropriate. Pearson correlation coefficients (r) were used to test the correlations between the completion of the scripts, illness script scores, and examination scores. We also did a linear regression analysis to identify the relationship between illness script scores and examination scores in the high-response group. Independent variables were illness script scores from the ten orthopedic problem domains, and total scores were controlled for gender and grade point average, while the dependent variable was the examination scores from the MCQs in summative assessment. The associations are presented using regression coefficients, and statistical significance was considered at p<0.05. All data were analyzed using the R program version 4.0.3 (The R Foundation for Statistical Computing, 2020, Vienna, Austria).

Results

Demographics

Of the 83 fifth-year medical students in one academic year, 41 (49.4%) were female, and 42 (50.6%) were male. All were Thai and single. The average cumulative GPA was 3.3 (SD=0.32). After separation into two groups, 56 (67.47%) were in the high-response group and 27 (32.53%) were in the low-response group. When characteristics, examination scores, and completed illness scripts were compared between the two groups (Table 1), there were no significant differences in gender, cumulative grade point average, or examination scores between the two groups. However, there was a statistically significant difference in the completed illness scripts between the two groups (t₍₂₇₎=13.72, p<0.001).

Relationship between completed illness scripts and examination scores

Negatively weak correlations without statistical significance were found between the completion of illness scripts and examination scores (r=-0.09, p=0.44).

Relationship between illness script scores and examination scores

We found a negative, weak, and non-significant correlation between overall illness script scores and examination scores (r=-0.03, p=0.78). Specifically, we explored the relationship between illness script scores in each domain and examination scores only in the high-response group. The results of the linear regression analysis are shown in Table 2. We found no significant relationship between illness script scores in any orthopedic domain and examination scores, even in the high-response group. There was also no significant relationship between total illness script scores and examination scores after controlling for gender and grade point average (β=-1.07, p=0.143).

Discussion

We assumed, following various studies, that practicing clinical reasoning from illness scripts could improve the ability to organize medical knowledge and framework thinking.^6,9 Therefore students who completed a higher number of web-based illness scripts would be more likely to show better performance in their orthopedics rotation. Contrary to our hypothesis, however, we found no positive relationship between the number of web-based illness scripts completed and the students' performance as assessed through summative examination results in their orthopedic surgery placement.

There are several possible explanations as to why the number of illness scripts completed did not correlate with the performance of the students. One explanation could be the performance assessment method. After one month of placement, we conducted summative examinations to assess the medical students' performance in relation to the learning objectives, including appropriate diagnosis, treatment principles, disease prevention, and fundamental orthopedic procedures. Most of the questions in the examination were comprehensive. In contrast, illness script questions consisted of MCQs and short answers, looking for only a diagnosis and its vital features, including the disease's signs and symptoms, epidemiology, and pathophysiology, following a previous case-based illness script pattern.¹² Inferring from the study results, the medical students who completed more illness scripts for diagnosis may have had more potential to integrate the newly learned knowledge into their existing fund of knowledge, leading to obtaining higher scores in the comprehensive questions seeking a correct diagnosis. Nevertheless, they may not have obtained high total MCQ examination scores since they could not apply the stored knowledge acquired from the illness scripts for diagnosis to choose the appropriate answer for treatment or management. Moreover, non-diversified type of questions for evaluation was suspicious for non-significant results. In our research, we compared the results by calculating only MCQ scores. The MCQ is one of the preferred assessment tools for evaluating undergraduate medical students,²⁰ and although assessment with MCQs is reliable, various types of theoretical examinations for evaluating students' knowledge and performance should be utilized.^21,22 Our results may have been different if other methods of assessment had been included in the study.

Another possible factor that may have impacted our results was the large number of online illness scripts which expanded the students' workloads.^23,24 We intended to provide scripts thoroughly covering all common orthopedic problem domains, and created a total of 74 scenarios. The students needed to spend a significant amount of time working on each script while at the same time learning from the answer page. However, they had only one month in the orthopedic placement, and doing all of the online illness scripts may have reduced the time available to prepare for the summative examination. Because of the time limitations, the students may have paid more attention to the examination to attain high scores, enhancing their cumulative GPA and perhaps influencing future careers.²⁵ As a result, some students may have gained high examination scores despite being in the low-response group.

Additionally, online learning courses have their own limitations, as reported in previous studies,^14,26 which identified various problems or barriers such as skill deficits in practice, lack of technology for reaching the online course, a negative attitude amongst instructors, or learner motivation. We suspect that the personal learner motivation factor may have influenced the number of completed illness scripts assigned as optional and thus leading to our expectations not being met.

Although our study failed to confirm that completing our online illness scripts would increase the summative examination scores, we still believe that studying online illness scripts can be very useful in helping the students develop and organize the knowledge required for skilful disease diagnosis, including understanding which further investigations may be required, and treatments, all of which are essential for providing proper management for their patients.^9,13 The benefits from illness scripts are realized more in medical students who have the dedication to practice and learn from modern instruction tools such as web-based scripts.⁴ We learned from the first use of this teaching technique that some modifications of web-based illness scripts in orthopedics are required, especially by considering an appropriate number of illness scripts, adding some scripts about orthopedic disease management, deleting some scripts about diagnoses, and combining the scripts with face-to-face teaching.^27,28

There were some limitations to our study. This research was a retrospective study which has inherent weaknesses. Some important data were not recorded, and we could not control all of the baseline characteristics and attitudes of students that may be confounding factors, but this was unavoidable due to the study design. Moreover, this was conducted in a single-center academic hospital with a small number of medical students. Thus, we did not have much information in our online database system for analysis. Another limitation was that we studied the relationship of illness scripts and performance in fifth-year students in an orthopedic surgery placement only; hence, the generalization of the results in this study to other departments may be limited. Lastly, we used a fixed cut-off point without randomized allocation to separate the high-response and low-response groups, leading to possible selection bias and internal validity inconsistencies.

Conclusions

In this study, the use of web-based illness scripts in orthopedics was not correlated with the medical students' performances, as assessed through examination results, following a one-month placement. We identified several factors which potentially obscured the benefits of the web-based illness scripts. For modifiable factors, a perhaps too high number of illness scripts only in diagnosis and inadequate time for practice in a short one-month rotation may have negatively affected the performance assessment of the medical students in the study. These factors should be considered and adjusted in the proper context for each medical school when creating illness scripts and assigning them as tasks.

A further multi-center, prospective study with a larger sample size from more than one academic year is suggested to identify the relationship between web-based illness scripts in orthopedics and the students' performance. It is also recommended to investigate the factors that positively or negatively affect this relationship in future studies. The strengths and weaknesses of web-based illness scripts in orthopedics are still uncertain.

Acknowledgements

We would like to thank Assoc. Prof. Boonsin Tangtrakulwanich of the Orthopedics Department for his contribution in revision and editing, and Mr. Dave Patterson of the International Affairs Office of the Faculty of Medicine, Prince of Songkla University, for his assistance in proofreading the English of this study. We would also like to express our appreciation to the undergraduate medical education team in the Orthopedics Department for their assistance.

Conflicts of Interest

The authors declare that they have no conflict of interest.

References

1. Bowen JL. Educational strategies to promote clinical diagnostic reasoning. N Engl J Med. 2006; 355: 2217-2225.
   Full Text PubMed
2. Lubarsky S, Dory V, Audétat MC, Custers E and Charlin B. Using script theory to cultivate illness script formation and clinical reasoning in health professions education. Can Med Educ J. 2015; 6: 61-70.
   PubMed
3. Keemink Y, Custers EJFM, van Dijk S and Ten Cate O. Illness script development in pre-clinical education through case-based clinical reasoning training. Int J Med Educ. 2018; 9: 35-41.
   Full Text PubMed
4. Hennrikus EF, Skolka MP and Hennrikus N. Applying metacognition through patient encounters and illness scripts to create a conceptual framework for basic science integration, storage, and retrieval. J Med Educ Curric Dev. 2018; 5: 2382120518777770.
   Full Text PubMed
5. Custers EJ. Thirty years of illness scripts: Theoretical origins and practical applications. Med Teach. 2015; 37: 457-462.
   Full Text PubMed
6. Custers EJFM, Boshuizen HPA and Schmidt HG. The role of illness scripts in the development of medical diagnostic expertise: results from an interview study. Cognition and Instruction. 1998; 16: 367-398.
   Full Text
7. Monajemi A, Rikers RM and Schmidt HG. Clinical case processing: a diagnostic versus a management focus. Med Educ. 2007; 41: 1166-1172.
   Full Text PubMed
8. Barrows HS and Feltovich PJ. The clinical reasoning process. Med Educ. 1987; 21: 86-91.
   Full Text PubMed
9. Lee A, Joynt GM, Lee AK, Ho AM, Groves M, Vlantis AC, Ma RC, Fung CS and Aun CS. Using illness scripts to teach clinical reasoning skills to medical students. Fam Med. 2010; 42: 255-261.
   PubMed
10. Charlin B, Tardif J and Boshuizen HP. Scripts and medical diagnostic knowledge: theory and applications for clinical reasoning instruction and research. Acad Med. 2000; 75: 182-190.
    Full Text PubMed
11. Schmidt HG and Rikers RM. How expertise develops in medicine: knowledge encapsulation and illness script formation. Med Educ. 2007; 41: 1133-1139.
    Full Text PubMed
12. Levin M, Cennimo D, Chen S and Lamba S. Teaching clinical reasoning to medical students: a case-based illness script worksheet approach. MedEdPORTAL. 2016; 12: 10445.
    Full Text PubMed
13. Monajemi A, Schmidt HG and Rikers RM. Assessing patient management plans of doctors and medical students: an illness script perspective. J Contin Educ Health Prof. 2012; 32: 4-9.
    Full Text PubMed
14. O'Doherty D, Dromey M, Lougheed J, Hannigan A, Last J and McGrath D. Barriers and solutions to online learning in medical education - an integrative review. BMC Med Educ. 2018; 18: 130.
    Full Text PubMed
15. Mooney GA and Bligh JG. Information technology in medical education: current and future applications. Postgrad Med J. 1997; 73: 701-704.
    Full Text PubMed
16. Greenhalgh T. Computer assisted learning in undergraduate medical education. BMJ. 2001; 322: 40-44.
    Full Text PubMed
17. Bediang G, Stoll B, Geissbuhler A, Klohn AM, Stuckelberger A, Nko'o S and Chastonay P. Computer literacy and E-learning perception in Cameroon: the case of Yaounde Faculty of Medicine and Biomedical Sciences. BMC Med Educ. 2013; 13: 57.
    Full Text PubMed
18. Ozuah PO. Undergraduate medical education: thoughts on future challenges. BMC Med Educ. 2002; 2: 8.
    Full Text PubMed
19. Johnson T, Owens L. Survey response rate reporting in the professional literature. Proceedings of the section on survey methods, American Statistical Association; Nashville, TN: 2003. [Cited 10 January 2021]; Available from: http://www.srl.uic.edu/publist/Conference/rr_reporting.pdf.
20. Nimavat N, Singh S, Fichadiya N, Sharma P, Patel N, Kumar M, Chauhan G and Pandit N. Online medical education in India - different challenges and probable solutions in the age of COVID-19. Adv Med Educ Pract. 2021; 12: 237-243.
    Full Text PubMed
21. Preston R, Gratani M, Owens K, Roche P, Zimanyi M and Malau-Aduli B. Exploring the impact of assessment on medical students’ learning. Assessment & Evaluation in Higher Education. 2020; 45: 109-124.
    Full Text
22. Moss E. Multiple choice questions: their value as an assessment tool. Curr Opin Anaesthesiol. 2001; 14: 661-666.
    Full Text PubMed
23. Hill MR, Goicochea S and Merlo LJ. In their own words: stressors facing medical students in the millennial generation. Med Educ Online. 2018; 23: 1530558.
    Full Text PubMed
24. Friedman E, Karani R and Fallar R. Regulation of medical student work hours: a national survey of deans. Acad Med. 2011; 86: 30-33.
    Full Text PubMed
25. Barber S, Brettell R, Perera-Salazar R, Greenhalgh T and Harrington R. UK medical students' attitudes towards their future careers and general practice: a cross-sectional survey and qualitative analysis of an Oxford cohort. BMC Med Educ. 2018; 18: 160.
    Full Text PubMed
26. Muilenburg LY and Berge ZL. Student barriers to online learning: a factor analytic study. Distance Education. 2005; 26: 29-48.
    Full Text
27. Childs S, Blenkinsopp E, Hall A and Walton G. Effective e-learning for health professionals and students--barriers and their solutions. A systematic review of the literature--findings from the HeXL project. Health Info Libr J. 2005; 20-32.
    Full Text PubMed
28. Smits PB, de Graaf L, Radon K, de Boer AG, Bos NR, van Dijk FJ and Verbeek JH. Case-based e-learning to improve the attitude of medical students towards occupational health, a randomised controlled trial. Occup Environ Med. 2012; 69: 280-283.
    Full Text PubMed