Evaluating an e-learning course's impact and challenges on genomic literacy among medical professionals

Introduction

Recent advancements in medical genetics and genomic medicine have resulted in a significant expansion of their application in day-to-day clinical practice and a subsequent increasing demand for genomic literacy among healthcare professionals. To improve genomic literacy, defined as ‘the capacity to obtain, process, understand, and use genomic information for health-related decision-making’,^1,2 many countries hold workshops for healthcare workers, have online information repositories for the general public and rare disease patients and families, and develop guidelines, standards, and national programmes for implementation of genomic education into formal education.³ The advancements in medical genetics and genomic medicine have necessitated all medical professionals to be proficient in genetic treatment and understand the pertinent ethical issues; indeed, education in medical genetics is becoming increasingly important in undergraduate education.⁴

While genetic information requires ethical consideration and appropriate information management owing to its private and sensitive nature, as medical information, it also needs to be shared by medical professionals.⁵ Therefore, it is essential that medical professionals have the opportunity to learn the ethical issues and to practice appropriate genetic medicine. However, there have traditionally been relatively few opportunities to study medical genetics as part of a typical medical education curriculum. To address both advances in genetics and genomics as well as trends in medical education, the Association of Professors in Human and Medical Genetics (APHMG) updated its medical school core curriculum in genetics in 2013.⁶ Various methods of genetic education have been examined, such as active learning materials and needs-based education, and reports on their effectiveness have been published.⁷ Simulation-based learning environments have been shown to improve the quality of medical education by allowing students to interact with patients and patient data in a virtual environment.⁸ The means for simulation-based exercises are diverse, from programmable simulator mannequins or computer-based virtual reality to nontechnical applications using methods such as a situation-based roleplaying, standardised patients, or case-based, problem-based learning.⁸ The simulation-based learning environment has been reported to increase students’ learning, intrinsic motivation, and self-efficacy, and to increase the perception perceived relevance of medical educational activities;⁸  however, working medical professionals often lack time to attend in-person classes. For busy medical professionals, e-learning is a very convenient mode of study as it allows them to study where and when they like. While there have been numerous reports on the learning effects of e-learning in medical education, few have evaluated medical genetics e-learning courses for health care professionals.

In Japan, the Model Core Curriculum for Medical Education, revised in 2016, added ‘genetic and genomic medicine’ to ‘physiological changes, pathogenesis, diagnosis, and treatment throughout the body’.⁹ Currently, medical educators are endeavouring to reform curricula from the viewpoint of competency-based medical education (CBME).¹⁰ The quality of learner assessment is very important to control quality regarding whether such competencies are achieved.¹¹ In Japan, it is necessary to promote seamless training of physicians in undergraduate education and post-graduate clinical training;¹² therefore, undergraduate and post-graduate education in genetic medicine is currently under consideration.¹

Previous research on genetic education seminars for healthcare professionals in Japan reported that self-assessments of satisfaction and understanding improved among those who attended more than one session.¹³ In addition, methods for appropriately assessing outcomes have been explored in genetic medicine, including self-assessment systems and new methods for performance assessment.⁴ However, course evaluation is largely based on self-reported evaluation and less so on objective or behavioural evaluation. In addition, there are no reports on the effectiveness or evaluation of genetic education for physicians or other health professionals as post-graduate or professional education.

The University of Tsukuba Hospital conducted an e-learning course on genetic medicine with the aim of improving the genetic literacy of medical professionals. The learning objective of this course was to develop physician competency for understanding genetic medicine and handling of genetic information at our hospital and to acquire basic knowledge to perform genetic testing. The aim of this study is to evaluate the learning impact of our e-learning course on enhancing genomic literacy among physicians and to explore its efficacy in teaching appropriate genetic medicine practices.

Methods

Outline of the e-learning

University of Tsukuba Hospital began conducting its e-learning course on genetic medicine for medical professionals from 2019. The course comprises eight comprehensive sections: i) ‘Introduction to the Genetic Medicine Department,’ offering an overview of the practices within genetic medicine; ii) ‘Basic knowledge of genetic information,’ which covers the fundamentals of genes, genomes, and personal data; iii) ‘Basic knowledge to perform genetic testing,’ details the procedures and unique characteristics of genetic testing; iv) ‘Genetic testing and genetic counselling in practice,’ which explores clinical scenarios involving genetic counselling; v) ‘Genetic information and electronic medical record description,’ outlines the proper protocols for documenting medical information; vi) ‘How genetic information can be accessed for genetic testing,’ describes the specific methods used to access genetic data; vii) ‘Genetic testing,’ enumerates the various genetic tests available; and viii) ‘Considerations and guidelines for genetic medicine,’ which presents critical considerations and standards for practice in genetic medicine. The course lasted approximately 90 minutes.

Study design and Participants

This study was reviewed and exempted by the Institutional Review Board, Office of Human Research Ethics, University of Tsukuba Hospital. Study participants were medical professionals at the University of Tsukuba Hospital who were taking the e-learning course on genetic medicine. Based on Kirkpatrick’s four-stage evaluation concept,^14,15 we conducted an evaluation of the e-learning course using participant feedback and assessment of learning and behaviour outcomes. Written informed consent was obtained from the participants. Before the e-learning course, participants completed a behavioural questionnaire and answered a test on their knowledge of genetic medicine (Appendix 1),(Appendix 2). After the course, the participants took the same test again and evaluated the course. Three months later, the participants completed a second behavioural questionnaire. Figure 1 provides information on the study design and flow of study participation.

Figure 1

Study design and flowchart of study participation

Study participants completed the e-learning course and then completed a behavioural questionnaire three months later. They took a genetic medicine knowledge test before and after the e-learning course and evaluated the program upon completion. Self-assessment of the participants’ genetic practice behaviour before and three months after the e-learning was also carried out.

The study population comprised 124 medical professionals, of whom 103 were physicians, who took the e-learning course between January 2020 and September 2022. A total of 96 (93.2%) of these physicians were included in the analysis. Behavioural assessment was analysed using data gathered from 28 participants who responded to questionnaires both before and three months after the course. Exclusion criteria were those who did not respond to the pre- and post-course questionnaires and tests. Only physicians were included in this study.

Data collection

In collecting data, we obtained cooperation from the Education Cloud Promotion Office, which was established to promote e-learning and improve the quality of education using Information and Communication Technology at the University of Tsukuba. A learning management system was used to record lectures, produce and distribute content, and electronically record e-learning attendance and response status (Appendix 3).

Quantitative data

Participants evaluated the e-learning course across the following criteria: learning location, implementation time, length of the lecture, ease of reading text and slides, content of the lecture, clinical usefulness, and operation of the e-learning software. Responses were given using a 4-point Likert scale (1 = Poor, 2 = Fair, 3 = Good, and 4 = Very good). The participants then evaluated their understanding of the learning content for each of the eight chapters using a 4-point scale (1 = Did not understand at all, 2 = Understand somewhat, 3 = Understood well, 4 = Understood very well). They completed pre- and post-test to measure any changes in genetic medicine knowledge due to the e-learning course. The degree of mastery was scored in terms of the number of correct answers to 10 questions that pertained to ethical considerations of genetic. The questions test were prepared by referring to the Act on the Protection of Personal Information ¹⁶and the Medical Fee Score Table,¹⁷ and were discussed and prepared by the co-researchers, several genetic specialists outside the hospital, general medical staff, and general medical staff in the hospital. Behavioural evaluations with regards genetic medicine-related actions/behaviour were conducted using a self-assessment before and three months after completion of the e-learning course. Respondents answered either ‘Yes’ or ‘No’ for each item of the behavioural questionnaire. The questionnaire was prepared by several medical professionals involved in genetic medicine, based on the guidelines on genetic testing and diagnosis in medicine⁵and the hospital’s manual on the prevention of medical accidents.

Qualitative data

A self-administered, open-ended questionnaire was conducted after the e-learning course in which participants were asked to write comments and opinions on the management of genetic information, workshops, and the e-learning course.

Data analysis

Quantitative data

The self-assessment scores for the genetic medicine knowledge test are expressed as means with corresponding standard deviation (SD). Pre- and post-test scores were compared using paired-samples t-tests. Mann-Whitney U test was used to compare the difference between physicians with clinical experience in genetic medicine and those without. The mean test scores of physicians with and those without such experience were compared using independent t-tests.

Pearson correlation was used to determine the association between two numerical variables. Univariate and multivariate analyses were conducted using the regression model to evaluate potential confounders. Multivariate regression analysis was used to evaluate the contribution of the measured parameters to increase in pre- to post-test scores as the dependent variable.

All statistical tests were two-sided, and p-values of ≤0.05 were deemed to be of statistical significance. The effect size was calculated to demonstrate the strength of differences between pair measures and estimated using Cohen’s d. All statistical analyses were conducted using SPSS 28.0 (IBM Corporation, Armonk, New York, USA).

Qualitative data

Qualitative analysis of the written responses was performed to generate emergent themes.

Results

The characteristics of the participants (physicians) are listed in Table 1. Of these participants, 51% (n = 49) had clinical experience in genetic medicine and paediatricians accounted for the highest proportion (31.3%; n = 30). The time required to implement the e-learning course varied from ≤15 to 90 minutes, with the highest percentage of respondents taking 30-60 minutes (55.8%; n = 53).

Participant feedback

Approximately 80% (n = 75-93) of respondents expressed positive opinions (Good/Very good) regarding the content of the e-learning course and its usefulness for medical practice (see Table 2).

Learning assessment

As presented in Table 3, Mann-Whitney U testing indicated no significant differences in the self-reported scores pertaining to understanding of genetic medicine between physicians with and without experience in genetic medicine. For example, physicians with clinical experience (Mdn=3.24) underperformed those who had not it (Mdn=3.30) on the self-reported scores of Basic knowledge of genetic information (U=1104.5, p=0.669)

The mean scores for the pre- and post-tests regarding genetic medicine knowledge were 71.25 and 74.58, respectively; a significant increase (t ₍₉₅₎ =2.70, p=0.008), with a small-medium effect size (d=0.275). For physicians with no clinical experience in genetic medicine, the pre- and post-test scores increased significantly from 68.94 to 75.53 (t₍₄₆₎=3.82, p<0.001), respectively, with a medium effect size (d=0.557). For those with clinical experience, scores did not differ significantly from 73.47 to 73.67 (t₍₄₈₎ =0.12, p=0.903), respectively, with a small effect size (d=0.018). In addition, Mann-Whitney U test revealed that physicians with clinical experience in genetic medicine (Mdn=73.47) significantly outperformed those without (Mdn=68.94) on pre-test scores (U=1424.5, p=0.037). Meanwhile, physicians with clinical experience in genetic medicine (Mdn=73.67) did not outperform those without (Mdn= 75.53) on post-test scores (U=1056.5, p=0.466) (see Figure 2).

With or without clinical experience in genetic medicine, years of clinical experience, gender, and pre-test scores were used as independent variables in the multivariate regression analysis. According to the results of the multivariate regression with the increase in pre- to post-test scores as the objective variable, with or without clinical experience in genetic medicine (β=−0.208, t=2.21, p=0.03) and pre-course test scores (β=−0.459, t=4.91, p<0.001) were significantly and negatively associated, while years of clinical experience (β=0.188, t=2.06, p=0.042) was significantly positively associated (R²=0.278, F_(4,88)=9.87, p<0.001).

Behaviour assessment (translation of learning to clinical setting)

The rate of those with opportunities to work with patients with genetic diseases in their practice remained unchanged at 96.4% (or 27/28 people) in the three months after the e-learning course. The implementation rate for behaviour(Appendix 1)is 72.2%-100% (or 13/18-18/18 people) before e-learning and 74.1%-100% (or 20/27-18/18 people) after 3 months of e-learning. The implementation rate increased in the following behaviours: (1) keeping genetic test reports in the Genetic Medicine Department or asking for them to be scanned into electronic medical records with security; (2) having noted in the electronic medical record that you provided genetic counselling (or test explanation) when performing the test; (3) explaining to the patient in advance if there is a possibility that a variant of unknown significance may be found when performing genetic testing; and (4) confirming in advance that the original genetic test result report belongs to the patient when informing them of the genetic test results. However, no statistically significant differences were found for the behaviour questions before and 3 months after the e-learning course.

Qualitative data

Responses indicate the usefulness of the e-learning course, but also the difficulty of understanding the contents in a single viewing. The importance and challenges of genetic information management were also raised in terms of ethical, legal, and social issues. In the following, we explore the different primary themes and provide clarifying quotes.

Usefulness of e-learning

‘It was a useful e-learning course’. (No. 10, male, 30s, no clinical experience in genetic medicine, cardiology; No. 35, male, 40s, no clinical experience in genetic medicine, endocrinology and metabolism; No. 74, male, 30s, ≤5 years clinical experience in genetic medicine, oncology; No. 90, male, 50s, ≤5 years clinical experience in genetic medicine, otolaryngology.)

“In a relatively compact time, I was able to learn the ethical knowledge necessary for genetic treatment”. (No. 12, male, 30s, no clinical experience in genetic medicine, cardiology.)

Difficulty in understanding in one time

“I couldn’t understand all the details just by listening to it once”. (No. 76, female, 30s, ≤5 years clinical experience in genetic medicine, paediatrics.)

Learning tool issues

“It was difficult to find the video in the e-learning”. (No. 65, female, 50s, no clinical experience in genetic medicine, urology.)

“The test was difficult”. (No. 76, female 30s, ≤5 years clinical experience in genetic medicine, paediatrics.)

The importance of genetic information management

“It is important to respect the right to privacy and to deal with various ethical issues.” (No. 38, 50s, With clinical experience in genetic medicine within 5 to 10 years, Obstetrics and Gynaecology, Female)

Issues in the genetic information management

“If there is a template for the procedure of performing genetic testing, it would be easier to omit the description”. (No. 87, female, 40s, 5 to 10 years clinical experience in genetic medicine, breast, thyroid, and endocrine surgery.)

Figure 2

Pre/post-test scores for e-learning among physicians (n=96)

Discussion

The findings of this study show that our e-learning course on genetic medicine was highly evaluated by about 80% (n = 75-93) of the participants, indicating a high level of satisfaction. Many physicians commented on the usefulness of the e-learning course and the importance genetic information management. Regarding knowledge level, in the objective evaluation, post-course test scores significantly increased, indicating a positive learning effect. In particular, the scores of physicians with no experience in genetic practice increased significantly. However, no significant differences were found in the self-assessment of understanding between those with and without genetic practice experience. No learning effects were evident in the behavioural assessment.

Effectiveness and Challenges of E-learning seminars

Previous studies have demonstrated that e-learning courses appeared to be an effective strategy for residents’ knowledge acquisition and indicated computer-based learning platforms to be feasible and helpful for the learning processes.¹⁸ E-learning courses have a number of advantages, such as a uniform format, accessibility, and incentive for regular use.¹⁸Furthermore, it is generally believed that computer-based instruction offers learner flexibility in terms of controlling and pacing of the learning process, provides for diverse learning needs, gives opportunities for practice through simulation, increases information retention, and reduces instructional time.¹⁹

In-line with previous reports that attest to the benefits of e-learning, the mean score for the post-test revealed a significant increase from that of the pre-test. The study findings indicate that although there were no differences in self-assessment scores regarding understanding of the course content, post-test scores were significantly higher than pre-test scores, particularly for physicians without experience in genetic practice; years of clinical experience was significantly and positively related to increased scores from pre- to post-test. It was suggested that physicians with no experience in genetic practice may utilize e-learning to acquire new knowledge, while physicians with experience in genetic medicine may find useful ways of learning by considering how education should be based on years of clinical experience. Physicians with fewer years of clinical experience may have limitations in e-learning, suggesting that more practice-based educational methods need to be devised.

American adult educator Malcolm Knowles initially developed the adult learning theory in 1968, with the following key points: Readiness to Learn, Orientation of Learning, Motivation to Learn, Reason to Learn, Self-conceptualization, and Experience.¹⁹In line with this Knowles’ theory, we found that clinical experience would be associated with learning effectiveness. We found that physicians with no experience in genetic medicine had lower pre-test scores and thus had room to grow, while those already with experience in this field, had high pre-test scores and thus the course offered them little new knowledge. It was suggested that the e-learning course should be divided into different levels with different educational content depending on whether the physician has experience in genetic medicine or not, and provide advanced content for physicians with experience. Previous studies suggest that medical specialists believe that experiential learning in genetic medicine was necessary to develop the confidence and skills needed for clinical care and they expected to look to experts within their own medical specialty who have gained genomics expertise for specific and contextualised support as they develop their knowledge.²⁰Learners are said to benefit from communicating with other learners (horizontal communication), from putting questions to experts (vertical communication), and, in the event of technical problems, receiving prompt support.²¹ Clinical teachers have stated that the links between theory and practice would be easier to achieve through dialogue and discussion ¹⁸of which case studies and group discussion forums are useful for physicians. In summary, it was considered necessary to provide physicians with genetic clinical experience with more educational content relevant to their own clinical speciality, and include experiential and blended learning opportunities. In addition, as genetic medicine involves other professions, the use of interprofessional education may be useful.²²

Limitations

This study has several limitations that should be acknowledged. First, owing to the limited number of respondents who responded to the behavioural assessment survey, it is difficult to draw any firm conclusions from our assessment of learning at the behavioural level. Thorough assessment of learning at the behavioural level benefits from long-term observation; therefore, our study design has further limitations in terms of its capacity to adequately assess learning at the behavioural level. Assessment through on-site work, referred to as work-based assessment,¹¹ is important, however, reliable assessment methods were considered too difficult to implement in the current study. In addition, the results of our study do not reflect the situation of Japanese healthcare professionals as a whole; the number of non-physician healthcare professionals who took the e-learning course was limited. Therefore, further consideration is needed regarding the provision of a more accessible e-learning system/course.

Future studies should examine how to educate physicians with experience in medical genetics and how to evaluate the learning effects of such education interventions. We should also focus on medical genetics education for non-physician healthcare professionals and consider ways to assess their improvement in genomic literacy.

Conclusions

In the current study, the usefulness of our e-learning course in medical genetics was demonstrated at a learning level for physicians with no previous experience in medical genetics. For doctors with experience in medical genetics, for whom no significant improvement in post-test scores was observed, it is necessary to provide more practical content and consider different learning methods. This study focussed on assessing the efficacy of our medical genetics e-learning course for physicians; therefore, further studies are required to develop and assess the efficacy of such courses for other medical professionals. Genetic medicine is an issue of growing importance within medical education; however, for busy medical professionals, finding time for continuing education is a challenge. Furthermore, as genomic medicine develops, the learning needs of physicians are expected to change. Therefore, qualitative and quantitative studies are needed to develop and evaluate educational interventions that will meet the evolving needs of physicians with regard to their ongoing education in genetic medicine. The results of this study point to the potential for similar courses to aid medical genetics education for health professionals, which, with developments in our understanding of human genetics and gene therapy, is becoming an ever more important feature of medical care.

Acknowledgements

The authors wish to thank all medical professionals who participated in this study. We would like to thank the Center for Medical Education and Training of the University of Tsukuba Hospital for creating the E-learning system. We are indebted to Mr T. Mayers, Medical English Communications Center, University of Tsukuba, for grammatical revision of this paper.

Conflict of Interest

The authors declare that they have no conflicts of interest.

Supplementary materials

References

1. Hurle B, Citrin T, Jenkins JF, Kaphingst KA, Lamb N, Roseman JE and Bonham VL. What does it mean to be genomically literate?: National human genome research institute meeting report. Genet Med. 2013; 15: 658-663.
   Full Text PubMed
2. Whitley KV, Tueller JA and Weber KS. Genomics education in the era of personal genomics: academic, professional, and public considerations. Int J Mol Sci. 2020; 21: .
   Full Text PubMed
3. Zimani AN, Peterlin B and Kovanda A. Increasing genomic literacy through national genomic projects. Front Genet. 2021; 12: 693253.
   Full Text PubMed
4. Makita T. The future of clinical genetic education based on adult learning theory 1. Why is adult education in Japan failing? Gene & Medicine. 2021;11(4):18-23.
5. Guidelines on genetic testing and diagnosis in medicine. Japan Medical Association. February 2011, revised March 2022 [Cited 22 November 2024]; available from: https://jams.med.or.jp/guideline/genetics-diagnosis_e_2022.pdf.
6. Plunkett-Rondeau J, Hyland K and Dasgupta S. Training future physicians in the era of genomic medicine: trends in undergraduate medical genetics education. Genet Med. 2015; 17: 927-934.
   Full Text PubMed
7. Hagiwara N. Application of active learning modalities to achieve medical genetics competencies and their learning outcome assessments. Adv Med Educ Pract. 2017; 8: 817-829.
   Full Text PubMed
8. Makransky G, Bonde MT, Wulff JS, Wandall J, Hood M, Creed PA, Bache I, Silahtaroglu A and Nørremølle A. Simulation based virtual learning environment in medical genetics counseling: an example of bridging the gap between theory and practice in medical education. BMC Med Educ. 2016; 16: 98.
   Full Text PubMed
9. Model Core Curriculum for Medical Education in Japan; 2016. [Cited 20 April 2024]; Available from: https://www.mext.go.jp/content/20230323-mxt_igaku-000028108_00005.pdf.
10. Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990; 65: 63-67.
    Full Text PubMed
11. Onishi H. Learner assessment and competency-based education. Japan Journal of Health Professional Development. 2017;4(1):21-8.
12. Tanabe M. The seamless transition of medical education from pre- to post-graduate training: approach from outcome-based education. Med Educ. 2021;52(4):305-11.
13. Sone M, Kondo A, Nambara S, Iba T, Mori K, Itou M, et al. Genetic education from chromosome analysis laboratories: from the questionnaire survey. J Jpn Soc Chromosome Genet Test. 2016;34(1):29-34.
14. Kirkpatrick DI. Evaluation of training. In: Craig R, Mittel I, editors. Training and development handbook. New York: McGraw Hill; 1967.
15. Nishigori H, Saiki T, Tagawa M. Curriculum program evaluation in medical education. Med Educ. 2014;45(2):79-86.
16. Government of Japan. Personal information protection commission; 2019. [Cited 20 April 2024]; Available from: https://www.ppc.go.jp/files/pdf/personal_law.pdf.
17. Ministry of Health, Labour, and Welfare of Japan. Medical fee score table; 2019. [Cited 20 April 2024]; Available from: https://www.mhlw.go.jp/stf/seisakunitsuite/bunya/0000188411_00026.html.
18. Verheijden MJH, Martens H and Heeneman S. A dermatology E-learning programme is perceived as a valuable learning tool in postgraduate general practice training. Int J Med Educ. 2021; 12: 169-178.
    Full Text PubMed
19. Knowles MS, Holton EF III, Swanson RA, Robinson PA. The adult learner. 9th ed. London: Routledge; 2020.
20. McClaren BJ, Crellin E, Janinski M, Nisselle AE, Ng L, Metcalfe SA and Gaff CL. Preparing medical specialists for genomic medicine: continuing education should include opportunities for experiential learning. Front Genet. 2020; 11: 151.
    Full Text PubMed
21. de Leeuw RA, Westerman M and Scheele F. Quality indicators for learner-centered postgraduate medical e-learning. Int J Med Educ. 2017; 8: 153-162.
    Full Text PubMed
22. Maeno T, Takayashiki A, Anme T, Tohno E, Maeno T, Hara A. Japanese students’ perception of their learning from an interprofessional education program: a qualitative study. Int J Med Educ. 2013;4:9-17.