Digital competencies are more and more required in everyday work, and training future healthcare professionals in digital health is highly important.
Aim of this study was to assess medical students’ gain of knowledge by participation in a teaching module “Digital Health”, and to evaluate their attitudes towards digital health and its role in medical education.
Students of the module were asked to complete a questionnaire and a multiple-choice-test before and after completing the classes. Students of the same educational level in different modules served as reference group.
34 students took part ( 17 “Digital Health group”; 17 “reference group”). There was no significant difference in pre-existing knowledge between the groups. After having completed the module, participants reached significantly higher scores, compared to their preexisting knowledge ( 0.05) and the reference group ( 0.05). Most students found that digital medicine is not sufficiently represented in undergraduate medical education, but will influence everyday work of physicians in the next five years.
Students showed a high awareness for the impact of digital health on physicians’ work. The results suggest that the format can sufficiently transfer knowledge about digital health. Teaching of digital knowledge and competencies should be firmly implemented into medical education to form digitally competent future doctors.
The past decade was characterized by a global paradigm shift in the perception and estimation of digitalization in healthcare . It is commonly postulated that the COVID-19-Pandemic has acted as a catalyst of digitalization in healthcare and has sped up a development that had started long before [2, 3]. When first digital technologies emerged around fifty years ago, their use for healthcare was mainly seen in accelerating and optimizing analogous tools and communication . In the last decade, it became apparent that digitalization can meet the demand for ubiquitous data availability in healthcare - one example being the conversion of paper files into digital files, and plans for the implementation of electronic health records in various countries . Digital tools, subsumed by the term health information technology, seem indispensable for today’s patient management and professional communication. Meanwhile, “Digital Health” implies a lot more. Mobile medical applications (apps), wearable devices, digital therapeutics, telehealth/telemedicine or digital biomarkers have a formative influence on medical care, making some authors naming digital health as “cultural transformation of traditional healthcare” . Some innovations, such as the use of artificial intelligence in analyzing big data even have the potential to change established ways of diagnostics or therapies to such an extent that they may be referred to as disruptive technologies [7, 8].
Consequently, digital competencies are more and more required in everyday work . Hence, training future healthcare professionals in the field of digital health in understanding, using, and developing digital health applications is of utmost importance [10, 11]. Various international examples have already reported on courses and electives in undergraduate medical education, usually with a focus on a defined field of the digital health sector. Examples include the use of the internet and gamification  and telehealth . In postgraduate and undergraduate teaching for other health professions e.g. for pharmacy , nursing , and psychiatry , digital aspects of the working environment are taught at some institutions. In the United States, some medical schools have been reported to offer digital health teachings, reaching from support for individual research and development projects to formal curricular integration as a “digital health track” . In Germany, the Federal Government had made considerable investments in the fields of digital education and issued a “Digital Agenda”, aiming to increase digital competence and – amongst others – research into digital learning and teaching processes . Some German medical schools have already begun to teach digital health courses or electives . Yet teaching of digital health topics still does not have its regular place in medical education in Germany and Europe [19, 20, 21, 22].
After first experiences with a new elective study module on digital health , this study aimed at assessing medical students’ gain of knowledge and their attitude toward digital health related topics, compared to students who had not participated in the module.
A 3-week elective module “Digital Health” had been established at Charité – Universitätsmedizin Berlin medical school in winter semester 2019/2020. It comprises of 60 educational units of 45 minutes each and was designed to teach digital health including ethical, legal, and social implications by a variety of teaching formats with theoretical seminars as well as hands-on practical training in digital technologies. The elective is offered in third year of medical school amongst a variety of other elective classes with different topics (for detailed description of the elective module see reference ).
In winter semester 2020/2021, a questionnaire on attitudes towards digital and a multiple-choice-(MC-)test on digital health knowledge (see detailed description below) were given out to the students who had chosen the elective “Digital Health” before (pre-test) and after (post-test) completing the class. As a reference group, students of the same educational level who had chosen elective courses other than “Digital Health” were asked via an email distributing list to complete the MC-test and the questionnaire on their attitudes as well. All procedures performed in studies involving human participants were in accordance the Declaration of Helsinki of 1964 and its later amendments. Participation in the study was voluntary, and data collection anonymous. The local ethics committee gave its approval for the module evaluation (EA1/236/19).
2.2Questionnaires on attitude regarding digital health
For the purpose of this study, a questionnaire with 5 items (5-point-Likert-scaled) was analyzed to inquire students’ attitudes and mind-set towards topics of digital health, covering the following aspects:
1. Perception of the role of digital medicine in future daily routine of physicians,
2. Whether digital medicine is appropriately taught in medical school,
3. Perceived awareness for digital health among fellow medical students,
4. Whether future physicians should be educated regarding aspects of digital health,
5. Whether digital medicine should be mandatory during undergraduate medical education.
2.3Multiple-choice-test on digital health
An interdisciplinary, interprofessional team of experts designed a set of 21 multiple-choice questions adhering to the guidelines for design of MC-questions [23, 24]. The contents matched the previously identified digital health competencies, skills, and knowledge defining the curricular elective module “Digital Health” (as mentioned above). Required knowledge to correctly answer the questions comprised of general and specific knowledge about digital health, e.g. legal and ethical aspects, health apps, use of artificial intelligence (AI), aspects of telemedicine, digitally supported medical devices, digitally supported communication between patients and medical staff, and framework conditions for digital health in society. All questions underwent a multilevel review process before being accepted for the final test.
Validity of both the MC-test and questionnaire was ensured by means of an additional multi-level review process. Both instruments underwent pilot-testing with 11 volunteer students who had taken the course during the previous semester. Minor adjustments were made accordingly. Reliability was not yet formally assessed due to small sample size and the only biannual enrolment of students into the elective course. Students’ answers to questionnaires as well as multiple-choice-tests were transferred electronically via Survey Monkey (Survey Monkey, San Mateo, CA, USA). Descriptive analysis of questionnaire answers was performed using Excel (Microsoft Inc., Redmond, WA, USA). Statistical analysis of multiple-choice-test results was performed with Mann-Whitney-U and Wilcoxon Signed Ranks test using SPSS statistics program (Version 26, IBM, Armonk, NY, USA). Level of statistical significance was assumed as 0.05.
34 students participated in the study. The elective was chosen by 17 students in winter semester of 2020/2021 (“Digital Health” group). Students of the same educational level who had chosen different elective courses served as “reference group” ( 17).
3.2Questionnaires on attitude regarding digital health
Regarding the questions on attitude regarding digital health as a baseline, over 90% of all students (31/34) agreed or strongly agreed that digital medicine will influence everyday work of physicians in the next five years. In the “Digital Health” group this number was even greater with almost 95% (16/17) (see Fig. 1).
When asked about the situation in current undergraduate medical education, more than 85% of all students (29/34) found that digital medicine is not sufficiently engrained within current undergraduate medical curricula. When asked about their opinion on their fellow students’ knowledge about digital health, answers of students differed considerably: about 29% (10/34) agreed or strongly agreed, 41% (14/34) disagreed or strongly disagreed, and 29% (10/34) were undecided when asked whether they perceive their fellow students to be literate of “Digital Health” (see Fig. 1).
While most students (91%, 31/34) agreed or strongly agreed that future physicians should be educated regarding aspects of digital health early and specifically during medical education, for the question of whether digital medicine should be mandatory during undergraduate medical education, more students were indecisive (18%, 6/34 students). Yet the majority (79%, 27/34 students) still agreed or strongly agreed to this statement (see Fig. 1). In the group of “Digital Health” participants, this picture was painted more clearly.
3.3Multiple-choice-test on digital health
Overall, 34 students completed the MC-test on digital health as a baseline (without having received specific education on the topic). The average score was 9.26 out of 21 points, equaling approx. 441%. There was no significant difference in pre-existing knowledge between “Digital Health” group (average 9.24 points/44.0%/ 17) and the reference group (9.29 points/44.3%/ 17) ( 0.05).
After the elective, the “Digital Health” group scored significantly higher compared to their pre-test results and reached 57.1% correct answers ( 0.018). The “Digital Health” group reached thus also significantly higher test results than the reference group’s pre-test ( 0.022) (see Table 1.)
|Group||Pre-test (score)||Pre-test (%)||Post-test (score)||Post-test (%)||-value|
Level of significance between testing a difference in the post-test result of the “Digital Health” group versus the pre-test result of the reference group.
The present study amongst third year medical students provides the missing proof of an efficient transmission of knowledge by an undergraduate module in digital health. In brief, it showed increased awareness for the importance and potential future impact of digital health on physicians’ work and a strong agreement amongst students that digital health should be part of the education of future doctors, while its implementation in the actual German undergraduate medical education is not perceived as sufficient. This is in line with previous findings . The evaluation of pre- and post-test-results of medical students participating in the course showed a significant increase in knowledge, also compared to a control group without particular teaching on the topic.
The here presented module has been able to provide the participating medical institutes of the medical school with first-hand experience in the teaching of digital health which they may benefit from for further teaching formats. This can be especially important since a timely inclusion of digital health into the curriculum has been demanded by students’ and faculties’ organizations for some years now [25, 26] and is planned to be introduced as a requirement with the implementation of the new German National Catalogue of Competency-based Learning Goals NKLM  for all German medical schools. This will also bring the need, the requirement, and subsequently the implementation of standardized testing of the medical students’ digital literacy.
Education in general and medical education in particular always faces the problem of measuring and proving the success of educational methods through different means of assessment. When introducing the elective class “Digital Health” into the medical school’s curriculum first subjective general information on students’ and teachers’ perception of the topic and the way it was taught was collected . In a response to that paper, Grzeska et al. recommended assessing individual knowledge by a standardized exam before and after participating in the elective rather than merely enquiring subjective opinions . As all educational efforts aim on increasing students’ knowledge and skills in a particular field, we hope to have found a way of assessing knowledge in a standardized way.
Even though there are growing numbers of European and international examples of implemented digital health courses and classes at medical schools, a preexisting standardized tool to measure the grade of general digital literacy of medical students or doctors is still not known to the authors of this paper. Such as in this paper, teaching and research projects usually establish their own way of assessing or certifying specifically their formulated teaching goals [12, 17, 29]. Here, focusing on the mere gain of knowledge, specifically created multiple-choice tests were used . Having participated and completed the elective “Digital Health” resulted in significantly better results of the MC-test, suggesting that the format of the module and the chosen teaching methods prove sufficient to actually transfer the specific knowledge. The pre-test results were comparable between students who chose the elective “Digital Health” and those who chose different subjects, suggesting the effectiveness of the course format itself. The test questions covered many different topics, including artificial intelligence, law and robotics, as well as the presence of a specific mindset. They were therefore suitable to not only prove the learning of the specific knowledge, but in its sum allows for attesting the students’ general basic knowledge of digital health and its real-life applications. Still, future studies will also have to include analysis of students’ gain in skills and competencies .
Students’ answers to the questionnaire on attitude regarding digital health revealed a high awareness of the importance of the topic regardless of the group. This is in line with findings from a study on European level that has shown the gap between students’ perceived need and academic offers of digital health teaching . Since both groups shared these attitudes, it can be postulated that a higher intrinsic motivation and interest in the topic was not the reason for better results in the post-test, which could have been assumed among students choosing an elective module voluntarily. In contrary, it has earlier been reported that students enrolling in digital health elective classes do not necessarily have advanced skills in the field, but can learn them as a measurable learning outcome .
The fact that digital health is an emerging educational subject and has not yet found its way into broad medical education also accounts for some limitations of this study. These include the single center study design and the mere focus on Germany. Furthermore, a relevant shortcoming of this study is the small cohort size, which does not meet a power calculation. A larger cohort would give more clarity on the topic. Well-structured larger studies should be conducted in the future. Another limitation arises from the elective character of the teaching format and the voluntary participation of the control group, suggesting that students already prefigured and interested in the field of digital health preferentially participated in the survey. The rather low increase in knowledge in the post-test might be explained by the fact that in the chosen winter semester 2020/2021 teaching was performed under “COVID-19” conditions, lessons were performed merely digitally and hence less interaction in presence was possible than in normal semesters.
A possible future problem in everyday medical care for patients will be a lack of knowledge of and competencies in “digital health”, resulting in inefficiency and maybe even susceptibility to errors . Digital health should thus have its regular place in medical education. Further research is necessary to provide specific learning objectives for digital health skills and competencies along with elaborate recommendations for their implementation in medical education.
Digital health is gaining in importance, and digital competencies are more and more required in everyday work. Although medical students seem to have accepted this fact and find that digital health is underrepresented during medical education, teaching of digital competencies does not yet have its regular place in medical education Elective teaching formats like the here presented one can lead to a measurable increase in students’ knowledge about the topic and should be taken as basis for strengthening overall curricular approaches to educate digitally competent future doctors.
The authors would like to deeply thank all participating students who their voluntary contribution in this study. Furthermore, we would like to express our gratitude to Mr. Louis Agha-Mir-Salim and Mrs. Mona Prendke for their kind support in the revision of the manuscript.
Conflict of interest
The authors declare that they have no conflict of interests.
The authors report no funding.
WHO guideline Recommendations on Digital Interventions for Health System Strengthening. Geneva: World Health Organization; (2019) .
Seemann RJ, Herbstreit S, Weber M, Erne F, Ansorg J, Dgou AGDD, et al. [Potential of digitalization in undergraduate and postgraduate medical education and training in orthopedics and trauma surgery]. Unfallchirurg. (2020) ; 123: (11): 836-42.
Keyserling K, Janetos E, Sprague C. Teaching Telehealth During a Pandemic and Beyond: an Intern’s Survival Guide for Virtual Medicine. J Gen Intern Med. (2021) ; 36: (10): 3219-23.
Meister S, Deiters W, Becker S. Digital health and digital biomarkers – enabling value chains on health data. Current Directions in Biomedical Engineering. (2016) ; 2: (1): 577-81.
Fragidis LL, Chatzoglou PD. Implementation of a nationwide electronic health record (EHR). Int J Health Care Qual Assur. (2018) ; 31: (2): 116-30.
Meskó B, Drobni Z, Bényei É, Gergely B, Györffy Z. Digital health is a cultural transformation of traditional healthcare. mHealth. (2017) ; 3: : 38.
García JF HS, Spatharou A, Beck J-P, Jenkins J. Transforming healthcare with AI: The impact on the workforce and organisations: EIT Health and McKinsey & Company; (2020) . https//eithealth.eu/wp-content/uploads/2020/03/EIT-Health-and-McKinsey_Transforming-Healthcare-with-AI.pdf.
Zhang D MS, Brynjolfsson E, Etchemendy J, Ganguli D, Grosz B, Lyons T, Manyika J, Niebles JC, Sellitto M. The AI Index 2021 Annual Report: Stanford University; (2021) . https://aiindex.stanford.edu/report/.
Haag M, Igel C, Fischer MR, German Medical Education Society CDT-AL, Teaching, Joint working group Technology-enhanced T, et al. Digital Teaching and Digital Medicine: A national initiative is needed. GMS J Med Educ. (2018) ; 35: (3): Doc43.
Foadi N, Koop C, Behrends M. Medizinische Ausbildung: Welche digitalen Kompetenzen braucht der Arzt? Dtsch Arztebl International. (2020) ; 117: (12): 596.
Gray K DA, Maeder A, Henderson KB, Chenery HJ. Advancing Ehealth Education for the Clinical Health Professions: Final Report 2014. Sydney, New South Wales, (2014) .
Mesko B, Gyorffy Z, Kollar J. Digital Literacy in the Medical Curriculum: A Course With Social Media Tools and Gamification. JMIR Med Educ. (2015) ; 1: (2): e6.
Pathipati AS, Azad TD, Jethwani K. Telemedical Education: Training Digital Natives in Telemedicine. J Med Internet Res. (2016) ; 18: (7): e193.
Alsahali S. Awareness, Views, Perceptions, and Beliefs of Pharmacy Interns Regarding Digital Health in Saudi Arabia: Cross-sectional Study. JMIR Med Educ. (2021) ; 7: (3): e31149.
Theron M, Redmond A, Borycki EM. Nursing Students’ Perceived Learning from a Digital Health Assignment as Part of the Nursing Care for the Childbearing Family Course. Stud Health Technol Inform. (2017) ; 234: : 328-35.
Torous J, Chan S, Luo J, Boland R, Hilty D. Clinical Informatics in Psychiatric Training: Preparing Today’s Trainees for the Already Present Future. Acad Psychiatry. (2018) ; 42: (5): 694-7.
Aungst TD, Patel R. Integrating Digital Health into the Curriculum-Considerations on the Current Landscape and Future Developments. Journal of Medical Education and Curricular Development. (2020) ; 7: : 2382120519901275.
German Federal Government. National Strategies for Digital Transformation. (2021) . https//germandigitaltechnologies.de.
Aulenkamp J, Mikuteit M, Loffler T, Schmidt J. Overview of digital health teaching courses in medical education in Germany in 2020. GMS J Med Educ. (2021) ; 38: (4): Doc80.
Poncette AS, Glauert DL, Mosch L, Braune K, Balzer F, Back DA. Undergraduate Medical Competencies in Digital Health and Curricular Module Development: Mixed Methods Study. J Med Internet Res. (2020) ; 22: (10): e22161.
Behrends M, Paulmann V, Koop C, Foadi N, Mikuteit M, Steffens S. Interdisciplinary Teaching of Digital Competencies for Undergraduate Medical Students – Experiences of a Teaching Project by Medical Informatics and Medicine. Stud Health Technol Inform. (2021) ; 281: : 891-5.
Tudor Car L, Kyaw BM, Nannan Panday RS, van der Kleij R, Chavannes N, Majeed A, et al. Digital Health Training Programs for Medical Students: Scoping Review. JMIR Med Educ. (2021) ; 7: (3): e28275.
Haladyna TM. Developing and Validating Multiple-Choice Test Items: Taylor & Francis, (2004) .
Al-Rukban MO. Guidelines for the construction of multiple choice questions tests. J Family Community Med. (2006) ; 13: (3): 125-33.
Machleid F, Kaczmarczyk R, Johann D, Balciunas J, Atienza-Carbonell B, von Maltzahn F, et al. Perceptions of Digital Health Education Among European Medical Students: Mixed Methods Survey. J Med Internet Res. (2020) ; 22: (8): e19827.
European Deans; meeting “Training Future-Proof Doctors for the Digital Society” Rotterdam: Erasmus MC; (2019) https//www.erasmusmc.nl/en/articles/deans-meeting-april-2019.
Fischer MR, Bauer D, Mohn K, Projektgruppe N. Finally finished! National Competence Based Catalogues of Learning Objectives for Undergraduate Medical Education (NKLM) and Dental Education (NKLZ) ready for trial. GMS Z Med Ausbild. (2015) ; 32: (3): Doc35.
Grzeska A, Ali S, Szmuda T, Sloniewski P. Objective Outcomes Evaluation of Innovative Digital Health Curricula. Comment on “Undergraduate Medical Competencies in Digital Health and Curricular Module Development: Mixed Methods Study”. J Med Internet Res. (2021) ; 23: (5): e26034.
Pontefract SK, Wilson K. Using electronic patient records: defining learning outcomes for undergraduate education. BMC Med Educ. (2019) ; 19: (1): 30.
Fernando J, Lindley J. Lessons learned from piloting mHealth informatics practice curriculum into a medical elective. J Am Med Inform Assoc. (2018) ; 25: (4): 380-4.
Seemann R, Munzberg M, Stange R, Russeler M, Egerth M, Bouillon B, et al. [Interpersonal competence in orthopedics and traumatology: Why technical and procedural skills alone are not sufficient]. Unfallchirurg. (2016) ; 119: (10): 881-4.