Evaluation of an e-learning module under different tutorial guidance

This multivariate, prospective, monocentric, single blind, observational study evaluates the experimental application of an interactive e-learning module under different tutorial guidance on the topic of EbD (Evidence based Dentistry).

The voluntary extracurricular module was offered to dentistry students (n=53) during the first clinical semesters. During a processing time of 4 weeks the students were supervised in small groups with an e-tutor (electronic tutor) via e-mail. The tutor supervised n=26 in a facilitative (f) and n=27 non-facilitative (nf) mode. In a final evaluation n=47 students participated, subdivided into n=24 (f) and n=23 (nf). The questionnaire used in the study comprised 10 items assigned to the evaluation of e-learning settings and tutoring effectiveness.

The concept of the tutor guided e-learning module was evaluated marginally better by the nf-group (Scale 1-10: 6.04±2.1) than by the f- group (Scale 1-10: 5.74±2.49). The f-group rated the tutor marginally better than the nfgroup. Differences in tutorial behavior were perceived as the f-tutor being described more determined and intervenient than the nf-tutor. All participants wished to supplement e-learning with face-to-face instruction (Likert Scale 1-5: 3.55±1.23). In the free text answers, many students pointed out of having problems with the voluntary course because of setting priority on courses requiring attendance duty and final exams.

The students rated the module with online tutorial guidance positively. We conclude that the tutorial support should be oriented to the needs of the participants: It should possibly change from initial facilitative supervision to non-facilitative mode. The integration of EbD into dental education will only succeed if offered as a curricular, apparent subject with interdisciplinary integration and specially trained teachers. One can also conclude from this that in the future a blended learning format with lectures and e-tutorial support will meet the needs of the students.

Introduction

The fully trained dentist as well as the student of dentistry is faced with an unmanageable flood of information [1,2]. EbM (Evidence based Medicine) was defined by Sackett et al., [3] as integration of the best current external evidence with individual clinical expertise and patients’ choice. However, the special clinical and material aspects of dentistry should be dealt within the framework of EbD (Evidence based Dentistry).

Knowledge of EbM is referred to as a "survival qualification" that enables clinical decisions to be made, scientific work to be done and critical assessment of one's own work and that of others [2]. It is a tool of lifelong learning [4]. If one looks at the training or continuing education provided in EMCDDA (European Monitoring Centre for Drugs and Drug Addiction), it is quickly sobering: Although training courses on EBM are offered internationally [5,6] and also in Germany [7,8], these are mainly attended by human physicians. EbM is also a seemingly integral part of the study of human medicine, but not of dentistry. Thus, the German Network for Evidence Based Dentistry calls for explicit curricular teaching in the methods of EbD in dentistry [9,10]. This establishment is supported by the newest regulation of the Dental Licensing Regulation [11], as well by NKLZ (National Competence-based Learning Objectives Catalogue for Dentistry) [12].

Various settings are available for teaching EbM skills: classic frontal teaching [6,13], PBL-setting (ProblemBased-Learning) [14], e-learning with tutorial support [5,15-18] or Blended Learning [19]. According to the current study situation, an e-learning-setting with tutorial support is suitable for teaching EbD and seems not be inferior to a face-to-face teaching format [17,20]. EbM-led action comprises steps such as "asking an answerable question" or "evaluating external evidence". It is noticeable that there are parallels here to the steps of PBL-teaching, where the steps "identification of the problem" and "re-evaluation" are also known [21]. PBL seminars are mostly accompanied by a tutor whose behavior can be classified as either facilitative (f) or nonfacilitative (nf) [14]. Which tutor behavior in an online format is suitable for imparting knowledge is unclear. Since 2011, the Dental University of Frankfurt has been offering the e-learning module "Toothache Clinic Focus" in order to meet the requirements of a scientifically research-oriented education. This module is freely available on the Internet and integrates 32 EbD orders.

EbD teaching plays a significant role in the education of students in dentistry. Every approved dentist should integrate the steps of EbD into his work flow, weigh different diagnostic and therapeutic options against each other and critically question his own clinical decisions. Therefore, the objective of this study was to find out to what extent the assessment of an EbD-e-learning module by dental students in the first clinical semester (= novices) depends on a different tutorial support.

In this study, the students were accompanied by an online tutor (e-tutor) for the first time. The following research questions seem important for the transfer of EbD knowledge: 1) How do the students rate the e-learning module with e-tutorial support? 2) Do students who were supervised either facilitative or non-facilitative by the tutor evaluate the e-learning module and the e-tutor effectiveness differently?

Materials and Methods

The e-learning module

The e-learning module "Toothache Clinic Focus" focuses on interdisciplinary diagnostics for patients with socalled focal diseases. In cooperation with experts from the fields of internal medicine, angiology and dermatology, three "focus patients" were designed as guidelines for knowledge transfer. On the basis of these cases, the student as a virtual practitioner goes through 7 clinical steps: from the general and special anamnesis (steps 1+2), findings (step 3), diagnosis (step 4), therapy (step 5) and interdisciplinary cooperation (step 6) to the end of treatment (step 7). Various interactive elements are available to the student: Drop-down menus, drawing functions, magnification function "magnifying glass" for X-ray images, video clips, tips for unknown drugs or terms, drawing function, drag-and-drop and a self-learning control in MCQs with feedback.

An "EbD button" is also integrated into the 56-page offer. This enables students to access 32 EbD assignments, which can be divided into basic and user knowledge and relate to the patient case. At the beginning, the patient is introduced, the topic is defined and the learning goals for the patient case and for EbM knowledge are set. The EbM assignments and learning objectives are based on the NKLZ [12], the core curriculum basic module [10,22] and the requirements of the updated licensing regulations [11].

The module (“Toothache Clinic”), which was evaluated in this study, is freely accessible to all users. It can be downloaded from the Internet at the following URL: http://elearning.med.uni-frankfurt.de/spielwiese-fokus/

Study population and setting

In this study all students of the first clinical semester in dentistry at the University of Frankfurt were included. 53 students took part in the study, of which n=30 were female and n=23 male (Table 1). The average age was 24 years (min 21, max 31). The previously defined inclusion criteria of the population included the following parameters: 1. students of the first clinical semester, 2. students without previous EbD knowledge, 3. students will carry out the patient treatment in the future. The previously defined exclusion parameters were as follows: 1. students from other semesters or repeaters, 2. students with previously EbD knowledge 3. students already have patient-treatment experience.

The e-learning module was an extracurricular voluntary offer. A total of 12 computers were available for free use in the university rooms and the e-learning module can also be accessed at any time from any location via the Internet address. The duration of the intervention was 4 weeks (May until June 2019).

Preparatory measures by those learning

The e-learning module was presented in a classroom event at the beginning of the study (45 min). The user interface of the website, the additional functions, the self-learning controls, the integrated links and materials were discussed in detail. The students gave a pseudonym and a mail address for the communication with the tutor. On the basis of this information, the students were randomly assigned to two groups. 26 were assigned to facilitative (f) tutoring, n=27 to non-facilitative (nf) tutoring. A further subdivision into 6 small groups (5 groups with 9 participants, 1 group with 6 participants) was made. The e-learning tutor introduced himself personally and explained the electronic communication of the participants with the tutor and the group members among each other. The students were asked to report to the tutor at least once a week via a group speaker with feedback on the current status or possible difficulties.

Preparatory measures by those teaching

The tutor qualified as an EbD teacher by attending EbM courses of the German Network of Evidence-based Medicine. In addition, he completed the Tele Academy's Tele-Tutor-Training program, which covered the specific requirements of tele media learning and tutoring.

Facilitative (f) and non-facilitative (nf) tutoring

A group development can be divided into 4 phases [23]: Forming describes the first meeting, forming the group. Storming is characterized by conflicts and lack of unity. In norming, rules are established and roles defined, at the end the group can solve the task in the last phase of performing. On the basis of these phases, special recommendations on tutor behavior [14,21], in a PBL set have been formed. Depending on the group to which the students belonged, the e-tutor in our study oriented himself according to the characteristics listed in Table 2 [14]. Thus the weekly mails and the answer mails to questions from the students were answered either in f- or nf- mode.

Course flow and materials

The e-learning module was handed over to the students after the group membership was announced. A courseaccompanying script and a concept map were made available. The rest of the learning materials were made available directly on the website in line with the EbD order. This beginning was followed by a three-week training period with the module. One case per week was to be developed independently. The fourth week should be used for repetition. During the four weeks, the tutor contacted the students according to a fixed predetermined scheme. Both groups were contacted equally often and received the same learning materials, but the mail was adapted in content and expression to the respective f - or nf- mode.

The questionnaire

Two formats were used to present the evaluation form used: The questionnaire "BEMSEL-IHS" (Conditions Motivated Self-directed Learning Instrument for Assessment at University) [24], focuses on the process of selfdirected learning. An evaluation instrument of the reform study course Medicine at the University Medicine Berlin [25], focused on the evaluation of the tutorial according to content, scope, organization and the feeling of under- or overstraining during the tutorials. Both questionnaires were combined and adapted to different learning settings. Ten questions on the evaluation of e-learning settings and tutoring effectiveness were extracted. The answer was a 5 step Likert scale (1=totally disagree, 2=disagree, 3=undecided, 4=agree, 5=totally agree), 4 questions could be evaluated on a numerical scale. Free text comments were also possible. Pseudonym and group number had to be specified in a recognition code in order to trace the affiliation to the facilitative or non-facilitative group.

Results

Response rate

After returning the evaluation forms, a population of n=47 students remained (Table 2). The response rate was 88.7%. 22 students used the free text field for suggestions for improvement and criticism.

Non-group specific results

The results are shown in Table 3. The students stated that the e-learning module represented an additional burden parallel to the other courses (3.85±0.99). If the tutorial support was helpful remained unclear (3.17±0.94). The majority of the students worked at their own pace independently of the learning objectives, wanted to supplement e-learning with face-to-face instruction (3.55±1.23) and wished tutor's instructions for the group meetings. Within the 4 weeks the tutor received 11 mails from 7 different students. 14 out of 16 questions were asked by the non-facilitative group. 10 of these 14 questions concerned the content of the module.

Group specific results

The results are shown in Table 4. F-supervised students rated the tutor marginally better (7.04 ± 2.38) than the nf -group (6.57±2.09). Tutor motivation was rated the same by both groups (2.87±1.22/1.01). The nf- group was of the opinion that the tutor let the group run too much (5.24±1.64) and intervened too little (4.56±1.15). In comparison, the f -group evaluated the tutor being determined (3.57±1.50) and too intervenient (3.81±1.17). Overall, the concept of the tutorial e-learning module was evaluated marginally better by the nf-group (6.04±2.1) than by the f -group (5.74±2.49). On average (5.89±2.28) it achieved the grade "sufficient".

Free text information for evaluation

The 22 free text answers aim at 5 topics that are repeated in the comments: 1) In addition to the e-learning module, the students wish to have a personal exchange with the tutor. This should take place once a week to discuss difficulties, exercises and learning material. The content can, but does not have to be specified. 2) The elearning offer with the advantage of flexible time management and repetition of contents as well as EbD as a key qualification in the dental professional life were evaluated positively. 3) The learning materials provided by links and pdfs on the website seem to be partly confusing. A hand out at the beginning named as "one holy EbM script" were desired. 4) As this is an optional offer and is not relevant for passing the course, many students were not able to deal with the module as desired due to 5) lack of time: "My priority was to pass courses that were relevant for progress. I feel like I have missed a lot of important things”.

Answering the main research question

After completing a four-week e-learning module with EbD content and e-tutoring, 47 students rated the concept as sufficient (5.89±2.28). Tutor effectiveness was evaluated differently by the two groups, so the f -tutor was perceived as "more decisive and more influential" than the nf-tutor. Most of the students wished to have face-toface instruction by the tutor in addition to the e-learning module.

Discussion

Teaching EbD knowledge via e-learning module

Several studies have shown that the transfer of EbM knowledge in an e-learning format is positively evaluated [14,26,27]. The e-learning format is also by no means inferior to traditional classroom teaching, but leads to at least as much knowledge growth [5,28-30]. This increase in knowledge also applies to EbM e-learning modules in different countries, languages and settings [18,29]. Groups who are offered an EbM online course in a blended learning format in addition to the face-to-face course show more self-confidence in dealing with statistical variables, greater effectiveness in literature research and easier transfer from theory to practice than control groups without an online module [19,31]. The international study by Kunz [6], examined which factors influence the increase in knowledge in EbM mediation: small group size and close tutor-student ratio, separate statistics event, small number of topics and the prerequisite that participants actively participate in the learning process have a positive effect on the increase in knowledge. Each form of teaching EbM improves knowledge and competence, and a varied and flexible lesson structure is important [32]. The module should provide clear learning objectives tailored to the participants and provide individual feedback [33]. In our study we tried to implement these requirements as follows: Classification into small groups, individual EbD tasks addressed exclusively the calculation of statistical parameters such as NNT (number needed to treat) and provided formula collection and exercises, EbD tasks and learning objectives presented clear "learning nuggets", the "activation" of participation through regular e-mail exchange with the e-tutor and knowledge queries in MCQ- format and the varied design of the user interface of the e-learning module through the interactive elements. Individualized feedback to the students on learning progress was provided indirectly through the evaluation of the intermediate knowledge checks and through feedback from the tutor on the weekly feedback.

E-learning tutor

It is not enough to simply provide learners with a learning program. Taking into account the Laurillard’s framework [34], timely knowledge transfer requires a feedback loop.

Students wish to engage in interactivity [15], which means to enter into a dialogue with a lecturer or tutor in order to receive individual feedback on their performance and knowledge progress. The advantage of such a tutorial-supported online measure is also shown in a higher increase in knowledge of a group compared to groups with no tutorial guidance [16]. The tutor must find the balance between the targeted provision of information and the promotion of self-study [21].

Out of 53 students, only 7 have contacted the tutor. Most of the questions were asked by the non-facilitative group. Since voluntary extracurricular e-learning in itself already demands a high degree of self-directed learning [35], non-facilitative tutor behaviour seems to increase the initial uncertainty [14].

The evaluation of the free texts shows that the lack of contact with the tutor was due to lack of time and personal contact. Hillenburg [36], points out that asynchronous communication by e-mail can never replace the one-to-one relationship between teacher and student, face-to-face the motivation of a student is probably better and elearning promotes contact with the faculty and should not make it unnecessary.

The fact that in this study the majority of students would like to have attendance courses in addition to tutorial elearning shows the importance of personal motivation and support. This can be due to the complexity of the subject EbD with included mathematical calculations, on the other hand to the more personally flexible and faster response to comprehension difficulties [15,37].

Evaluation of the e-learning module and the tutor

A working group around Gerhardt-Szep [14], investigated in a study the influence of different tutorial care in a PBL setting concerning endodontics on the dimension of knowledge acquisition and group function. The tutors had the same behaviour characteristics listed in Table 2. F- tutors were rated significantly better in terms of support and effectiveness. No difference was found in the evaluation of tutors' overall performance. In contrast, the students in this study rated the f- group (7.04±2.38) slightly better than members of the nf- group (6.57±2.09). The f- tutor was rated better in the dimension motivation than the nf-tutor (4.01±0.66 versus 3.85±0.46). In contrast, our study showed lower values (2.87±1.22) and no difference between the two groups. It should be pointed out that comparability of this study with ours is limited: type of intervention (e-learning concerning EbD contents versus PBL concerning endodontics) and tutorial support (e-tutor vs faceto-face-tutoring) differ. In a focus group discussion [14] students stated that nf- leadership is difficult for PBL beginners because it requires a high degree of autonomy and preparation effort. This also corresponds to the comments of our evaluated free text answers. For example Gerhardt Szep [14] recommends also changing tutorial support during PBL training from facilitative to non-facilitative. In 2017, the present e-learning module with included EbD content, but without tutorial support, was evaluated by 114 students [26]: For the question "Overall, I would give the following mark from 1 to 6 (where 1=very good, 6=insufficient), 114 students awarded a grade of 2.26±0.64. In a replication study [27], a grade of 2.40±0.66 was awarded out of 65 students. On the numerical scale of 1-10 (6=sufficient, 10=excellent) used by us, these grades correspond to a value of 6.24 ± 1.00 in the primary study and 6.00±1.10 in the replication study. These values are very close to ours (5.89±2.28), but our question included tutorial support and was aimed at a smaller study population. The different assessment of the tutor by the students confirms that students have perceived the facilitative or nonfacilitative support style of the tutor. The facilitative tutor was perceived as "more decisive and more influential" than the non-facilitative tutor, exactly as shown in the characterization in Table 2. In general, the f-tutor was rated marginally better than the nf- tutor. This result corresponds to the results presented by Gerhardt-Szep [14].

Voluntary module

One problem concerning the extracurricular module is that students voluntarily work on it in their free time in parallel with other exam relevant courses. The EbD course is not relevant for passing the semester. On the one hand, this is due to the fact that the demand for a scientific education and integration of EbM is not new, but EbD is still not an integral part of the German study of dentistry [38]. Davis points out that students in particular allow themselves to be determined externally by curricula and examinations and only acquire knowledge voluntarily if this is relevant to the examination [20].

Limitations and problems

The scope of our investigation of whether students who were supervised by the e-tutor either facilitative or non- facilitative is completely new and has so far only been investigated in face-to-face PBL intervention. This makes it difficult to compare the results in terms of tutoring effectiveness or the general concept. The questions asked are also a subjective personal assessment of the student. A formative knowledge check, which measures the increase in knowledge between the two differently tutored groups, is intended to provide objective data on the mode in which tutors should behave in the future. The questionnaire used is partly inconsistent in the response scales (Likert scale 1-6, numerical choice 1-7 and 1-10), and a number of 10 questions and 47 subjects can also be classified as low. When using a Likert scale, there is always the danger of aquiescence [39]. This could be avoided by changing the orientation of the answer scale or incorporating negative statements. The burdens put on the students and the tutor and their possible solutions are listed in Table 5.

Conclusion

After completing the 4-week e-learning course with e-tutorial support, the students awarded the overall grade sufficient (5.89±2.28). The nf- group rated the overall concept marginally better. The f- group rated the tutor as "more decisive and more influential" and in the overall grade more positive than the nf- tutor. Since the majority of students would also like face-to-face exchange with the tutor with organization and guidelines for the group meetings, the used format seems to be in need of improvement: In future, a blended learning format could be made available with attendance lectures included, small groups accompanied by tutorials and the provision of uniform learning material. The e-learning module and e-mail support could remain supplementary. Since elearning requires a high degree of self-control and there was greater uncertainty at the beginning, especially in the nf- group, we agree with the recommendation of the study by Gerhardt-Szep [14]: At the beginning of the group phases Forming and Storming the tutor should actively intervene, motivate the learners and give corrective feedback. If a group enters the norming or performing phase, the tutor can withdraw and act non-facilitative, i.e. participative and delegating. Special instruments for the evaluation of EbM knowledge such as the Fresno Test [40], or the Berlin Questionnaire [41], are already available and could be transferred to dental scenarios. This would make it possible to determine the influence of different tutorials on the objective increase in knowledge of an EbD e-learning measure. If under pressure of time, students only learn what they are asked to learn, digital knowledge transfer must also be made verifiable and certifiable in the future. A personalized result should be provided for motivation and follow-up in case of possible knowledge deficits [33]. Only an interdisciplinary integration of EbD as well as the introduction of EbD as an apparent curricular subject can do justice to the claim of a scientifically founded dentistry with mature practicing physicians. Education programs should be made known, teachers should be specially trained and the establishment of journal clubs should be promoted. Studentcentered e-learning will continue to be used to distribute EbD knowledge, with approaches such as mobile teaching via App [42], preparing course material in video format [43] and virtual or augmented reality [44].

1. Ballini A, Capodiferro S, Toia M, Cantore S, Favia G, et al. (2007) Evidence-based dentistry: what's new? Int J Med Sci 4: 174-178. Link: https://tinyurl.com/y3djevj2
2. Chiappelli F, Prolo P, Neagos N, Lee A, Bedair D, et al. (2004) Tools and Methods for Evidence-Based Research in Dental Practice: Preparing the Future. J Evid Base Dent Pract 4: 16-23. Link: https://tinyurl.com/yy2nnpn4
3. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS (1996) Evidence based medicine: what it is and what it isn't. BMJ 312: 71–72. Link: https://tinyurl.com/y2ycrcgv
4. Straus S, Glasziou P, Richardson WS, Haynes RRB (2018) Evidence-Based Medicine: How to Practice and Teach EBM. 5th Edition, Elsevier. Link: https://tinyurl.com/y464hbcf
5. Bergold M, Strametz R, Weinbrenner S, Khan KS, Zamora J, et al. (2013) Evidence-based Medicine online for young doctis - a randomised conrtolled trial. Z Evid Fortbild Qual Gesundhwes 107: 36-43. Link: https://tinyurl.com/y62ey9w5
6. Kunz R, Wegscheider K, Fritsche L, Schünemann HJ, Moyer V, et al. (2010) Determinants of knowledge gain in evidence-based medicine short courses: an international assessment. Open Med 4: e3-e10. Link: https://tinyurl.com/y24j8sua
7. Dörr J, Valipour A, Strametz R, Meyer G, Lühmann D, et al. (2013) Survey on evidence-based medicine teaching activities in the German-speaking area: a target-group specific analysis. Z Evid Fortbild Qual Gesundhwes 108: 166-174. Link: https://tinyurl.com/y3d85use
8. Weberschock T, Dörr J, Valipour A, Strametz R, Meyer G, et al. (2013) Evidence-based medicine teaching activities in the German-speaking                area:        a              survey.    Z Evid     Fortbild   Qual        Gesundhwes         107: 5-12. Link: https://tinyurl.com/yyryps6d
9. Matthias K, Nolting A, Bertelsmann H, Weber F, Pietsch B (2013) Training courses in Evidence-based Medicine at the Federal Joint Committee. Z Evid Fortbild Qual Gesundhwes 107: 62-69. Link: https://tinyurl.com/y28dnbnl
10. Siebolds M, Albrecht M, Kahl C, Langer G, Lühmann D, et al. (2013) Mission paper for the further development of the curriculum "Evidencebased Medicine" by the German Medical Association and the German Network Evidence-based Medicine. Z Evid Fortbild Qual Gesundhwes 107: 70-73. Link: https://tinyurl.com/y4w5o7o5
11. Federal Ministry of Health (2019) New regulation of dental education. Link: https://tinyurl.com/y6trzzu5
12. Medical Faculty Day of Germany (2015) National Competency-Based Learning Objective Catalog Dentistry. Link: https://tinyurl.com/yx9m29tu
13. Ubbink DT, Legemate DA, Koelemay MJ (2016) The Merits of a Two-Day Evidence-Based Medicine Course for      Surgical Residents. World J               Surg 40: 1809-1814. Link: https://tinyurl.com/y4hmre89
14. Gerhardt-Szep S, Kunkel F, Moeltner A, Hansen M, Böckers A, et al. (2016) Evaluating differently tutored groups in problem-based learning in a German dental curriculum: a mixed methods study. BMC Med Educ 16: 14. Link: https://tinyurl.com/yxryturj
15. Wong G, Greenhalgh T, Pawson R (2010) Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ 10: 12. Link: https://tinyurl.com/y5mb3p7s
16. García-Iglesias MJ, Pérez-Martínez C, Gutiérrez-Martín CB, Díez-Laiz R, Sahagún-Prieto AM (2018) Mixed-method tutoring support improves learning outcomes of veterinary students in basic subjects. BMC Vet Res 14: 35. Link: https://tinyurl.com/y3vs85qw
17. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, et al. (2008) Internet-based learning in the health professions: a meta-analysis. JAMA 300: 1181-1196. Link: https://tinyurl.com/y29b7x26
18. Sabouni A, Bdaiwi Y, Janoudi SL, Namous LO, Turk T, et al. (2017) Multiple strategy peer-taught evidence-based medicine course in a poor resource setting. BMC Med Educ 17: 82. Link: https://tinyurl.com/y6rqrwkr
19. Ilic D, Nordin RB, Glasziou P, Tilson JK, Villanueva E (2015) A randomised controlled trial of a blended learning education intervention for teaching evidence-based medicine. BMC Med Educ 15: 39. Link: https://tinyurl.com/y5lfse7b
20. Davis J, Chryssafidou E, Zamora J, Davies D, Khan K, et al. (2007) Computer-based teaching is as good as face to face lecture-based teaching of evidence based medicine: a randomised controlled trial. BMC Med Educ 7: 23. Link: https://tinyurl.com/yy9lhwvm
21. Walsh A (2005) The Tutor in Problem Based Learning: A Novice’s Guide. Link: https://tinyurl.com/y65zw3fm
22. EBM Network (2017) Core Curriculum Base Module - Evidence-based Decision Making. Link: https://tinyurl.com/y6cld4nm
23. Tuckman BW (1965) Development sequence in small groups. Psychol Bull 63: 384-399. Link: https://tinyurl.com/y65mgqwq
24. Aeppli J (2005) Self-directed learning by students in an Blended Learning Arrangement: Learning style types, learning success and usage of web-based learning units. Posted at the Zurich Open Repository and Archive, University of Zurich. Faculty of Arts. Link: https://tinyurl.com/y6p8s2yv
25. University Berlin Charite (2019) Learning Center Charité: Tutorials Evaluation. Link: https://tinyurl.com/y42abheb
26. Gerhardt-Szép S, Dreher S, Rüttermann S, Weberschock T (2017). Conception and implementation of a novel E-learning module with EbM learning contents in operative dentistry. Z Evid Fortbild Qual Gesundhwes 128: 72-78. Link: https://tinyurl.com/y25wo37s
27. Dreher S, Weberschock T, Giraki M, Uhse A, Parvini P, et al. (2017) Dental training in evidence-based dentistry: A replication study. J Dent Probl Solut 2014: 66-71. Link: https://tinyurl.com/y3m2zgsc
28. Weinbrenner S, Meyerrose B, Vega-Perez A, Kulier R, Coppus SF, et al. (2009) Ebm – Integrating a Europe-wide Harmonised Training and Continuing Medical Education in Evidence-based Medicine (EbM) with Patient Care. Z Evid Fortbild Qual Gesundhwes 103: 35-39. Link: https://tinyurl.com/y4tmwkga
29. Kulier R, Hadley J, Weinbrenner S, Meyerrose B, Decsi T, et al. (2008) Harmonising evidence-based medicine teaching: a study of the outcomes of e-learning in five European countries. BMC Med Educ 8: 27. Link: https://tinyurl.com/y35fcyej
30. Hadley J, Kulier R, Zamora J, Coppus SF, Weinbrenner S, et al. (2010) Effectiveness of an e-learning course in evidence-based medicine for foundation (internship) training. J R Soc Med 103: 288-294. Link: https://tinyurl.com/yy6zdubm
31. Schiling K, Wiecha J, Polineni D, Khalol S (2006) An Interactive Web-based Curriculum on Evidence based Medicine: Design and Effectiveness. Fam Med 38: 126-132. Link: https://tinyurl.com/yxa9tbnj
32. Ilic D, Maloney S (2014) Methods of teaching medical trainees evidence-based medicine: a systematic review. Med Educ 48: 124-135. Link: https://tinyurl.com/yydha82z
33. Kavadella A, Kossioni AE, Tsiklakis K, Cowpe J, Bullock A, et al. (2013) Recommendations for the development of e-modules for the continuing professional development of European dentists. Eur J Dent Educ 1: 45-54. Link: https://tinyurl.com/y649w4mo
34. Laurillard D (2002) Rethinking Teaching for the Knowledge Society. EDUCAUSE Review 37: 16–25. Link: https://tinyurl.com/y4o75mjk
35. Garrison R (2003) Cognitive Presence for effective asynchronous online learning: The role of reflective inquiry, self-direction and metacognition. University of Calgary. 2003. Link: https://tinyurl.com/y4h89bso
36. Hillenburg KL, Cederberg RA, Gray SA, Hurst CL, Johnson GK, et al. (2006) E-learning and the future of dental education: opinions of administrators and information technology specialists. Eur J Dent Educ 10: 16977. Link: https://tinyurl.com/yytxc7z2
37. Cutler CW, Parise M, Seminario AL, Mendez MJ, Piskorowski W, et al. (2016) Should Attendance Be Required in Lecture Classrooms in Dental Education? Two Viewpoints: Viewpoint 1: Attendance in the Lecture Classroom Should Be Required and Viewpoint 2: Attendance Should Not Be Required in the Lecture Classroom. J Dent Educ 80: 1474-1478. Link: https://tinyurl.com/y5q48rs3
38. German Network EBM (2016) Statement of the EBM network: Evidence-based medicine must be more than a voluntary offer in the new regulation of dental education (ZÄPrO). Link: https://tinyurl.com/yx9jxhvp
39. Welkenhuysen-Gybels J, Billiet J, Cambré B (2003) Adjustment for Acquiescence in the Assessment of the Construct Equivalence of Likert-Type Score Items. J Cross Cult Psychol 34: 702-722. Link: https://tinyurl.com/y22kp7rz
40. Ramos KD, Schafer S, Tracz SM (2203) Validation of the Fresno test of competence in evidence based medicine. BMJ 326: 319-321. Link: https://tinyurl.com/y2qs2a2v
41. Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R (2002) Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ 325: 1338-1341. Link: https://tinyurl.com/yxrdt6rv
42. Khatoon B, Hill KB, Walmsley AD (2013) Can we learn, teach and practise dentistry anywhere, anytime? Br            Dent        J 215: 345-347. Link: https://tinyurl.com/yym2c2n7
43. Mukhopadhyay S, Kruger E, Tennant M (2014) YouTube: a new way of supplementing traditional methods in dental education. J Dent Educ 78: 1568-1571. Link: https://tinyurl.com/yymr42b2
44. Dutã M, Amariei CI, Bogdan CM, Popovici DM, Ionescu N, et al. (2011) An Overview of Virtual and Augmented Reality in Dental Education. OHDM 10: 42-49. Link: https://tinyurl.com/y3wyucz5