Designing formative feedback in collaborative online international learning
Abstract
BACKGROUND:
The adequate delivery of formative feedback in higher education is vigorously discussed to support the development of students’ learning activities. Most of the literature refers to individual feedback to students in the context of web-based training and teaching in higher education. However, concrete design recommendations are scarce, especially regarding collaborative online international learning modules.
OBJECTIVE:
This paper aims to identify and systematize the need for formative feedback from students in Virtual Exchange modules and concludes with implications for the design of formative feedback activities.
METHODS:
The research employs a two-step sequential explorative mixed methods and longitudinal approach. A quantitative pre-test is followed by a qualitative self-reflective journal survey with two data collection dates using a thematic co-occurrence analysis.
RESULTS:
Based on 11 abductively coded themes, findings include the effects of agents on students and barriers and prerequisites for implementing formative Feedback in COIL modules.
CONCLUSIONS:
The effects of feedback can vary depending on the agent. While e-tutors need to build trust and give orientation, teachers support reflection, and peers affect motivation. Applicable and quick responses are fundamental for a positive perception of formative feedback. Peer feedback can potentially improve learning and reduce the teacher’s workload.
Mattis Altmann is a research associate and Ph.D. student at the Chair of Business Information Systems –Information Management Faculty of Economics at TU Dresden. He teaches in the scope of virtual and international collaboration in small groups, case study design, and formative feedback. Further, he is the project manager in an Erasmus+capacity-building project that aims to implement Virtual Mobility and Internationalization@home in the Western Balkan regions. His research areas include the design of case studies to foster virtual collaboration and formative feedback in virtual group work.
Maik Arnold, Dipl.-Kfm., Dr. rer. soc., FRSA, is a professor of Social Work Management at the Fachhochschule Dresden, Germany. He has conducted various projects in intercultural communication, game-based learning, virtual collaborative learning, and organization research. He holds a Ph.D. in Social Sciences from the Ruhr-Universität Bochum (Germany) and is a certified systemic coach and intercultural trainer. He serves as a board member of the International Association of Social Work Management (INAS e.V.) and speaker of the German Network of Scholarship of Teaching and Learning (SoTL).
1Introduction
Formative feedback is an essential part of education that can impact learning, achievement and motivation, depending on the correct implementation [1, 2]. Nevertheless, it is harshly criticized by students [3]. As previous research has shown, although feedback is one of the main influences, the type of feedback and how it is provided can vary in effectiveness [4]. To ensure proper implementation, the completion of a feedback loop is needed [5]. Especially formative feedback about progress, strengths and weaknesses, and the areas for improvement given during the learning process can help learners improve their understanding and performance. Regarding Feedback in Virtual Exchange (VE), Geister et al. [6] found that initial motivation is a moderating variable on the improvement caused by the online feedback system, which affects students’ satisfaction and performance.
In this article, we define feedback as information about a person’s performance or understanding provided by an agent (e.g., in most cases, teachers or peers) [4 p81] with the intention to change the learning behavior positively [2].
The current feedback literature in higher education focuses mainly on individual feedback in online modules. However, the focus on group learning in collaborative learning platforms in project-based learning is scarce. This paper aims to refine Altmann’s [7] semantic network that attempts to map feedback in collaborative learning activities. Altmann’s [7] Semantic Network is the outcome of a previous iterative step of the Action Research Framework that follows a cyclical process where each cycle builds upon the outcome of the previous one [8, 9]. The data generated in earlier cycles is validated in another cycle, collecting further empirical data and supporting the refinement of the previous framework [10]. This semantic network [7] sheds light particularly on students’ perception of and interaction with formative feedback within Collaborative Online International Learning Modules. The results show that the relationship between the e-tutor and the students’ motivation has a significant influence on the positive or negative perception of the formative feedback and that this perception is also connected to the implementation or non-implementation of the feedback. Other influencing factors are the availability of the e-tutors and the positive formulation of the formative feedback but also its applicability and the connection to the grade.
A more robust validation of the relationships determined in the semantic network [7] is to be carried out by an application in another COIL module in order to test the results under other circumstances. Thus, the data of [7] were collected before and during the COVID-19 pandemic, in which students were confronted exclusively with virtual teaching formats. Furthermore, re-validation of qualitative research results is essential to ensure the consistency of the results and, thus, the scientific quality criteria such as rigor and validity [11]. It also improves the transferability of the results [12] and can make a theoretical contribution to improving qualitative research methods and confirming their robustness and rigor [13].
For this purpose, 13 hypotheses were generated from the previously developed semantic network, which will be tested and extended in this paper using a mixed-method approach and the following research question:
RQ1: What characteristics should formative feedback in COIL modules have for successful implementation, and how are these characteristics linked?
For this purpose, the underlying theoretical background of learning and teaching in higher education will be provided first before the module, considered a teaching laboratory, will be described. Thereafter, the sequential explorative mixed methods approach will be described, which forms the basis for the empirical analysis. A quantitative pre-test is used in the first sequence to gather students’ expectations and previous experiences in COIL modules. This is followed by two qualitative self-reflective journals at mid-term and end-term of the module. Subsequently, the abductively generated themes and codes will be analyzed and discussed based on Braun’s and Clarke’s [14] thematic analysis in order to verify or falsify the existing hypotheses and to gather further insights that will, finally, culminate in concluding remarks, the reflection of limitations and future research directions.
2Theoretical background
According to [8], this paper aims to refine the semantic network developed in [7] within the Action Research (AR) Framework.
The Action research framework offers the opportunity to close the gap between theoretical and practical perspectives on feedback processes, it shifts attention to the stance of the teacher as a researcher, allows for active engagement in the research process and addresses real-world problems in the educational domain [15–17]. The iterative and reflective procedures inherent in action research enable the translation of findings into theoretical frameworks, permitting the improvement and growth of theoretical comprehension grounded in practical experiences and collaborative involvement with stakeholders [17, 18].
For this purpose, the semantic network was used to form hypotheses that will be analyzed in this paper and supplemented by new findings. The research presented in this paper is informed by the Scholarship of Teaching and Learning (SoTL) through AR [19]. While SoTL is understood as a scientific inquiry of teaching in higher education, which looks for innovative, evidence-based, and evidence-creating ways to develop academic teaching, as well as to foster communities of practice [20], AR complements the research process with a collegial, evidence- and theory-based reflective-evaluative approach to analyze interventions, innovation and methodology of teaching and learning [21]. We understand teaching as a design-based process in line with Goodyear [22 p28]: “that spending more time on design will allow individual teachers and teaching teams to cope with intensifying pressures on the quality of their work, and to create better learning opportunities for their students.” This also involves constantly redefining roles and responsibilities in the teaching and learning process. Depending on students’ experience level, subject matter expertise, and needs, teachers can act as instructors, facilitators, assessors, mentors, curriculum developers, and researchers. According to Euler & Seufert’s [23] three-level framework of higher education, this study primarily focuses on individual learning scenarios and learning resources at the micro-level.
The module under analysis was developed within the Collaborative Online International Learning (COIL) framework, or VE. These terms are considered interchangeable according to [24], so this paper will limit itself to the definition of VE for reasons of simplicity. Using web-based technologies, VE offers innovative opportunities for structured online collaboration among participants from geographically dispersed locations. According to O’Dowd [25], VE creates a unique learning environment that can complement traditional classroom-based learning environments:
“Virtual Exchange is an umbrella term which refers to the numerous online learning initiatives and methodologies which engage learners in sustained online collaborative learning and interaction with partners from different cultural backgrounds as part of their study programs and under the guidance of teachers or trained facilitators” [25 p11]
Previous research findings have shown that, in addition to promoting digital skills [26], VE can contribute to enhancing motivation for foreign language acquisition [27], developing intercultural competencies [28], raising awareness of foreignness and difference in intercultural communication processes [29], and advancing critical thinking [30]. VE has also been employed in coping with challenging experiences during the Covid-19 pandemic [31].
Within case-based COIL modules, there is a high demand for supporting the learning process [32]. The shift from traditional forms of teaching and learning to a digital, networked, and constantly expanding approach is a current challenge in higher education [33]. Figure 1 presents the theoretical framework of formative feedback and provides the basis for this paper.
Fig. 1
Formative Feedback in COIL (authors’ own illustration).
As can be seen in the theoretical framework of formative feedback in Fig. 1, the feedback should be adapted to the respective learning situation to ensure the learning success of the students [34]. Implementing formative feedback depends on moderating variables that favor or hinder its implementation [2]. In particular, student reflection stimulated by feedback plays an essential role in the implementation but also in the permanent change of learning behavior, which can be influenced by the feedback [1, 35, 36].
3Module description
The study module, which served as a teaching laboratory for this paper, took place in the winter semester of 2022/2023 with participants from TU Dresden (Germany), HTW Dresden (Germany) and WUNU Ternopil (Ukraine). The 70 participating students are enrolled in the bachelor’s degree programs level Economics, Business Administration, Business Education and Business Informatics. The entry requirements for the students were, on the one hand, an active participation in a bachelor’s degree in business, economy or related discipline and a willingness to engage in self-directed learning in asynchronous and synchronous learning activities during the module [37]. All conversations took place predominantly in German since this language is a minor in the Ukrainian students’ study program and their economics-oriented major.
The module’s structure (see Fig. 2) is based on the paradigm of constructive alignment, according to Biggs [38]. In addition to the content-related learning objectives in entrepreneurship, electromobility and platform business models, students also acquire soft skills in intercultural competencies and collaborative work in heterogeneous teams. The learning activities within the module are designed within the scope of project-based learning in a collaborative online learning environment using complex scenarios and real-world problems within a fictive case study [37]. The assessment is conducted formatively in terms of individual and group performance and summatively in terms of the content outcomes of the project phases.
Fig. 2
Timeline and survey timepoints of the module (authors’ own illustration).
Within the module, feedback is provided by three different agents: for example lecturers, e-tutors and peers. In the first phase of the module, two obligatory feedback units by the e-tutors were scheduled to discuss process- and tool-related feedback (after week one and week two). An additional obligatory appointment with the lecturers was also integrated into the first phase to discuss task- and assessment-related feedback. Subsequently, formative feedback was available exclusively upon request from both e-tutors and instructors. Formalization of peer feedback was not yet intended but also not forbidden. The platform used for online collaboration was Microsoft Teamstrademark, which stands out for its all-in-one solution. With a single license, students can use the complete office suite of Wordtrademark, PowerPointtrademark, Exceltrademark and any other software linked with this cloud services, such as a calendar, timeline, and a video chat function, so that collaborative online activities can be a central focus in the module.
4Research design
4.1Sample
Prior to the start of the module, the participants (n = 67) enrolled in the course had to fill out a questionnaire, including some demographic survey questions and metadata, expectations, and prior (virtual) experiences (see Fig. 3). On average, students’ year of birth was 1999, so most participants were, on average, 23–24 years old, with the oldest student born in 1990 and the youngest in 2003. Gender is almost evenly distributed, with 47% women (n = 30) and 53% (n = 34) men; other gender identities were not mentioned. However, the answers to the question about their experience in previous virtual exchange modules were somehow unexpected, as 75% (n = 54) of students indicated that they had no prior experience with virtual exchange, and only 25% (n = 18) answered that they had some expertise (see Fig. 3). This ratio is particularly notable considering all the changes that happened in online teaching since the COVID-19 pandemic.
Fig. 3
Results from preliminary survey: Age and previous experiences (authors’ own illustration).
4.2Procedure
The underlying research design used in this paper is an exploratory mixed method and longitudinal approach, according to Creswell & Plano Clarke [39]. Mixed-methods research offers the advantage of exploring complex social phenomena in greater detail than traditional single-method approaches. It also allows the researcher to “construct, confirm and theorize at the same time” [40 p34], which helps to “widen the impact and scrutiny of the research” [21 p9]. The qualitative component of the research plays a dominant role in weighting, information density and influence on the research outcome. The research process can be found in Fig. 4 and is explained below.
Fig. 4
Sequential explorative research process (authors’ own illustration).
In the first step, data about students’ pre-knowledge, expectations and motivation was collected in a quantitative pre-test to which students responded in the form of a digital questionnaire before the start of the module. This provides quantitative data for the research process and a basic understanding of the sample under study regarding their prior experience and expectations for the module.
In the second step, with the help of qualitative self-reflective journals, we collected feedback-related learning experiences of students at two stages of the module after four weeks (n = 70), in the middle of the module, and after eight weeks (n = 68) at the end. These provide a deeper understanding of the perceptions, effects, and implementation of formative feedback and are intended to provide insights into the context and interaction of the individual factors [39]. The journals collected students’ feedback experiences regarding their relationship and interaction with the e-tutors and teachers, motivation to deal with positive or negative feedback, perception of the feedback, implementation of the feedback, and critical learning incidents. All journal entries have been collected with a questionnaire in MS Formstrademark. Finally, the pre-test was analyzed descriptively and according to Döring and Bortz [41].
4.3Analysis
The systematic qualitative analysis of students’ reflective journals aligns with the Scholarship of Teaching and Learning (SoTL) paradigm [20], allowing for an empirical-grounded and theory-driven reflection of the feedback processes during teaching in the module. The thematic analysis was the interpretative method chosen to analyze the journal entries, defined as “the process of identifying patterns or themes within qualitative data” [42 p3325]. Based on this assumption, researchers construct common or contradicting themes from the data to serve as a foundation for data interpretation [14] and read and reread all written accounts using categorization and category-building procedures. The interpretation of the different quotes follows the methodological principles based on hermeneutics of symbolic action theory and cultural psychology, including a comparative analysis [43].
For the data analysis, the authors first used the software Atlas.ti Webtrademark for synchronous collaborative coding of all data. Additional meetings during the coding process ensured that the codes created were intersubjectively conclusive and valid for both researchers, which involved dividing and merging codes, creating sub-codes and the reduction of the codebook. The interpretation of raw data followed an inductive and deductive approach adapted from Fereday and Muir-Cochrane [44], while deductive coding was the dominant procedure. Codes such as motivation, availability, positive/negative reception, relation to the e-tutor, peer feedback, feedback links to grades, implementation and application of received feedback were explored from previous research [7], refined and differentiated. Later, the codebook and quotes were transferred to the desktop version of Atlas.ti for further refinement of the themes and deeper analysis. In Atlas.ti Desktop, all codes were structured in a hierarchical order and relationships between two or more codes were explored with the co-occurrence analysis because many codes were embedded in one another. Code co-occurrences refer to the “complexities of ambivalent experiences using qualitative data” [45 p545] in two or more codes in a particular segment of data [45]. This can be useful for identifying patterns or relationships between different codes and can help researchers better understand the links between different themes in the data collected. Based on these co-occurrences, we discovered the hypotheses presented in section 1 of this paper as part of an abductive thinking process [46]. Abduction is a type of reasoning that involves making an educated guess or hypothesis based on incomplete or uncertain information to explain an observation or pattern in the data. The hypotheses can be understood as the best possible and intuitive explanations of the data, which were tested and later justified by data.
5Results
According to the mixed method approach employed in the present study, we present the first findings from the quantitative testing at the start of the module and, based on these, will provide an overview of the themes discovered during data analysis. Students were then asked why they chose the virtual exchange module (see Fig. 5). The most frequently cited reasons were related to the improvement of digital/technological literacy skills and the learning content of the module. Furthermore, the social form of learning in the group was another important reason for the selection. However, project-based teaching and the expectation to learn from virtual exchange experiences were less frequently mentioned.
Fig. 5
Results from preliminary survey: Selection of the module.
In our view, the data clearly show that, despite the nearly entire cohort of students, which we assume to have higher technology affinity (“digital natives”), prior experience with virtual exchange was reasonably low. The term “digital natives” was coined by the education consultant Marc Prensky, referring to the cohort of individuals who have grown up with digital technology, such as computers, the internet, mobile phones, and social media, which helps to understand that use and see those technologies as an integral part of their everyday lives [47]. Furthermore, it seems essential for this age group in particular (what could be compared to the Gen Z population to strive to improve their digital skills and deal with sustainability and entrepreneurship [48]).
Additionally, from data analysis of 100 randomly selected self-reflective journals emerged 11 themes from 2089 quotes. The coding of the material mainly considered the content, metadata, and feedback effects. The effect-related topics will be presented in the first, followed by the content-related topics in the second and the metadata in the last section. Finally, the two survey time points (middle and end of the module) were compared. Quotations from the codebook (translated by authors) provide evidence and help to validate the themes, which can be identified as such: the number before the colon refers to the document, while the number after the colon marks the quotation in this document. All quotes from the first survey time point range from 2–71 and from the second time point from 72–139. The code co-occurrences and the document-code-occurrences were tested using cross-tabulation and the Sankey diagram.
5.1Feedback perception & realization
This theme captures students’ positive, negative, or neutral perceptions of formative feedback. Of the 423 quotes, 296 (69,97%) were coded positive, and 110 (26,00%) negative. In this context, Positive Perceptions are almost evenly distributed across both survey periods, while Negative Perception was higher in the second (74) than in the first (36). Comparing all Negative Perceptions, 38 codes have co-occurrences with the theme of Feedback needs. Of these 38 codes, the desire for more Task-related Feedback (11) and Obligatory Feedback (11) is particularly noteworthy. Furthermore, the theme of Feedback Effects with 22 codes is interesting, as 10 codes have a co-occurrence in negative perception and motivation. Finally, the theme Content forms 21 co-occurrences with Negative Perceptions, 13 of which are related to the subcode Task-related Feedback, as one student has put it:
121:15: “Task-related Feedback was only given to our group in the workshops. In my opinion, this is far too little. I would feel it would be better if the feedback continued longer to keep the group motivated.”
The quote suggests that the amount of task-related formative feedback provided to their group in the module’s second half was insufficient. They believe feedback is important for motivation and improving their performance, and they desire more obligatory feedback to continue throughout the module. Furthermore, the negative perception induced by the lack of task-related feedback negatively influenced the students’ motivation.
When it comes to feedback realization, 167 codes have been assigned to this theme, whereby 135 (80.83%) were coded with “realization yes”, 16 (9.58%) with “partly,” and 17 (10.18%) with “no”. Considering the differences between the survey periods, the data show that in 77 (57.03%) cases, feedback realizations took place in the first survey period and 58 (42.97%) in the second. From 135 feedbacks realized, 83 (61.48%) were also perceived as “positive”, 7 as “negative” (5.19%) and in 45 (33.33%) cases, students did not specify positive or negative perceptions. Of the 7 feedbacks that were perceived as negative but were –according to students’ self-evaluation –realized, there is also a code-co-occurrence with the subcode Link to grade in 5 different quotes. Notably, the 5 quotes mentioned above are always linked to the e-tutor or lecturer as the sender of the feedback, while the other two cases of negatively perceived –but realized –Feedback came from fellow students. For example, this student provided adequate reasoning concerning Negative Feedback:
35:11: “However, the e-tutor wanted us to communicate with each other via the post function. But no one told us that at the beginning. And then to write in the feedback that he can not evaluate anything in the communication because we do not write, although we constantly communicate with each other I found it very bad and demotivating, because I hope for a good evaluation. I found that then also rather uninsightful. We then addressed it and wanted to create a kind of report on the communication.”
This quote highlights that the student was somehow frustrated by the e-tutor’s response and could not evaluate their communication even though they were communicating.
After careful reinspection of the data, we also realized that with 29.41%, almost one-third of the feedback perceived as negative was realized for fear that it could have a negative impact on the final grading. However, in reconsidering the code co-occurrences that have been discussed before, it is also noticeable that all realized feedback that had been perceived negatively is linked to the grade and has been provided by the teacher or e-tutors.
5.2Feedback effects
Content, Effects & Needs-related codes can be found in Fig. 6 and will be discussed in the following chapters. Under the theme Effects of Feedback, we summarized the effects of formative feedback as perceived by the students. All effects related to content-related feedback have been analyzed under the content-related themes in the next session. For the Effect of Feedback, a total of 228 codes were collected under this theme, with the most frequent being in motivation (96), Reflection (61), Link to grade (35), and Orientation (35). Other effects are trust (18), Appreciation (17), not necessary (13), and acceptance obligatory (7), which were less frequently mentioned. Our code-co-occurrence analysis showed that motivation, 59 (61.45%) out of 96 cases, strongly correlated with the positive perception of feedback, while it is in only 9 cases related to a negative perception. Having said this, the students in this study perceived the feedback provided by their e-tutor very positively, as can be seen in the following quote of a student:
Fig. 6
Content-related and effect-and-needs-related codes.
107:12: “The perception of the feedback from me personally was always very positive. I personally found the feedback very appreciatively formulated, which was reinforced by the fact that it was expressed in detail and well-founded. One had the feeling that the e-tutor had really looked into our work and also, insofar as no negative points, highlighted the positive aspects of our work as well. This led to generally higher team motivation and is definitely part of a good feedback culture.”
More generally, based on the described code co-occurrences, it can be assumed that the positive perception of feedback is connected with a positive influence on the student’s motivation. Other strong code co-occurrences in this theme have been found between Positive Recognition and Reflection (36), orientation (18), Trust (17), and Appreciation (14). The most frequent code co-occurrences in this document group are related to Tools Feedback (7), E-Tutor (6) and Teacher (7).
5.3Feedback needs
Self-reflective journals can provide valuable insights into students’ needs for feedback, highlighting areas where feedback is most needed and guiding instructors to tailor feedback to meet these needs. The strongest of the 127 codes are Obligatory Feedback (32), Task-Related Feedback (27) and Individual Feedback (17). Among these codes, in 10 cases, needs were connected to Positive Perception, compared to 38 cases of Negative Perception. Moreover, it is striking that there are 14 code co-occurrences between the theme Feedback Needs and Feedback Not Requested, which has already been discussed in the section Recipient and Sender.
When comparing the feedback needs throughout the survey periods, it is noticeable that Individual Feedback was coded mainly at survey time point 1 (15) and hardly at survey time point 2 (2), while the need for more Obligatory Feedback was coded 3 times in the first and 29 times at the second survey time point. The need for more Task-related Feedback shows a similar concurrence, coded 3 times at the first survey time point and 24 times at the second. Furthermore, Assessment Feedback was mentioned as a need 2 times at the first survey time point and 10 times at the second. Finally, the equal distribution of Peer Feedback needs over both survey time points is also worth mentioning.
Based on the analysis, it can be concluded that students’ needs for feedback have changed over time. While there was a decrease in the need for individual feedback between the first and second survey time points, there was a notable increase in the need for more Obligatory, Task-related, and Assessment-related Feedback at the second survey time point. The need for Peer Feedback remained consistent across both survey time points. These findings suggest that feedback needs can vary throughout a module or program and that instructors should be aware of these changes to provide tailored feedback that meets the evolving needs of their students.
5.4Recipient and sender
The theme Recipient and Sender comprises all codes related to the feedback the different agents gave. Of the 226 codes under this theme, 104 are connected to the E-Tutor, 72 to teacher, and 51 to Peers. Furthermore, students also noted that feedback was not obtained (20) or did not receive a response from the agent (4). In this theme, 136 (60.17%) code co-occurrences were identified with Positive Perception and 40 (17.69%) with Negative Perception. Regarding the perception of the E-Tutors, 74 were coded as positive and 13 as negative, while among the lecturers, 47 as positive and 15 as negative occurred, and among the peers, 30 as positive and 7 as negative. Overall, the coding data suggests that the majority of feedback provided by E-tutors, lecturers, and peers was perceived positively by the students. Looking at the code co-occurrences between the different senders of feedback and the feedback effects, it became apparent that the most substantial effect of Peer Feedback is Motivation (18), while the most potent effect of E-Tutors is Trust (9) and Orientation (7). On the other hand, for lecturers, the strongest effect is reflection (9). In the following quote, the student discusses their motivation to engage with feedback provided by their group members, highlighting both the positive relationships:
102:7: “I have been motivated to deal with the feedback of my group members. There were several reasons for this: On the one hand, I got along well with all the group members, which is why I generally enjoyed interacting with them. On the other hand, the feedback was always very constructive and advanced my thinking process regarding my tasks so that I could work out the best possible results.”
This student was highly motivated to engage with peer feedback. Moreover, the positive relationship within the group and constructive feedback were key factors for their motivation. This finding points to the fact that both the quality of the feedback and the interaction with group members can play a crucial role in motivating students and engagement in the virtual learning environment.
Due to the relatively small number of peer feedback codes (51) compared to the other recipient- and sender-related codes in this theme and the related most significant percentual proportion with 18 codes, the effect of peer feedback on student motivation seems to be the most prominent.
Regarding the occurrence of relationship with the e-tutor and the implementation of feedback, the effect size is not particularly strong with 7 code co-occurrences. In contrast, out of 51 Peer Feedback codes, 30 (58.82%) are coded with Positive Perception and 17 (33.33%) with realization yes, forming the strongest effect in this comparison.
Furthermore, it is striking that the students who did not Request feedback (20) simultaneously expressed a Need for Feedback in 70% (14) of the cases. Within the 14 code co-occurrences, the sub-code More Task Feedback occurs 7 times and the sub-code More Obligatory Feedback 5 times. During the second part of the module, consultations were available upon request at any time. These results can be interpreted as a desire for more obligatory feedback by lecturers and e-tutors. Finally, it is worth mentioning that among the code co-occurrences of e-tutors and the theme Style and Tone, the codes Concrete and Positive Formulation stand out with 13 mentions each.
5.5Content
All content-related codes (183) were grouped and analyzed within this theme. The Task- and Tool-related sub-codes account for 62, followed by 42 codes for Process, 22 for Assessment, and 13 for Role Clarification. When looking at the survey periods, it is noticeable that tool feedback is mainly needed at the beginning of the module. For example, the document-co-occurrences reach 46 codes at survey time point 1 and only 16 at survey time point 2. Feedback on the role-related issues was unevenly distributed between the survey periods (at survey time point 1:11 and 2 at time point 2). In contrast, Assessment-related feedback was mainly coded in the module’s second half (11).
Furthermore, Task-related Feedback was coded 27 times at the first survey time point and 35 times at the second survey point. This is supported by the code More Task Feedback from the Feedback Needs topic, which was coded 3 times in the first survey period and 24 times in the second survey period. Overall, the consistent increase in Task-related Feedback, as evidenced by the coding data and the Feedback Needs topic, suggests that providing feedback on specific tasks is essential to promoting learning and growth in educational settings.
When examining how feedback is perceived, the data show that Task-related Feedback was perceived positively 24 times but also quite often negatively (19). The Realization of Task Feedback is well-grounded in 14 code co-occurrences. In contrast, Tool Feedback is most often perceived positively (27) and most often realized (24) within this comparison. One of the students in this study values content-related feedback as it provides them with valuable insights on how to improve their work and optimize their performance:
36:8: “I find the feedback I receive very important, because it always gives me food for thought to change things, to optimize them, or to keep good things. This can concern the simple working together as well as technical things. Since we as a group and I are not yet very familiar with MS Teams, tips such as marking contributions, the mention function, and a good folder and document structure are particularly valuable.”
What can be seen is that feedback covers not only technical aspects but also organizational recommendations on how to improve group collaboration. For example, the student found the tips on using MS Teams, such as marking single contributions and using the mention function, precious. This suggests that content-related feedback can support student performance development and foster effective teamwork.
As already outlined in the quantitative pre-test, only a quarter of the students have previous experience with VE modules and, thus, presumably also with the collaboration software used in the context of these learning activities. Due to the easy-to-learn user interface of MS Teams, the high demand for feedback in this area is limited to the first half of the module.
5.6Style and tone
Under this theme, the modalities of how to formulate feedback were summarized. The strongest of the 117 codes related to this theme were Positive Formulation (54), Constructive Formulation, Precise Formulation (27) and Applicability (26). Furthermore, 89 Positive and 19 Negative perception code co-occurrences were found in this theme. The strongest positive perceptions in terms of code co-occurrence were Positive Formulation (45) and Constructive Formulation (32) as well as Applicability (22). Negative Perceptions were scarce to be found here. Regarding feedback effects, Positive Formulation has the strongest code co-occurrence with motivation (14), followed by Constructive Formulation and Motivation (10). In the following quotes, students reflect on the feedback provided by their e-tutor, highlighting the constructive nature of the feedback and how feedback increases motivation to participate in the learning process:
47:8: “I found the feedback very constructive. However, I would have liked more quantitative feedback at one point or another. Two or more positive things were mentioned, and the negative thing was that our communication was somewhat below average. I think it’s good that positive things were mentioned first. That was actually very motivating because you first received praise for what you had already done. This was certainly also the intention of the e-tutor.”
93:9: “Constructive Feedback was given by both people, which motivated me personally to deal with the feedback. It was always clear what you needed to work on, so there was no reason for me to react to the feedback in a demotivated way.”
These two quotes highlight the importance of constructive feedback in promoting student motivation. The students appreciate clear and specific feedback in identifying areas for improvement while acknowledging positive aspects of their work. Additionally, receiving feedback in a balanced manner, with positive aspects mentioned first, was particularly motivating. These findings suggest that mainly positive and concrete constructive feedback can play a crucial role in fostering student motivation and, thus, can positively affect the implementation of the feedback discussed in the section Effects of Feedback.
5.7Timing and frequency
This theme deals with the agents’ temporal availability for feedback and the frequency of how often feedback was given. The codes Availability (79), Frequency (39) and Scaffolding (9) occurred most often compared to the total of 108 codes under this theme. In the code co-occurrence analysis, Availability and Positive Perception were both coded 19 times, and Availability and Relation to E-Tutor were both coded 13 times. It is particularly striking that all code co-occurrences of Availability and Relation to E-Tutor were also related to the code Positive Perception. The positive perception of the timing and frequency of feedback provided by their E-Tutor is emphasized by one of the students:
39:6: “I feel the relationship with our e-tutor to our lecturers is very good. I like the fact that you can get answers promptly at any time. I also like the short communication path via teams. You don’t have to write an e-mail or anything like that first but can make the most of the advantages of the platform. I thought it was good that there was an online meeting with our tutor right at the beginning, so we could clarify all questions (and there were a lot of them at the beginning).”
As can be seen, the student appreciates the prompt responses and communication provided online and found the initial online meeting with the e-tutor helpful in clarifying questions and establishing a good rapport. These findings suggest that timely and frequent feedback linked to appropriate communication channels can be highly beneficial in promoting effective virtual learning.
Moreover, timely availability significantly impacts the relationship with the e-tutor and, thus, the positive perception of the feedback. Contrarily, it can be assumed that the e-tutor’s non-availability would negatively influence the perception. However, this has been only the case in four code co-occurrences. Nevertheless, this phenomenon requires further examination. In the first part of the module, students mentioned timing issues with their teachers and felt that the E-Tutor was not present. Compared to this, in the second part of the module, the negative perception was related to the fact that they missed obligatory feedback.
5.8Delivery mode
The mode of delivery can play a crucial role in the effectiveness of feedback communication. Under this theme, a total of 42 codes were set, with 22 set to Video Chat, 15 to channel (Written), 5 to audio call (Oral), and 3 to E-Mail (Written). Regarding code co-occurrences, the delivery mode via video chat (9, Written) combined with Peer Feedback has the most substantial relationship compared to the Recipient and Sender theme. The Positive and Negative Perceptions within the code co-occurrence are distributed equally. The following quote highlights the importance of effective delivery of feedback, as feedback that is not implemented correctly can have negative consequences on group performance and overall results:
93:15: “The Feedback given during the meeting regarding the tasks was then also no longer implemented by all. This worried me a bit myself because I’m aiming for a good result in this module, and this only works if we pull together as a group.”
The student’s concern for achieving a good outcome underscores the need for clear and impactful feedback communication to ensure that all group members understand and act upon the feedback provided. This also indicates that the Delivery Mode of Video Chat is the most popular for Peer Feedback.
6Discussion and integration of triangulation outcomes
The present study addresses the characteristics of formative feedback in COIL modules, which are necessary for a successful implementation, and the connections between these characteristics. The findings, as presented above, have not only theoretical and practical implications for the further development and optimization of the COIL module but also contain limitations that are thoroughly considered in this section.
6.1Theoretical implications and future research directions
Table 1 presents the hypotheses derived from previous empirical research [7], which are tested based on the results of the thematic data analysis in this study. Theoretical implications of the analysis will be discussed in the same order as in the results section and compared with previous research on the impact of formative feedback in virtual exchange.
Table 1
Hypotheses generated by Altmann [7] and tested in this paper
| Hypothesis | Status |
| H1: The availability of an E-Tutor has a positive influence on the perception of the E-tutor | Confirmed |
| H2: Positive formulation of formative feedback has a positive impact on student motivation | Confirmed |
| H3: Praise included in formative feedback fosters increased student motivation | Not Confirmed |
| H4: Peer feedback is always perceived positively | Not Confirmed |
| H5: The relation to the E-Tutor affects the positive, or negative perception of formative feedback | Confirmed |
| H6: The relation to the E-Tutor influences the implementation of formative feedback. | Not Confirmed |
| H7: Positive perception of feedback leads to its implementation | Not Confirmed |
| H8: Feedback linked to grades is implemented, regardless of its positive or negative perception | Confirmed |
| H9: Tool feedback is more needed at the beginning of a module | Confirmed |
| H10: Process Feedback is more needed at the beginning of the module | Not Confirmed |
| H11: Task-related Feedback is more important in the middle and end of the module | Confirmed |
| H12: Students who feel motivated through formative feedback tend more to recognize feedback positively | Confirmed |
| H13: As not applicable perceived feedback often leads to negative perception, or is directly not implemented | Confirmed |
6.1.1Feedback perception & realization
The data have a strong code co-occurrence regarding feedback realization and positive perception. Despite the substantial overlap, a small proportion of the positively perceived feedback is said to have only been partially implemented, meaning that H7 cannot be confirmed. Regarding the only partially realized feedback, there is the possibility that students might reflect on the received feedback and decide together in the group what will be implemented. Regarding the relationship with the e-tutor, it can be stated that it also significantly impacts the positive perception of the feedback. This effect is significantly weaker in the available data concerning the realization of the feedback. Thus, H5 can be confirmed, and H6 cannot be confirmed.
6.1.2Feedback effects
It became apparent that students who link Feedback to the grade and assessment seem to implement it even though they perceive it negatively. This confirms H8. Concerning this ambivalent phenomenon, a disentanglement of feedback and assessment, as demanded by [49], could be a challenge in future iterations of the module and, in general, for the formative feedback designs. H12 has been confirmed, which assumes a link between Motivation through Feedback and its realization. Furthermore, other factors not analyzed in this paper may have also affect students’ motivation, e.g. student engagement, learning success, traits etc., but cannot be adequately evaluated due to the limitations of this study (see section 6.3).
6.1.3Content
When analyzing the theme Content, special attention must be paid to the survey time points during the module. We found that Tool-related Feedback, role-specific Feedback and individual Feedback are particularly needed at the beginning of the module and thereby confirm H9. Especially at this module stage, there are uncertainties regarding using the collaboration platform. However, these can be quickly overcome with the support of the E-Tutor. In contrast, Peer feedback, Task-related Feedback and Assessment-related Feedback are of higher demand in the later module stage, which confirms H11. Finally, H10 was disproved. Process-related feedback is seen as unnecessary at the beginning of the module; it turned out that such feedback is needed continuously throughout the module. Also, [50] argues that feedback should be seen as a continuous dialogue between instructors and learners rather than a one-way transmission of information.
6.1.4Timing and frequency
Our research shows that the Availability of the E-Tutor has an essential impact on the Positive Perception of the feedback. Thus, feedback is perceived positively when students receive a quick response. Hattie & Timperley [4] suggest that feedback Timing depends on the level and purpose of feedback and should be provided when students’ efforts can be affected. In contrast, a negative perception results if a response takes too long. Thus, due to the substantial occurrences in the data, H1 can be confirmed in this study. There is also a relationship between applicability and negative perception. In those cases, we also found a code co-occurrence with the code Not Realized so that H13 can be confirmed. Furthermore, an expansion of the Obligatory Feedback should be considered since the need was significantly stronger, especially in the second half of the survey.
6.1.5Recipient and sender
Considering the code co-occurrences between the theme Recipient & Sender and Feedback Needs, the following thing stands out: Peer feedback has the most substantial effect on motivation, e-tutors show trust and orientation as the strongest effect, and instructor feedback shows the most potent effect with reflection. Thus, it can be assumed that the respective agent best serves different feedback effects. In this context, peers as feedback senders should not be underestimated, as they can provide valuable feedback with the right tools and training, as [1] has also noted. Furthermore, developing a trustful relationship with the e-tutor can help improve students’ engagement with feedback. [51] also notes, virtual teams with trust achieve better learning results when interacting with feedback. The study showed that peer feedback is not always perceived positively, which disproves H4. The data reveals that negative perceptions were often caused by conflicts within the group.
6.1.6Style and tone
When considering the theme of style and tone, the code’s positive formulation and constructive formulation become apparent. Both codes have strong code co-occurrences with the code positive perception. However, it can be seen that positive formulation, as well as constructive formulation, influence the student’s motivation. As already analyzed in the sub-section, Feedback Effects, Motivation and Positive Perception also have a strong code co-occurrence. At this point, it becomes clear that there is a chain of effects between Positive and Constructive Formulation, Motivation and Positive Perception of the feedback so that H2 can be confirmed. Data also showed that praise influences motivation, but with only four code co-occurrences, the connection is too weak, so H3 cannot be confirmed.
6.1.7Delivery mode
Peer feedback is preferably provided via video chat. In this mode, feedback is given to improve the learning results and resolve conflicts in the group. The mainly used written Delivery Mode seemed accepted by most students as there were only marginal Negative Perceptions. Nevertheless, some students complained that the teachers were only available via e-mail instead of chat. In future interactions with the module, the availability of teachers on the learning platform should be considered.
To conclude, our analysis highlights the multifaceted nature of feedback in the virtual learning context, emphasizing the significance of timely responses, the nuances in content, and the contributing roles of both the sender and recipient. With the confirmation of 8 hypotheses compared to the falsification of 5 hypotheses, the findings also show the importance of constructive formulations in feedback, the delivery mode preferences, and the crucial role of e-tutors.
6.2Practical implications
With the following semantic network, we not only want to highlight the relationship of themes based on the hypotheses but also draw practical conclusions and provide guidelines for the development of COIL modules in higher education virtual exchange learning environments. In this model (see Fig. 7), the effects on students as feedback recipients are interpreted from the perspective of the impacting context.
Fig. 7
Refined semantic network of students dealing with formative Feedback in COIL modules.
6.2.1Providing regular feedback by e-tutors, teachers and technology-assisted feedback
Based on the sender of the feedback, the strongest effects differ. It can be suggested that e-tutors should focus on creating trust and orientation among students, while teachers are responsible for supporting reflection and increasing peer motivation. Additionally, positive and constructive wording positively affects motivation for all these senders of feedback, which in turn is strongly linked to the positive perception of the feedback. Furthermore, the availability of speedy responses determines whether the feedback is perceived positively or negatively. Likewise, not understood as applicable feedback is also not implemented and continues to be perceived negatively. However, if the feedback is perceived to be related to the grade, students realize it equally, regardless of whether they find it positive or negative. Finally, positive perceived feedback is usually implemented, or partially implemented, after students have reflected on it in the team. The model does not claim to be generally valid but represents an interpretation of the data collected about the current module that emerged in an iterative action research process. The semantic network (Fig. 7) could also be used to train a social learning chatbot, although further, predominantly quantitative analysis is needed.
6.2.2Peer feedback
Feedback between students is an essential component of any COIL module that should be enhanced more strongly. For example, by introducing formalized group work contracts at the beginning of the module in which the groups agree on the type and scope of Peer Feedback. Furthermore, e-tutors should also be responsible for encouraging the provision of peer feedback. Especially the tight link between peer feedback and motivation underlines the potential for more effective learning experiences for students. As can be seen in the different survey periods, feedback from teachers and e-tutors happened significantly more often in the first half than in the second half, while peer feedback would be more beneficial in the module’s second half. It is therefore advisable to strongly encourage peer feedback, especially in the module’s second half, which would also relieve the burden on teachers and e-tutors at this stage of the module. In doing that, learning can be improved, and teachers can relieve some of their workloads when peer feedback is used alongside teacher feedback [52].
6.2.3Content of the feedback
Furthermore, the feedback content at the beginning of the module should focus more on tools, roles and individual feedback, while the second part should focus on peer, task and assessment feedback. The continuation of the obligatory feedback should be carefully considered since it was frequently requested, but also further feedback could be obtained on request at any time. About half of the students who requested the continuation complained about a lack of but, at the same time, did not request feedback. In conclusion, the implementation of feedback cannot be forced, but by adjusting and optimizing the previously explained effects, better conditions for implementation can be created.
6.3Limitations
Potential sources of biases that may have influenced the results are related to the rigor and relevance of the research design and data collection. Firstly, the students’ reflective capacity may have influenced their responses, as not all students might possess the same level of ability for self-reflection. Secondly, the courtesy responses of the students, also known as the social desirability bias [53], could have impacted the rigor of the results. This phenomenon refers to the tendency of respondents to provide answers they believe are preferred or expected by the researchers. Additionally, our research was only partially anonymous; while individual names were not included, group numbers were provided.
Secondly, a selection bias was introduced as we did not use a random sample but chose a real group that was enrolled in the course, though this choice was justified by our research design. Observer bias, where the researcher’s expectations or knowledge might influence the observations, is also a concern. Finally, citation or publication biases in the discussion of the results might have affected the visibility and recognition of specific results based on their nature, potentially influencing the general perception of the research field.
Thirdly, despite our study’s thoroughly planned mixed method and longitudinal design, some limitations regarding the validity and generalizability of our findings need to be addressed. The research is limited on the one hand by the current cohort of students. Further, the online-based self-reflective journal did not allow for further conversation with the students, such as follow-up questions to better evaluate the relations between the themes. Despite an extensive search for thematic co-occurrences in the coded material, we tried to not only interpret emergent themes after checking co-occurrences using cross tables and Sankey diagrams but also tried to understand the relationships between those themes [45]. To refine the specific co-occurrences, we looked for document-code-co-occurrences. We extracted the relevant documents to discover new relationships between the themes and code co-occurrences within this specific set of documents that were not obvious in the first coding round. Even though the high number of students and journals we have reviewed, our results are not generalizable but need further quantitative testing.
7Conclusion
The present study shows that formative feedback is an essential component of the collaborative learning in COIL modules, as it provides learners with relevant information about their performance, and helps them to improve the quality of their work and their contribution to the study group. However, the effects of feedback can vary depending on the agent providing it. This paper aimed to explore not only the effects of feedback from e-tutors, teachers, and peers on learners’ trust, orientation, reflection, motivation, learning outcomes, and the teacher’s workload but also on the implementation of formative feedback. As highlighted in the present study, e-tutors play a crucial role in online learning environments, where building trust and providing orientation are essential. Teachers, on the other hand, can constantly support reflection through their feedback. Future studies should focus on peer feedback and its potential to improve learning outcomes and reduce the teacher’s workload.
Acknowledgments
We sincerely thank the Society of Friends and Sponsors (Gesellschaft der Freunde und Förderer –GFF) of the TU Dresden for co-sponsoring this paper. The module analyzed in the paper was awarded by GFF as an “Innovative Module in the Field of Sustainability and Entrepreneurship” in 2022. Part of the prize money provides funding for this research and the continuous development and improvement of the module. Additionally, we thank the Erasmus+Project G4CNGE: Playful Learning in Next Generation Entrepreneurship (2022-1-DE01-KA220-HED-000088935, FH Dresden) for financially supporting this research.
Author contributions
CONCEPTION: Mattis Altmann and Maik Arnold
METHODOLOGY: Mattis Altmann and Maik Arnold
DATA COLLECTION: Mattis Altmann
INTERPRETATION OR ANALYSIS OF DATA: Mattis Altmann and Maik Arnold
PREPARATION OF THE MANUSCRIPT: Mattis Altmann and Maik Arnold
REVISION FOR IMPORTANT INTELLECTUAL CONTENT: Mattis Altmann and Maik Arnold
SUPERVISION: Mattis Altmann
References
[1] | Panadero E , Lipnevich A A review of feedback typologies and models: Towards an integrative model of feedback elements, Educational Research Review. (2021) ;35: :100416. DOI: 10.1016/j.edurev.2021.100416. |
[2] | Shute VJ Focus on Formative Feedback, ETS Research Report Series ((2007) )1:1–47. DOI 10.1002/j.2333-8504.2007.tb02053.x. |
[3] | Boud D , Molloy E Rethinking models of feedback for learning: The challenge of design, Assessment & Evaluation in Higher Education. (2013) ;38: (6):698–712. DOI: 10.1080/02602938.2012.691462. |
[4] | Hattie J , Timperley H The Power of Feedback, Review of Educational Research. (2007) ;77: (1):81–112. DOI: 10.3102/003465430298487. |
[5] | Brookhart SM How to Give Effective Feedback to your Students. 2. Alexandria: ASCD; 2017. |
[6] | Geister S , Konradt U , Hertel G Effects of Process Feedback on Motivation, Satisfaction, and Performance in Virtual Teams, Small Group Research. (2006) ;37: :459–89. DOI: 10.1177/1046496406292337. |
[7] | Altmann M Optimizing formative feedback guidelines in collaborative online international learning. IDIMT-2022 Digitalization of society. 2022; business and management in a pandemic; 30th Interdisciplinary Information Management Talk/Chroust:12 pages. |
[8] | Herr K , Anderson GL The Action Research Dissertation. 3. Thousand Oaks: Sage Publications; 2005. DOI: 10.4135/9781452226644. |
[9] | Mills GE Action Research –A Guide for the Teacher Researcher. 4. Boston: Pearson; 2011 |
[10] | Donohoe A , Maddison W , Hughes J , Basha AMM , Dardari L , Kayal A Language Interpreting in Undergraduate Medical Education: Developing a Framework for Educational Practice. International Journal of Learning and Development. 2021. |
[11] | O’Connor C , Joffe H Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines. International Journal of Qualitative Methods. 2020. |
[12] | Thomas E , Magilvy JK Qualitative Rigor or Research Validity in Qualitative Research, Journal for Specialists in Pediatric Nursing ((2011) ) 16: (2):151–5. |
[13] | Daniel BK Empirical Verification of the “TACT” Framework for Teaching Rigour in Qualitative Research Methodology, Qualitative Research Journal ((2018) ) 18: (3):262–75. |
[14] | Braun V , Clarke V Using thematic analysis in psychology, Qualitative Research in Psychology. (2006) ;3: (2):77–101. DOI: 10.1191/1478088706qp063oa. |
[15] | Abrenica JT , Cascolan HMS Impact of Action Research in Education: Experiences and Challenges Faced by Teachers, International Journal of Scientific and Management Research ((2022) ) 5: (2):1–15. |
[16] | Mertler CA Action Research: Improving Schools and Empowering Educators. 2017. |
[17] | Sein MK , Henfridsson O , Purao S , Rossi M , Lindgren R Action Design Research, MIS Quarterly ((2011) ) 35: (1):37–56. |
[18] | Phelps R , Hase S Complexity and Action Research: Exploring the Theoretical and Methodological Connections, Educational Action Research ((2002) ) 10: (3):507–24. |
[19] | Harvey M , Jones S Enabling leadership capacity for higher education scholarship in learning and teaching (SOTL) through action research, Educational Action Research. (2021) ;29: (2):173–90. DOI: 10.1080/09650792.2020.1803941. |
[20] | Hutchings P , Huber MT , Ciccone A The Scholarship of Teaching and Learning Reconsidered: Institutional Integration and Impact. San Francisco, CA: Jossey-Bass; 2011. |
[21] | Gibbs P , Cartney P , Wilkinson K , Parkinson J , Cunningham S , James-Reynolds C , et al. Literature review on the use of action research in higher education, Educational Action Research. (2016) ;25: (1):3–22. DOI: 10.1080/09650792.2015.1124046. |
[22] | Goodyear P Teaching as design, HERDSA Review of Higher Education ((2015) ) 2: , 27–50. |
[23] | Seufert S , Euler D Learning Design: Gestaltung eLearning-gestützter Lernumgebungen in Hochschule und Unternehmen. St. Gallen: Inst. f. Wirtschaftspäd; 2005. 57 p. (SCIL-Arbeitsbericht). |
[24] | Stevens Initiative. Virtual Exchange Typology [Internet]. Aspen Institute; 2020 [cited 2022 May 13]. Available from:https://www.stevensinitiative.org/wp-content/uploads/2021/09/Stevens-Initiative-Virtual-Exchange-Typology_090121_singlepages.pdf. |
[25] | O’Dowd R Internationalising higher education and the role of virtual exchange. Abingdon, Oxon; NewYork,NY: Routledge; 2023. (Routledge research in higher education). |
[26] | Hauck M Virtual exchange for (critical) digital literacy skills development, European Journal of Language Policy. (2019) ;11: (2):187–210. DOI: 10.3828/ejl2019.12. |
[27] | Nishio T , Fujikake C , Osawa M Language learning motivation in collaborative online international learning: An activity theory analysis, Journal of Virtual Exchange. (2020) ;3: :SI-IVEC27–47. DOI: 10.21827/jve.3.35780. |
[28] | Lenkaitis CA , Calo S , Venegas Escobar S Exploring the intersection of language and culture via telecollaboration: Utilizing videoconferencing for intercultural competence development, International Multilingual Research Journal. (2019) ;13: (2):102–15. DOI: 10.1080/19313152.2019.1570772. |
[29] | O’Dowd R , Dooly M Intercultural communicative competence through telecollaboration and virtual exchange. In: Jackson J, editor. The Routledge Handbook of Language and Intercultural Communication. Milton Park; New York: Routledge; 2012. p. 361-75. |
[30] | Von Der Emde S , Schneider J , Kötter M Technically Speaking: Transforming Language Learning through Virtual Learning Environments (MOOs), The Modern Language Journal. (2001) ;85: (2):210–25. DOI: 10.1111/0026-7902.00105. |
[31] | González-Ceballos I , Palma M , Serra JM , Esteban-Guitart M Meaningful Learning Experiences in Everyday Life During Pandemics. A Qualitative Study. Frontiers in Psychology [Internet]. 2021 [cited 2022 Mar 9];12. DOI:10.3389/fpsyg.2021.670886. |
[32] | Schmitt K , Altmann M , Clauss A , Schulze-Stocker F , Arnold M , Rebane G Moving Beyond Mobility: Lessons Learned from a Project-Based Virtual International, Intercultural, and Interdisciplinary Collaboration, Lessons Learned. (2022) ;2: (2):12. DOI: 10.25369/ll.v2i2.57. |
[33] | Maraza-Quispe B , Caytuiro-Silva N , Castro-Gutierrez E , Alejandro-Oviedo M , Choquehuanca-Quispe W , Fernandez-Gambarini W , et al. Towards the Development of Collaborative Learning inVirtual Environments. IJACSA [Internet]. 2019 [cited 2021 Aug 6];10(12). Available from: http://thesai.org/Publications/ViewPaper?Volume=10&Issue=12&Code=IJACSA&SerialNo=37. |
[34] | Goldin I , Narciss S , Foltz P , Bauer M New Directions in Formative Feedback in Interactive Learning Environments, Int J Artif Intell Educ ((2017) ) 27: (3):385–92. |
[35] | Hattie J , Yates GCR Visible Learning and the Science of How We Learn [Internet]. Taylor & Francis; 2013. Available from:https://books.google.de/books?id=1RZBAQAAQBAJ. |
[36] | Vattøy KD , Gamlem SM , Rogne WM Examining students’ feedback engagement and assessment experiences: A mixed study, Studies in Higher Education ((2021) ) 46: (11):2325–37. |
[37] | Schoop E , Sonntag R , Altmann M , Sattler W Imagine it’s “Corona” and no one has noticed. LL [Internet]. 2021 Jul 21 [cited 2023 May 30];1(1/2). Available from:https://journals.qucosa.de/ll/article/view/33. DOI: 10.25369/ll.v1i1/2.33. |
[38] | Biggs J Enhancing teaching through constructive alignment, High Educ ((1996) ) 32: (3):347–64. |
[39] | Creswell JW , Clark VLP Designing and conducting mixed methods research. 3rd ed. Thousand Oaks: Sage; 2017. |
[40] | Dawadi S , Shrestha S , Giri RA Mixed-Methods Research: A Discussion on its Types, Challenges, and Criticisms, JPSE. (2021) ;2: (2):25–36. DOI: 10.46809/jpse.v2i2.20. |
[41] | Döring N , Bortz J Forschungsmethoden und Evaluation in den Sozial- und Humanwissenschaften [Internet]. Berlin, Heidelberg: Springer Berlin Heidelberg; 2016 [cited 2021 Aug 6]. (Springer-Lehrbuch). Available from: http://link.springer.com/10.1007/978-3-642-41089-5. |
[42] | Maguire M , Delahunt B Doing a Thematic Analysis: A Practical, Step-by-Step Guide for Learning and Teaching Scholars, All Ireland Journal of Higher Education. (2017) ;8: (3):3351–33514. |
[43] | Straub J Understanding cultural differences: Relational hermeneutics and comparative analysis in cultural psychology. In: Straub J, Weidemann D, Kölbl C, Zielke B, editors. Pursuit of Meaning Advances in Cultural and Cross-Cultural Psychology. Bielefeld: Transcript; 2006. p. 163-213. |
[44] | Fereday J , Muir-Cochrane E Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development, International Journal of Qualitative Methods. (2006) ;5: (1):80–92. DOI: 10.1177/160940690600500107. |
[45] | Scharp KM Thematic Co-occurrence Analysis: Advancing a Theory and Qualitative Method to Illuminate Ambivalent Experiences, Journal of Communication. (2021) ;71: (4):545–71. DOI: 10.1093/joc/jqab015. |
[46] | Burks AW Peirce’s theory of abduction, Philosophy of Science ((1946) ) 13: (4):301–6. |
[47] | Prensky M Digital Natives, Digital Immigrants Part 1, On the Horizon. (2001) ;9: (5):1–6. DOI: 10.1108/10748120110424816. |
[48] | Benítez-Márquez MD , Sánchez-Teba EM , Bermúdez-González G , Núñez-Rydman ES Generation Z Within the Workforce and in the Workplace: A Bibliometric Analysis, Front Psychol. (2022) ;12: :736820. DOI: 10.3389/fpsyg.2021.736820. |
[49] | Winstone NE , Boud D The need to disentangle assessment and feedback in higher education, Studies in Higher Education. (2022) ;47: (3):656–67. DOI: 10.1080/03075079.2020.1779687. |
[50] | Jensen LX , Bearman M , Boud D Understanding feedback in online learning –A critical review and metaphor analysis, Computers & Education. (2021) ;173: :104271. DOI: 10.1016/j.compedu.2021.104271. |
[51] | Peñarroja V , Orengo V , Zornoza A , Sánchez J , Ripoll P How team feedback and team trust influence information processing and learning in virtual teams: A moderated mediation model, Computers in Human Behavior. (2015) ;48: :9–16. DOI: 10.1016/j.chb.2015.01.034. |
[52] | Nicol D , Thomson A , Breslin C Rethinking feedback practices in higher education: A peer review perspective, Assessment & Evaluation in Higher Education. (2014) ;39: (1):102–22. DOI: 10.1080/02602938.2013.795518. |
[53] | Holden R.R. , Passey J. (2009). Social desirability. In M. R. Leary & R. H. Hoyle (Eds.), Handbook of individual differences in social behavior (pp. 441–454). The Guilford Press. |
