Assessment of problem and project-based activities in a CLIL course

Full Text

Introduction

The use of CLIL methodology in foreign language learning has proven to be a promising solution for the challenges faced by the current education system. It is an approach that integrates content and language learning, which allows students to learn a foreign language and a specific discipline concurrently. This approach is believed to have a huge didactic potential, as it develops advanced language skills in professional spheres (Coyle, Hood, Marsh, 2010; Vavelyuk, 2015); it provides an excellent context for meaningful input, communication and output (Mehisto, 2012); it is aimed at STT (student talking time) and reduce TTT (teacher talking time) (Vdovina, 2018); it appeals to different learning styles and is often cross-curricular (Gulaya, Romanova, 2016); it makes bilingualism/trilinguism in mainstream education a realistic and achievable aim(Kukulska-Hulme et al., 2021).

In order to effectively teach a foreign language, the focus must be on developing the necessary language skills required for communication in the professional field (Sysoev, 2021).  Language courses should aim to solve professional problems and require the development of analytical, creative, critical skills and group work. This approach necessitates assessing, monitoring, and reflecting on the process of project- and problem-based activities within the framework of a CLIL course.

However, peer assessment, reflection, and self-assessment are currently weak points in the educational process. Students are not involved in monitoring or assessment, nor are they included in the development or selection of assessment criteria, peer and self-assessment, or reflective activities after completing a course. As a result, students cannot correlate their learning outcomes with the teacher's goals. This is why it is crucial to incorporate these components in the methodological system.

Assessing, monitoring, and reflecting on the process of project- and problem-based activities within the framework of a CLIL course can be divided into several stages. The first stage involves assessing the students' initial language level and determining their ability to understand and use professional vocabulary. The second stage involves monitoring the students' progress throughout the course and evaluating their performance in both language and content learning. The third stage involves reflection on the learning process, allowing students to evaluate their own progress and set goals for future learning.

The use of CLIL methodology in foreign language learning is a step towards achieving bilingualism/trilinguism in mainstream education (Marsh, 2012). However, it is important to integrate assessment, monitoring, and reflection into the methodological system to ensure that students are able to correlate their learning outcomes with the teacher's goals (Nicol, Macfarlane-Dick, 2006). By doing so, students will have a better understanding of their strengths and weaknesses, leading to more effective learning and improved language skills in the professional field.

The characteristics of problem-based activities used in CLIL

According to Professor D. Marsh, content and language integrated learning is based on four invariant components: content, communication, culture, cognition (Marsh, 2012). Content refers to a specific subject area where knowledge, skills, and abilities are developed. Communication involves exchanging information and ideas. Culture involves studying language within a community's or country's culture. Cognition includes developing analytical, critical, and creative skills.

CLIL has a positive impact on learners' attitudes towards language learning and their own abilities as language learners. According to D. Marsh, language learning becomes "acquisitional" rather than just "intentional" (Marsh, 2012). As a result, learners develop pluriliteracy skills, which are higher-order thinking skills. They become more curious, analytical, and interactive thinkers as well as better problem-solvers. CLIL also teaches learners how to work collaboratively, take creative risks, and manage their time effectively. This approach leads to greater engagement and motivation among learners (Ball, Kelly, Clegg, 2013).

To prioritize these benefits, CLIL courses emphasize group, problem- and project-based activities such as projects, case studies, brainstorming sessions, simulations, discussions, and roundtables. Table 1 lists the characteristics of common problem-based assignments integrated into CLIL courses (see Table 1).

Table 1. Characteristics of the problem-based activities integrated in CLIL

Web projects used for language learning have multiple benefits, as they help students develop language skills, analytical abilities, creativity, and research skills (Bertaux et al., 2009). These projects also promote collaborative skills necessary for interaction, peer-evaluation, and reflection among students (López-Medina, 2016).

Project-based activities typically involve several stages, including identifying and selecting a problem, gathering information and creating an electronic portfolio, proposing and discussing potential solutions, selecting online tools for publishing the project, presenting the project, and evaluating the project while reflecting on the process (Titova, 2017).

A group project assessment model was implemented in an online CLIL course for bachelor students in the Department of Regional Studies and International Relations at Lomonosov Moscow State University. This multi-stage assessment involves the use of various online tools and methodological technologies, as outlined in Table 2.

Table 2. The stages and technologies of assessment of project-based activities in the online CLIL course

During the initial stage of the project, students are monitored for their ability to identify problems, search for materials, and define project objectives. All materials are compiled into an e-portfolio, which serves as a formative assessment tool. Feedback given at this stage aims to correct any issues with the process of searching, collecting, and analyzing information. The e-portfolio also encourages students to improve their analytical and cognitive skills by identifying errors and inaccuracies in their written speech. Constructive feedback not only corrects mistakes but also contributes to developing a further plan of action, encouraging reflective activity, and increasing student motivation. Ultimately, constructive feedback helps students feel more confident, boosts their motivation, and creates a psychologically comfortable and trusting environment for learning (Hattie, Timperley, 2007).

Feedback provides concrete assistance in overcoming psychological and educational problems and difficulties, contributes to the individualization of the educational process. J. Hattie and H. Timperly identify five characteristics of effective feedback. It should be specific, addresses the learner’s advancement toward a goal, praise their efforts instead of intelligence. Feedback should be constructive, because it helps develop an awareness of student learning, and they are more easily able to recognize mistakes and eventually develop strategies for tackling weak points themselves. It should be fast and timely, balanced and correct. Too much praise can convey a sense of low expectation and, as a result, can be demotivating; constructive feedback given in front of others, even if it is well-intended, can be read as a public attack on them and their ability (Hattie, Timperley, 2007).

Today, feedback can be provided by the teacher in online form with the help of podcasts (audio comments), videocasts (video comments), screencasts (audio comments of the computer desktop), dynamic texts with comments based on digital tools (see Table 2). Online feedback is fast and timely. The teacher spends much less time providing, for example, audio comments. A video or audio file can be instantly published or sent to students. The next advantage is personification and targeting. It is well known that video messages create the effect of "personal presence" of the teacher (voice, video), that is extremely important for motivation and the interactivity of the educational process (Improving the effectiveness of language learning, 2014). In addition, digital feedback files can be saved, systematized, analyzed, they can be included in the student's electronic portfolio, used as recommendations for further development (Zaripova, Salekhova, Danilov, 2017). To publish portfolio materials in the course blogger.com is used, which is synchronized with Google. It has a mobile version, and allows the teacher and students to publish feedback in the form of text or audio files instantly.

At the second stage, students peer evaluate project products in terms of their content and design. The teacher formulates assessment criteria in advance, discusses these criteria with students, who make corrections and offer their own criteria. It is important to note that for peer-assessment it is necessary to use well-defined criteria. Complex formulations should be avoided, those criteria should be adapted for students (see Table 3). Only in this case, peer evaluation will help students not only objectively evaluate each other's projects, but also take a critical look at their own work.

The criteria are divided into two groups: for evaluating the quality of the content of the project and evaluating the quality of visual and language design of the project task. The proposed criteria of the first group correlate with the professional competencies of Educational Standards of Moscow State University in the field of Foreign Regional Studies (Educational Standard of Moscow State University, 2021).

Table 3. Peer-evaluation gradebook to assess the content and design of the project-based activities

At the second stage of assessment, the presentation skills of the project team members are also an object. As already mentioned, complicated methodological terms that are common in language assessment, such as accuracy, fluency should be avoided. An example of a jointly created table for assessing presentation skills is demonstrated in Table 4 (see Table 4).

Table 4. Peer-evaluation gradebook to assess the presentation skills

At the third stage, the content of the projects is evaluated by the teacher according to carefully developed criteria that evaluate both the content and the design of the online project product. To provide timely, constructive feedback, the teacher records a screencast or podcast, which is published on the course website.

Research data and evaluation of a reflective survey

At the final stage, students reflect. Reflection can be carried out with the help of questionnaires, interviews, checklists, essays, interviews. A qualitative survey was also conducted, revealing the attitude of students to innovative approaches to assessing tasks in the course. Conducting a final, reflective assessment helps to find out how successful the learning process was from the point of view of students. During the course, a questionnaire consisting of 5 open-ended reflective questions was used. Open questions for reflection on project activities were formulated as follows:

1. What did you learn about the problem you were working on during the project?
2. What new skills did you acquire while working on the project?
3. What skills and abilities did you improve while working on the project?
4. What problems did you face while working on the project? How did you solve them?
5. What resources did you use while working on the project? Which ones were especially helpful? Which ones would you use again?

Summarizing the results of a reflective survey in which 42 students from the Department of Regional Studies and International Relations took part in 2021 and 2022, we can say that all students noted that the problem-oriented project topic, clear instructions for implementation, group interaction and feedback from the teacher allowed acquire in-depth knowledge in their professional area. It was also found that many students chose topics related to the group project for their graduate papers, indicating that the project had a lasting impact on their academic pursuits.

Furthermore, the majority of students noted improvements in their foreign language skills, both in writing and speaking. This was attributed to the fact that the project was created in the form of an analytical site on the formulated problem. In addition to this, 20 students mentioned that they had developed their presentation skills while working on the project.

The survey also highlighted that all 42 students learned how to analyze various data (tables, graphs, infographics) on a chosen professionally oriented problem. Additionally, 38 students identified new skills related to the analysis of data on socio-economic, demographic, cultural, and other processes and phenomena in the region of specialization. This demonstrates that the project was successful in equipping students with practical skills that will benefit them in their future careers.

However, the survey also revealed some challenges faced by students during the project. The main difficulties included psychological difficulties related to group work, distribution of responsibilities, completing tasks on time, and creating a joint e-portfolio. Many students also spoke about the lack of time as a significant challenge.

Overall, the survey results suggest that the problem-based project topic, clear instructions, group interaction, and feedback from the teacher were key factors in promoting in-depth knowledge acquisition and practical skill development among the students. While there were some challenges faced during the project, the positive attitudes towards new assessment technologies suggest that students are willing to embrace innovative methods for learning and evaluation.

Research data of a qualitative survey

At the final stage of the survey, the students were asked to express their attitude towards new assessment technologies. The proposed Likert questionnaire with a five-point assessment scale included 10 questions, and all 42 students participated. The results showed that all students expressed positive attitudes towards new assessment technologies, indicating that they are open to using innovative methods for evaluating their academic performance. Table 5 presents the results of this survey.

Table 5. The results of the survey of students attitude to new assessment technologies, to peer assessment and reflection

Interpretation of a qualitative survey

The results of a qualitative survey indicate that students have a positive attitude towards new assessment technologies, peer assessment, and reflection. The majority of participants (40 people) found electronic portfolios to be an extremely useful tool for monitoring and correcting educational activities during project creation. Students (39 people) emphasized the importance of timely feedback at the portfolio stage and found teacher comments in the electronic portfolio to be helpful in overcoming psychological and educational problems. Most students (40 people) agreed that peer evaluation of project content and presentation helped correct mistakes and objectively evaluate groupmate work.

Additionally, all respondents (42 people) expressed the opinion that students should be involved in developing criteria for evaluating the project product, and reflection helped them see their own achievements and failures as well as the dynamics of their professional development. The majority of students (42 people) agreed that the proposed assessment and monitoring system contributed to the development of new skills and greater involvement in the learning process, as evidenced by the results of the reflective questionnaire.

However, some respondents had criticisms regarding questions 2, 4, and 5. This could be due to students' lack of peer assessment skills that need to be developed, as well as difficulties in overcoming psychological barriers during group work. Despite these criticisms, the analysis showed that the majority of students believe that the presented model for evaluating project assignments is effective, promotes their involvement in the learning process, and helps develop motivation, communication, and professional skills.

Conclusion

The approach presented for evaluating project tasks in a CLIL course involves multiple stages and utilizes digital tools to monitor and evaluate project creation progress, make necessary corrections, and provide timely and constructive feedback. Implementing online tools for feedback and assessment encourages student engagement and motivation in the learning process. Furthermore, it is essential that the final stage of assessment includes reflection, which is required for all problem-based activities and helps develop students' reflective skills. Potential directions for further research include exploring mutual assessment skills, addressing psychological barriers in reflective activity, and using reflective questionnaires to identify areas for improvement in the educational process.

About the authors

Svetlana V. Titova

Author for correspondence.
Email: stitova@gmail.com
ORCID iD: 0000-0002-7930-3893
Scopus Author ID: 56600196700

Doctor of Pedagogy, Deputy Dean for Continuous Professional Education,

Head of the Department of Theory of Teaching Foreign Languages,

Faculty of Foreign Languages and Regional Studies

References

Supplementary files