An evaluation framework

Published:

Online Teaching/Learning Strategies: An Evaluation Framework

ABSTRACT: The core of this work is the development of an evaluation framework for online teaching/learning strategies. As far as context is concerned, the module under analysis is Educational Software Evaluation (ESE), which is part of the Master's Course on Multimedia in Education of the University of Aveiro. Set as the conclusion of a wider study, this paper aims at (i) summarizing the teaching/learning strategies implemented, (ii) describing the assessment strategies regarding the learning process and products, (iii) presenting the results of the evaluation based on three vectors: the students' perceptions concerning the module, the teacher performance and the strategies implemented. The results indicate that the majority of the students considered the module academically and professionally relevant, but they felt uncomfortable with peer evaluation and that monitoring of the activities was scarce. Nonetheless, students underlined it helped them developed general competences, such as learning autonomy and online collaboration.

Lady using a tablet
Lady using a tablet

Professional

Essay Writers

Lady Using Tablet

Get your grade
or your money back

using our Essay Writing Service!

Essay Writing Service

Keywords: bLearning, evaluation of educational software, teaching/learning evaluation, teaching/learning strategies, eAssessment, collaboration.

1. Introduction

Due to the Bologna Declaration, Higher Education Institutions (HEI) are reorganizing and rethinking their educational offer, which also leads to the redefinition of the courses in terms of teaching/learning strategies. Moreover, the development of Information and Communication Technologies (ICT) has led several of these HEI to put forward eLearning or bLearning courses or modules. In this context, and as the European Association for Quality Assurance in Higher Education mentions, it is essential for the success of these courses that HEI develop evaluation mechanisms for "the approval, periodic review and monitoring of their programmes" [1:16].

Although evaluation in education is not a new concern, it should be revised when ICT tools are explored in innovative (online) contexts. This paper is the conclusion of a larger study that seeks to develop and test a framework to evaluate online teaching/learning strategies. Therefore, in the next section the theoretical framework that guided the design of the strategies explored in the ESE module is presented and in the third section the structure of the Educational Software Evaluation (ESE) module is described[1]. Then the methodology used and the results of the empirical study are presented and, finally, some final considerations are drawn.

2. Theoretical framework

Because of the fast evolution of ICT and the resultant ability to share information and build new knowledge, the emerging Information Society creates challenges for initial training, which is no longer adequate to meet the needs of a more and more demanding labor market that requires vocational skills to be in a process of constant updating. In this context, the Bologna Declaration introduced a new model for the organization of the European Higher Education (HE) systems in terms of study cycles. One of the principles underlying the study cycles pertains to the transition from a teaching system in which information is provided to one based on the development of competences.

Moreover, recently students tend to be digitally literate and, in the educational context, it is important that they develop ICT-related skills in order to be able to react to the challenges of a digital society. In this scenario, several authors (e.g. [3], [4]) assert that teaching/learning strategies should embrace the development of systemic skills, i.e. students should be able to (i) manage the information flow by filtering the most relevant information and verifying the authenticity of the retrieved information, (ii) build networks to stay updated and informed, (iii) share and discuss information inside online social networks,...

Bearing this in mind, the HE teacher will also have to develop new competences, such as to be able to: (i) act according to the research made on the curricular development and on the teaching/learning process, (ii) conceive and develop the course's curriculum, taking into account not only the European directives, but also the constant ICT mutations, (iii) develop an analytical-reflexive evaluation attitude towards the created expectations and the observed reality, (iv) share and discuss experiences with peers, so that best practices can be unveiled and (vi) monitor the teaching process - not leaving aside the fact that, in this process, the teachers would also be looking for ways of enhancing their pedagogical training in terms of lifelong learning [5], [6]. According to Shulman [5], teachers, while designing and teaching a module, should not only reveal the investigative spirit that characterizes them as researchers, but also make their activity public, i.e. teachers should be receptive to their peers' criticisms and evaluation.

Lady using a tablet
Lady using a tablet

Comprehensive

Writing Services

Lady Using Tablet

Plagiarism-free
Always on Time

Marked to Standard

Order Now

According to the European Commission [7], even though the evaluation of teaching is not a new concern, when it comes to the exploitation of the potentialities of ICT, its specificities are only recently being studied. Therefore, it is very important to design and implement teaching and learning strategies, as well as to plan the evaluation/assessment process, according to the competences and learning outcomes defined for the module.

In Portugal, the educational matrix in HEI was mainly based on relatively conservative models. Nevertheless, as a consequence of the Bologna Declaration, people resort more and more to eLearning or bLearning. According to the APDSI [8], the use of multimedia, combined with the use of the Internet, to design and make educational contents available and to develop competences at a distance (eLearning), is one of the greatest challenges of the coming years. Consequently, resorting to eLearning becomes a great challenge within the Portuguese HE context. Such a challenge encompasses not only the curricular integration of ICT but also research on evaluation and on the exploitation of ICT tools for evaluation and assessment. It is in this context, and bearing in mind the abovementioned aspects that the module characterised in the following section was designed.

3. ESE STRUCTURE and development

Educational Software Evaluation (ESE) is offered in a blended-Learning mode and it is part of the Master's Course in Multimedia in Education at the University of Aveiro, created in 2002/03. The first two authors of this paper are peer reviewers, contributing actively for the evaluation of the ESE module. The second author is a teacher of this Master's Course and also its Director. The other two authors are the tutor and the teacher of the ESE module.

ESE's main goal is that students develop evaluation competences, particularly in educational software evaluation. Considering the learning outcomes defined for the module, the purpose is to lead students to develop an educational software evaluation project, in which they have to define and describe: (i) the evaluation object and the purpose of the study, (ii) the nature of the study, (iii) the evaluation goals, (iv) the participants, (v) the criteria and indicators, (vi) the data gathering strategies and tools, (vii) the data collection itself, (viii) the analysis of the data and (ix) the conclusions reached.

Bearing in mind that we live in an Information Society, apart from the competences needed to face the challenges this raises, the ESE module also aims at promoting the development of systemic competences, such as (i) critical use of ICT in educational contexts (blogs, concept maps...), (ii) collaboration, (iii) research, (iv) information search, organization and treatment, (v) development and evaluation of work/education plans, among others.

The ESE module covered the time span of five weeks - January and February 2007, and it had two face-to-face sessions (twelve hours each). The proposed activities and tasks were conducted as follows.

Prior to the start of the module, the teacher and the tutor prepared the documents and the activities related to the module itself and made them available in the LMS Blackboard. This eLearning platform was also used to deliver information on bibliography, to discuss the tasks, clarify doubts and evaluate the module (a specific forum was created for that purpose). Besides the LMS, a blog was also created for this particular subject and it aimed at supervising the students' work, sharing information, discussing topics related with the tasks and the tasks themselves (http://ase07.blogspot.com).The diagnosis of initial students' understanding of the subject as well as the reflection carried out during the module was undertaken using the blog.

When the module started, structural documents were made available online: (i) the module guidelines, which included the goals of the module, the tasks that students were supposed to work on, as well as to general and specific competences they should develop and (ii) the task schedule - which was discussed with the students and, therefore, re-arranged (during the 2nd week).

The first task (T1) was designed to lead students to reflect upon the theoretical concepts of the module. The second task (T2) involved the analysis of a selection of research papers that were made available in Blackboard and students were asked to summarize at least one of them. They could also search for different papers, and in fact this is what happened in some cases. Students were then invited to organize themselves in groups and to prepare a presentation of the summarized texts and a first version of a concept map establishing in it the relationships found in the analyzed information (T3). Part of this work was prepared during the first face-to-face session and then shared and discussed. Students were asked to improve it by taking into account the teacher's and their peers' feedback. After this session, students were supposed to write a post, in which they would give their personal opinion about the tasks that they had already completed, as well as about their learning: what they had learned, how, etc. (T4).

Lady using a tablet
Lady using a tablet

This Essay is

a Student's Work

Lady Using Tablet

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

Examples of our work

The next task to accomplish, the one with a greater workload, was to select a theme for the group project, to be developed until the end of the module (T5). The guidelines given only mentioned that the project had to be related to software evaluation. In the module guidelines they also had guidelines concerning competences to be developed. The themes and the projects' organization were discussed using the module blog, so as to avoid the emergence of similar projects and thereby encourage a wider scope of subject matter. A preliminary version of the project plan had to be submitted by the end of the 2nd week (T6). A first version of the projects' report was shared and also discussed two weeks later (T7).

The projects were presented orally during the 2nd face-to-face session and a wide discussion followed each group presentation (T8). During this session several assessment tasks where completed by the students. After each presentation, the groups assessed the project, according to the discussed criteria (e.g. innovation and creativity, the use of media, etc.). Each student also made his/her self-assessment of the competences developed during the module (third reflexive task) and evaluated the other group members, aiming at the self- and peer assessment of the collaborative competences developed (T10). During the module, students were also encouraged to make final adjustments to the concept map (T9).

As it is implicit in the module's presentation, assessment included formative (for instance, during the discussions of ongoing work) and summative components. In terms of percentage and of the elements involved, the assessment framework agreed upon with the students was as follows: collaborative work (students) - 5%, self- and peer evaluation (students and teacher) - 20%, project work and oral presentation (students and teacher) - 40%, concept map (team work) - 20% and final considerations - 15%.

4. Empirical study

4.1. Methodology

The main aim of this study is to evaluate teaching delivered during the module described above. Therefore, and given the lack of evaluation conceptual models in the area of the evaluation of online teaching/learning [1], this work has an exploratory and descriptive nature. The evaluation framework focuses on different vectors, such as the assessment of the learning process and the learning products, as well as the students' self- and peer assessment of the collaborative competences developed and of the group work produced.

The evaluation process encompassed two different moments. As to the first, it took place at the end of the ESE module, i.e. the students' comments on the teaching/learning process, collected through T10 (see Table 1.). Once there were 21 students (out of 28) that actually wrote their final considerations, these were selected as the analysis units. To analyze the data, categories and indicators were defined, resorting to the software NUD*DIST (Non-numerical Unstructured Data Indexing, Searching and Theorizing).

The second evaluation moment happened a year later and the data analyzed were collected through an online questionnaire (http://wsl2.cemed.ua.pt/ ase). It aimed at gathering data on the students' perceptions concerning: (i) the module, (ii) the teacher performance, (iii) the proposed tasks and the assessment; and (iv) the communication tools explored. Moreover, this questionnaire only comprised closed answers, i.e. two scales were used - a 1 to 10 scale and a scale in which they stated their agreement level (completely disagree, disagree, agree, completely agree or don't know) -, depending on the issue. It was anonymous and from the 28 students enrolled in the module, a total of 19 (68%) responses were submitted and analyzed. These data were analyzed using quantitative techniques, namely descriptive statistics, and for that purpose the software Microsoft Excel was used.

It is believed that the time gap (a year) that separated the answers to the questionnaire from the students' final considerations allowed them to distance themselves from what they experienced during the module. In the next section, the students' perceptions on both evaluation moments will be compared.

4.2. Data analysis

As mentioned, regarding the students' final considerations (2nd evaluation moment), they did not have any worksheet to give the topics they were supposed to cover. Bearing this in mind, it looks reasonable to state that, students tended to focus on the most important issues concerning their own learning during the module and not on every single negative and positive experience.

The data were analyzed and compared according to different perspectives, such as follows.

In the end, just taking into account the comments that students wrote by the end of the module, one can say that the positive aspects referred (72 statements) were a lot more than the negative aspects mentioned (19 statements), and that out of the 21 final comments analyzed, only 15 students pointed out negative aspects and 19 students underlined positive aspects.

5. Final considerations

Just like [9:1] refer, "effective teaching evaluation is to collect data from multiple sources (triangulation), making sure that all education-related activities are rated by the people best qualified to rate them." Therefore, since the participants involved in the teaching/learning process are, on the one hand, the teacher and the tutor and, on the other hand, the students, at least both points of view should be taken into account when evaluating the teaching process. Some authors (e.g. [9], [10], [11]) present a teaching performance evaluation framework, in which it is valued, as data source, the students' rates, the peers' rates and the teacher's and tutor's discussions and self-rates. These were also the data sources used in this study. The data regarding the students' rates were collected through the already mentioned questionnaire and the students' final comments. As to the peers' rates, some teachers were invited to evaluate the module - namely, two of the co-authors of this paper - and their perceptions concerning the teaching strategies used in the ESE module were discussed with the teacher and the tutor of the module (the other two authors). This data triangulation provided a means to check (evaluation process) and to improve the teaching/learning process (using the evaluation results). Like Coughlan [10] and Kean [11] claim, a cycle of evaluation and improvement based on student feedback is a fundamental component of the process of quality improvement in HE.

The teaching strategies used in the ESE module revealed some pros and cons. Particularly because ESE is a bLearning module, it was very important that the activities/tasks were online and timetabled within the module. The teacher, however, chose to negotiate the proposed activities/tasks and their timetables with the students during the module and this was not well-accepted by some students. The fact is that Portuguese students are not (culturally) used to negotiate with teachers and, therefore, even though at first they are enthusiastic about this (because it is new), they also feel more comfortable with the idea of being the teacher the one to establish "the rules" from the beginning. Nevertheless, so that the students feel more secure with the negotiation process, it is advisable to restrict this negotiation and to promptly update the information available online, especially in a teaching context that is very limited in time (a month) and in which the face-to-face sessions only happen in the beginning and at the end of the module, being the remaining contact done online.

Regarding the online interaction, the students felt, once more, the need of being guided. In the ESE module, some students believed that the teacher should have given them feedback more frequently and should have been more directive. Nonetheless, most of these students admitted they had developed a systemic competence that was not written in the module's guidelines (but the teacher though it was very important), i.e. to be able to research and develop a project work autonomously.

One other conclusion is that, apart from face-to-face sessions (where there was a high level of interaction between groups), in online contexts, collaboration happened essentially inside each group, i.e. collaboration between the groups was scarce. Consequently, one suggestion to consider in future versions of the module is, for instance, to ask students to comment on and assess the work done by, at least, two other groups. With this strategy, it is believed that students would not only develop a deeper knowledge of each other's work, but also be more collaborative with the other groups. Another suggestion to increase interaction between groups is to create a blog to be used by the whole class, where students share and discuss information.

In conclusion, in this study, the organization of a bLearning module (ESE) and an evaluation of the teaching/learning process used were presented. The main concern was to establish and test an evaluation framework to evaluate bLearning modules in terms of teaching strategies and, from the data analysis, it is believed that this framework is valid and could be applied in the context of similar modules.

References

  1. ENQA - European Association for Quality Assurance in Higher Education (2005). Standards and Guidelines for Quality Assurance in the European Higher Education Area. Available at http://www.bologna-bergen2005.no/Docs/00-Main_doc/050221_ENQA_report.pdf (retrieved 08.11.2007).
  2. Pombo, L. et al. (2008). Teaching and evaluation strategies in online contexts - the case study of Educational Software Evaluation, in M. Muñoz, I. et al. (Org.), Proceedings of the IASK (International Association for the Scientific Knowledge) International Conference "Teaching and Learning 2008", Aveiro, Portugal, 26-28 May, pp. 148-155.
  3. Siemens, George (2006). Knowing Knowledge. Available at www.knowingknowledge.com (retrieved 06.05.2008).
  4. Redecker, C. (2008). Review of Learning 2.0 Practices. Deliverable 2 of the study: Learning 2.0 - The Impact of Web 2.0 Innovations on Education and Training in Europe, Institute for Prospective Technological Studies, European commissions.
  5. Shulman, L. (2004). Teaching as community property: Essays on Higher Education. San Francisco: Jossey-Bass, 272 p..
  6. Alarcão, I. & Gil, V. (2004). Teaching and learning in Higher Education in Portugal: an overview of studies in ICHED, in Challenges in teaching and learning in higher education, V. Gil, I. Alarcão & H. Hooghoff (eds.), Aveiro: Universidade de Aveiro, pp. 195-214.
  7. European Commission (EC) - Directorate-General for Education and Culture (2006). TELL: Towards Effective network supported collaborative learning activities. Introducing a Framework for the Evaluation of Network Supported Collaborative Learning. WP1 Deliverable, Project number: EAC/61/03/GR009, ELearning Initiative, 2006.
  8. APDSI - Association for the Promotion and Development of the Information Society (2007), e-Inclusão ­- Um desafio para Portugal. Available at http://www.apdsi.pt (retrieved 02.09.2008).
  9. Felder, R. M. & Brent, R. (2004). How to evaluate teaching, in Chemical Engineering Education, vol. 38, no. 3, pp. 200-202.
  10. Keane, E. & Labhrainn, I. M. (2005). Obtaining student feedback on teaching & course quality. Briefing paper, 2. Available at www.ucg.ie/administration_services (retrieved 23.10 2007).
  11. Coughlan, A. (2004). Evaluating the learning experience: the case for a student feedback system Quality Promotion Office, National University of Ireland: Maynooth.

--------------------------------------------------------------------------------

  1. Sections 2. and 3. were already presented in [2].
  2. Examples of project ideas: (i) selection and evaluation of a software or site that could be used for educational purposes. The criteria as well as the process should be grounded on the literature; (ii) development of tools to evaluate educational software either by teachers, experts or students; (iii) analyses of the educational possibilities of web 2.0 tools.