"Evaluation is a way of understanding the effects of our teaching on student's learning. It implies collecting information about our work, interpreting the information and making judgements about what actions we should take to improve practice.....
Evaluation is an analytical process that is intrinsic to good teaching."
Initially the very idea of evaluation raised eyebrows but now it is widely seen as a step towards accountability and also a part of professional good practice and the systematic development of teaching expertise ( Falk and Dow, 1971). Evaluation is closely linked to learning, allowing those involved to reflect and develop personal and professional practice. This developmental role of evaluation is highlighted in Quality Assurance Agency (2006) documentation and the Strategy for the Enhancement of the Quality of the Learning and Teaching and Assessment (2007-2011).
Evaluation can be done in two ways namely summative or formative. Summative evaluation occurs at the end of a process and is focused on the outcomes (Boulmetis and Dutwin, 2000). In the university it is related to exams at the end of term, submitting course work etc. Summative assessment is useful in determining whether the course, module or the person who is being evaluated meet the stated learning outcomes (Scriven, 1967).
"When the cook tastes the soup, that's formative; when the guests taste the soup, that's summative" Bob Stake, quoted in Scriven, 1991, p. 169
On the other hand Formative evaluation is concerned with fostering development and improvement within an activity, person, module or course (Scriven, 1967), both these evaluation technique shave a common thread running through in the form of quality assurance, enhancement , effectiveness and accountability.
How is evaluation at UWS:
At University of the West of Scotland and in the Business School department evaluation is moderated by various agencies. Externally it is the Quality Assurance Agency (QAA, 2006) who carries out reviews on higher Education establishments, providing support, advice, guidance and recommendation. The data for review is normally collected by standard module evaluation questionnaires, feedback forms and sometimes from focus groups.
Internally the UWS Strategy for Enhancement of the Quality of the Learning and Teaching in Higher Education (SEQLTA) assures academic standards are maintained and it formulates paths for necessary continual improvement of learning , teaching and assessment (UWS, 2007). Annual monitoring is facilitated by the Quality and Enhancement unit. Audits, Module evaluations also occur on a regular basis.
At the university student participation is encouraged by informing he students to attend the Subject Development Group (SDG) meetings, Student Staff Liaison Group meetings, and completion of Student Experience Questionnaires. Quality of Student Experience Reports (QSER) are also written and collated to populate the school annual report.
How to choose a particular Evaluation Method
Choosing evaluation methods depends on 'what it? is that you hope to obtain' and then determine the best methodology to obtain the information (Jarvinen, 2000). So when choosing a methods we should consider who is going to evaluate, the financial, time commitments involved, the size of the group to be evaluated, is it going to be formative or summative. Taking into consideration the above points a method can be chosen. Methods can be questionnaires, focus groups etc. The information should then be shared with students to keep them in the loop and also to help them improve their own learning strategies and become more successful self-directed learners and feel proud of what they have achieved. Another important aspect would be to look at what kind of evaluation it would be i.e. will it be one to one, individual or group process.
Individual self evaluation may involve reflection, carrying out a SWOT analysis on oneself, undergoing a Personal Development Review, self appraisal or developing a portfolio/eportfolio. Reflective questions can help as a starting point (Day, Grant and Hounsell, 1998). One-to one may involve taking part in a review, co-delivering a teaching session with a peer, taking part in supervision. Dialogue discussion and reflection are key to the success of this, encouraging continuous engagement and fostering a deep approach to learning. Group evaluation may involve questionnaires, focus groups, the one minute paper etc.
Critical Discussion - One Minute Paper
Angelo and Cross (1993 ) first described the 'One Minute Paper' in their book 'Classroom Assessment Techniques'. This was designed to provide instructors anonymous feedback on what was learnt by the students in the class.
OMP or One Minute Paper is a writing activity that can be done in class in a minute. This is in response to the instructor posed question, which reflects what has been taught in that day's lecture. The original purpose is to assess what the student had learnt in the lecture that day. Evaluation such as this can take place as a continuous or one off event to encourage active learning (JISC, 2006)
I first came across this concept in the 3 -day Teaching and Learning Introductory course organised by University of the West of Scotland in conjunction with University of Strathclyde.
I then felt, it was an innovative fresh approach to bring back wandering student's concentration and really reflect upon what was discussed in the lecture that day and answer the question in a minute. From then I have become a great fan of this technique and very frequently use this assessment technique in my lectures. It is a "simple and low technology" designed to gather feedback from students on a regular basis. (Chizmar & Ostrosky, 1998, 3) I mainly use it as a reflection strategy designed to help students discover their own understanding of a concept and to build student lecturer rapport.
In the following sections, I have looked up the kinds of OMP questions I had used, when I had used them and the benefits I felt from them.
Lecturer takes two to three minutes of class, normally towards the end and asks students to respond to some variation of two questions:
1) What important thing have you learned? And
2) What question(s) do you still have at this point? (Angelo & Cross, 1993).
This on-going feedback provides faculty with data to assess student learning, and it increases student involvement in the learning process.
Once students have responded to the Minute Paper, a simple analysis of the responses can provide the lecturer with valuable information. It can provide the lecturer with valuable information of whether the students understand the major points of a lecture, and it can also identify problem topics. On this information, the lecturer can then take a few minutes in the next class to review the results, providing praise when students have mastered a difficult concept or reviewing material when problems still exist.
The limited literature available indicates a range of using the Minute Paper four or five times per term (Wilson, 1986) to every class period (Chizmar & Ostrosky, 1998). I use it simply to get feedback, to check if the students are understanding the topic, to bring back their concentration and to keep them interested/motivated.
I have used OMP at the beginning of the class with some anticipatory questions to activate students' prior knowledge on the subject:
What do you think is the best marketing tool ever used?
What is the most advanced sales technique?
I have used OMP at the end of the class (usually) to help students reflect on what they have learnt today and think more deeply about the concept. It also brings helps any students who do not want to draw any attention to clarify their points anonymously.
Question used include:
What is the major point you learnt in today's class?
What example used today can you related to most and why?
One minute paper can also be used in the middle of the lecture. I use this to discuss a key or vital point of the day's lecture. I believe this midpoint interruption makes the student more alert; it intercepts the natural attention drift that takes place after perceiving information for some period of time. Research indicates that student attention and comprehension are strengthened by short pauses that encourage mental activity in the middle of class presentations-for example "Tear out half a sheet of paper and write your reaction to the presentation thus far" (Bligh, 2000).
Think of another example that fits in this scenario.
How can you sell to an in store customer a new product?
Advantages of One minute Paper.
I found out that there are some topics which the students understand better than others using the OMP.
It is quite good at targeting students needing specific help i.e. they may have come from a different subject background.
Students are more willing to ask questions and engage in a dialogue discussion in the following class.
It bridges the gap between two successive lecturers, when the students reflect on what happened last week.
Improves the quality of the class discussion. All students are involved and there is no opting out. Encourages equal participation, so a shy student may not feel left out.
OMP can promote class attendance, effectiveness, and the students feel involved.
They can be used as a learning journal.
Minute papers have encouraged me to think more carefully about how to prioritize course content and to identify "core" concepts that I want students to examine deeply.
The disadvantage is that it takes time both during and after the class.
People also give what they consider to be socially acceptable answers ( Polit and Hungler, 1991).
Selective observation and selective recording may also occur (Johnson, 1997).
Critical Discussion - Module Evaluation using the Module Evaluation Questionnaires:
The need for greater accountability and improvement in the quality of teaching has become a major issue in higher education in recent years (Coaldrake & Stedman, 1998, Ramsden 1991). Growing numbers of academic staff have begun to see feedback as an integral element in any effective teaching strategy, and therefore essential in its own right, whatever the external pressures might be (Hounsell & Tait & Day, 1997).
Feedback is key to becoming an independent learner (Orrell, 2006). Effective feedback has an emotional and intellectual impact which stimulates and empowers the receiver (GEES, 2010). So student's evaluations are carried out towards the end of each module and feedback obtained from these is discussed at module evaluation meeting attended by those involved in delivering the module. In the University of the West of Scotland, Module evaluation questionnaires (see appendix) are used to gather module feedback.
Module Evaluation Questionnaires at UWS
The University places great importance on student feedback to enable them to monitor and enhance the modules in courses and one of the ways in which the university obtains feedback is via the completion of Module Evaluation Questionnaires (MEQs) at the end of each module.Â Â MEQs give students a chance to comment upon the modules they have completed and inform staff about where improvements could be made in the areas of delivery, learning, resources and assessment.
These MEQs are given to the students in week 10/11 to be completed and which are then analysed by the Quality Enhancement Unit, where the information is analysed and the results in form of graphs, reports are then submitted to the heads of the department. At the Module Meetings, these reports are discussed and any changes noted will be implemented accordingly.
Why use Questionnaires:
Questionnaires might be used to obtain information and views, or to attempt to justify/quantify impressions. So the University of West of Scotland would like to get information on some of the following:
statistical information, perhaps to meet an external requirement, or to inform e.g. admissions policy
'research' - finding out what students think or do, possibly for academic publication, but not intending change at present
'feedback' - to help you change the way you do things
To change student's perceptions by making them be aware of the learning/teaching process.
Advantages of using questionnaires:
It helps in providing a useful method of obtaining information in a structured format.
By respecting privacy of individual views and
Not requiring the direct intervention of an interviewer
They have the disadvantage of being a substitute for genuine dialogue, with a limited scope in both questioning and responding, although this is overcome somewhat by the use of open questions.
Questionnaires with standard questions facilitate comparison of the student's experience over a period of time across all levels. They are appropriate when feedback has to be obtained from a large group of students. But these questionnaires should be well designed. Again at the University, each department has its own targets to achieve and the questionnaires are designed in such a way that this information is gathered, also they are relatively economical to administer, summarize and interpret. However, if they are time consuming to analyse any changes that will be implemented because of the results might come too late.
Questionnaires can be valid, if the questions in them are the ones the students can directly comment or have experienced it. If the University ask questions that they cannot relate to, the information obtained would be useless and it will not help in the feedback process. The questionnaires with closed questions will limit what the student can inform us, but other concern areas might not be highlighted. On the other hand open ended questions will be more difficult to analyse but often provide useful information and provide suggestions for making changes. Open-ended questions can be very illuminating, but may be best asked after some set-response questions which firstly deal with predictable, routine comments, and secondly may clarify for the student what they wish to say. To reduce the time and effort for the student, and produce constructive feedback, questions like 'Suggest one feature of this module that could be improved', and 'Which topic did you find most difficult?' can help.
In general, questionnaires should always ask students what aspects of the module are going well for them, what aspects cause problems, and what suggestions they have for changes that would improve their experience. The simplest questionnaire asks only that.
In Appendix 1, I was given a report from my module co-ordinator after I had submitted the exam marking; by then the MEQ's results were not available. I was very happy when I read the report.
Appendix II, after implementing the One - Minute-paper, I had some interesting questions.
The questions/suggestions/answers includes:
How are traditional marketing methods different from the modern marketing methods.
Marketing plan has advantages and disadvantages, how we choose the best plan to adopt.
Would like more examples to be included and also to be posted on Blackboard.
Where will marketing plan fit in the whole picture of an organisation's goals and objectives.
Can we refer to other definitions of marketing planning by other authors?
When did modern marketing methods gained importance.
As part of recommendation strategy, the limited literature available indicates a range of using the Minute Paper four or five times per term (Wilson, 1986) to every class period (Chizmar & Ostrosky, 1998), which is what I would recommend and do the same for my lecturer's.
Module Evaluation questionnaires: I would recommend them to be used halfway through the term and at the end of the term. The first questionnaire would cover the import points on which information is required like learning and teaching and resources. And at then end to concentrate also on the assessment techniques.
Important thing we should at is how valid can these evaluation techniques are, whether the information obtained is plausible, credible, trustworthy and defensible (Johnson , 1997) . Triangulation can enhance validity, which involves a combination of two or more theories, data sources, methods or investigators to enhance validity (Denzin and Lincoln, 1994). So if the students participated in a combination of questionnaire and a focus group this would increase the validity of the findings although there are obvious resource implications with this.
Timing of evaluation:
Evaluation conditions (are students tired, fatigued, confused, bored) as well as self awareness may impact on validity. A suitable timeframe is to be included in the student handbook or posted on Blackboard when an evaluation can take place (MEQ). So that the student is well prepared. Importance should also be given to the timing. As if it is too early in the process the students may not have the knowledge and if it is too late reflection may be inaccurate (LTDI, 1998).
In relation to reliability, summative evaluation should be delivered across all campuses on one day and at the same time. Consistency and transparency can be maintained this way. If it was done on different days though, the same person e.g. the module co-ordinator could seek feedback in the same way from all students, i.e. he has to be physically present in all the campuses on different dates to take charge. Explanation of why evaluation is being done, how important it is, is also important during formative evaluation as it can help increase organisational awareness and understand how an organisation is influenced by external circumstances (GEES, 2006).
Finally the findings and recommendations are to be made available to all of them involved and not just the management and lecturers. Not many students take part in SDG meetings. Incentives can be provide to students like free access to printing worth £ 5, in the campus, lecturers can remind the students that the SDG meeting are taking place. And the minutes of these meetings can be posted on the general page of the Blackboard for all the students to realise the importance in completing the questionnaire.
Formative and Summative evaluation can assist in professional and personal development. Having an understanding of frameworks and policy supporting evaluation is essential for individual planning to take part in evaluation (as participants). Any evaluation should be integral to teaching, allow for modifications, be participatory and enable those participating to draw out wider implications. There are a variety of skills involved when considering issues relating to validity, reliability of the evaluation. These include the ability to be consistent over time and place, the ability to engage participants in a professional manner and the ability to demonstrate knowledge and awareness.