The Department Of Arts At Hail University Education Essay

Published: Last Edited:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

With the technological advances, studies on CBF in writing have expanded lately El ebyary Windeatt, 2010; Denton et al, 2008; Huang, 1995; Li, 2002; Warschauer Ware, 2006 and have exerted an increasing influence on the writing instruction. However, to the best of my knowledge, no research has been conducted on the effect of using CBF along with TBF inside the classroom while looking at the nature of comments that CBF provides and how teachers can integrate CBF into their own classroom, the perceptions of students about the use of CBF before and after its implementation and the results achieved with such use. This project will try to fill this gap in the research.

The overall aim of the project is to investigate how far CBF can be integrated in a Saudi University-level ESL writing classroom. First, it will go over the similarities or differences between the feedback given by the computer (CBF) and by the teacher (TBF). Then, the study will investigate students' perceptions of CBF. Later, it will go over the students' writing performance change during the course. Finally, it will try to understand how CBF is integrated in the classroom.

This project involves both quantitative and qualitative methods. The study will investigate the similarities and differences between TBF and CBF through analysis of students' written text. Also, the experimental group students will be asked, in a pre- and post-questionnaire about their perceptions towards a range of issues all related to feedback in an ESL writing session, particularly towards teacher and computer based feedback (TBF & CBF). The research will also use a pre- and post-tests to measure the effects of introducing CBF to an experimental group of students, as opposed to a control group whose members are exposed only to teacher feedback to see if the performance will differ as a result of the type of feedback students will receive. At the end semi-structured individual interviews with randomly selected members of the experiment group will be conducted to complement the findings.

3.1.2 Research Main and Sub- Questions:

The overall aim of the project is to investigate how far CBF can be integrated in a Saudi University-level ESL writing classroom. The research main and sub- questions are:

How far is CBF similar to/ different from TBF?

What are the nature of comments does CBF provides?

What are the nature of comments does TBF provides?

What are the students saying about the differences between CBF and TBF?

What are students' perceptions of CBF and TBF?

What are the students' initial perception of CBF and TBF?

What are the students' final perception of CBF and TBF?

To what extent does the students' writing performance change during the course?

Does the writing of students who receive just TBF improve?

Does the writing of students who receive CBF and TBF improve?

Does the writing of students who receive CBF and TBF improve more than that of students who receive just TBF?

How CBF is integrated in the classroom?

How was the experience of integrating CBF in the classroom?

What are the advantages of this integration?

What are the disadvantages of this integration?

A summary of the research main and sub- questions and data collection methods are summarized in table 3.1.

Thesis Main Questions

Research Sub-Questions

Data Collection Methods

1-How far is CBF similar to/ different from TBF?

What are the nature of comments does CBF provides?

What are the nature of comments does TBF provides?

What are the students saying about the differences between CBF and TBF?

-Analysis of written texts

2-What are students' perceptions of CBF and TBF?

What are the students' initial perception of CBF and TBF?

What are the students' final perception of CBF and TBF?

-Pre- and Post-Questionnaire


3-To what extent does the students' writing performance change during the course?

Does the writing of students who receive just TBF improve?

Does the writing of students who receive CBF and TBF improve?

Does the writing of students who receive CBF and TBF improve more than that of students who receive just TBF?

-Pre and post-tests

-Analysis of written texts

4-How CBF is integrated in the classroom?

How was the experience of integrating CBF in the classroom?

What are the advantages of this integration?

What are the disadvantages of this integration?

-Field notes

-Pre- and Post-Questionnaire


Table 3.1 Research Questions, Sub-questions and Data Summary

3.2 The Context of the Study

This part of the methodology is about the research population that is actually involved in the study. This part contains detailed descriptions of the research location and participants of the study.

3.2.1 The Department of Arts at Hail University

Hail is a small size city in the northern region of Saudi Arabia, which has many villages scattered around the city with a population of just over 600,000 (Sector of statistics and information in the Kingdom of Saudi Arabia, 2010). The city has one university which is Hail University and it is a relatively a new one since it was officially established in 14 June 2006. The University used to be a group of colleges scattered among the city of Hail. The colleges have a longer history in Hail, some of which started in the early fifties, but recently they were grouped under a university roof in 2006. The university grouped different colleges and started to expand since then. The university is building a huge campus in the north of the city to bring together all of its colleges in one location. The new site covers an area of over 9,000,000 m². Some colleges have already moved into the new site, some are in the process of moving and others are still waiting for their buildings to be completed. The Department of Arts is one of the colleges which are in the process of moving at the time of this study and is located in a temporary place near the university main site.

University level students study English for six years in secondary and high school. In spite of studying English for a long time, few students achieve adequate results when they join a Saudi university (Alhazmi, 1998; Grami, 2004). Therefore, at Hail University in the English Arts Department students start with Basic English courses and move to higher ones each term in their five years of university study. For writing and composition, the Department requires all students to successfully complete four compulsory courses in writing. The textbooks normally used for teaching the two introductory writing courses are the international book versions of Mosaic I and II by Blass and Pike-Baky published by McGraw-Hill companies. Students in the first term study courses that teach basic writing with correct grammar and learn how to write paragraphs. In the students' second term, they take a subject called ENG213 Composition, the course provides students practice in producing a variety of grammatically correct sentences in unified paragraphs that are patterned logically. Topic sentence use in all writing is stressed as well as adequate development of ideas. Emphasis is given to both the writing of cohesive summaries and explanatory pieces on different topics. Methods of classifications are also introduced in the course.

In the students' second year or third term, they take a writing subject called ENG312 Essay Writing. This is the course that this study was done on. In this course, students practice producing longer and substantial essays of several paragraphs. Attention is given to the process of developing formal argumentative essays. Emphasis is given to rules of evidence and the methods of presenting it to support the points of view used. Moreover, grammar is emphasized throughout the course. Students are trained to write complete essays with an introduction, body and conclusion. The text book assigned for this course is Mosaic II by Blass and Pike-Baky published by McGraw-Hill companies. When students successfully finish the Essay Writing course they move to more advanced writing and various English classes in their third, fourth and fifth year.

The study was done during the 2011-12 first term which started on 21 September 2011 until 15 January 2012. The Department of Arts have just moved to a new location and things were not completely settled down at this term. Many classes were not fully equipped, especially the computer lab which the study was meant to be done in. The computer lab was not fully operational until the start of November in 2011. This delayed the study but did not obstruct it since there was sufficient time left for the lab sessions. The computer lab sessions started from the start of November until mid-January.

The reason behind choosing Hail University for this study is based on that the researcher is originally from the same university and has easy access to teaching at Hail University than any other place. Never the less, a written formal request was sent to Hail University prior to commencing the study to ask for permission to conduct the study there. An approval letter was sent back from the Dean of Arts in the university giving permission to commence the study in the first semester of the 2011/2012 academic year.

3.2.2 Participants of the Study

Two Saudi ESL classes at a second year university writing course with a beginner's to intermediate level of ESL writing will be involved in this project. Both classes are at the same level of study and both are studying the same subject chosen for this research which is ENG312 Essay Writing. At least one writing course of University level has been completed for the participants.

The overall number of participants in the two classes is fifty students (n=50) and are all male students, because universities in Saudi Arabia are either all male or all female universities. The control group has twenty seven students (n=27) while the experimental group has twenty three students (n=23). The students' numbers in the two groups were not controlled by the researcher because each student decides on which section to join, the 8am or 10am class, according to his preference or timetable conflicts, but pre-tests were made to examine the students' initial level. Their ages vary between 19 to 23 years-old.

The two classes study Essay Writing for almost two hours a week on Wednesday mornings. The term started on 21 September 2011 until 15 January 2012. The first class starts at 8 am and finishes at 10 am, while the second class starts at 10 am until 12 pm. The first class was chosen on random to be the experimental group and the second was the control group. The computer lessons were done in a computer lab and normal pen and paper lessons were done in a typical classroom with a board in front of the students and each student sat in an individual chair with an attached small writing board.

The research study and the students' role in it were fully explained to the students prior to commencing. Students were given a translated version of the consent form (Appendix I) explaining the study and their rights. All the students involved in this research have signed the consent form.

3.2.3 The Teacher/Researcher Role

The Teacher in this project is the researcher and is going to teach the two classes. The teacher/researcher is an MA holder and a doctoral candidate in a PhD program with a focus on computer assisted language learning and teaching. He has been involved in ESL-related pedagogy and research for over 5 years. Before commencing his PhD study, he has been teaching English at Hail University as an English lecturer with an MA degree.

Contrary to quantitative researchers' intention to "contaminate" the data as little as possible when collecting data, researchers conducting any form of qualitative inquiry usually serve as the instrument to collect data (Denzin & Lincoln, 2003; Lincoln & Guba, 1985). With specific focus on researchers' roles during the processes of study design, data collection and analysis, the report writing as well as information distribution, Stake (1995) suggests that a researcher should take multiple roles such as teacher, advocate, evaluator, biographer and interpreter. In the current study, the teacher is the researcher and data were collected from the class the researcher taught. Thus, the researcher played dual roles in the field, both the researcher and teacher. The researcher collected and analysed data from his students during and after delivering instruction and organizing the class. Some of the advantages of being a researcher/teacher in the study include: the researcher had a deeper understanding of the environment, the participants, and what occurred during the class while teaching the class, which benefited the later data analysis and interpretation; the data collection was less noticeable; logistically, it simplified the data collection process in terms of setting up time and place. During the data analysis, the researcher/teacher played a crucial role in terms of collecting and analysing data.

However, the dual roles the researcher took also may cause special weaknesses that the researcher needs to understand and try to avoid. For example, the researcher might not have sufficient time to take field notes due to his duties of being the teacher, e.g. walking around to provide necessary language or technical aids. In addition, as cautioned by many qualitative researchers (e.g. Denzon & Lincoln, 2003; Lincoln & Guba, 1985), conducting research while actively participating in the event under investigation for a prolonged period of time may obscure the researcher's view of important issues emerging in the field or even produce potential biases (Yin, 2003). In other words, being the teacher simultaneously might prevent the researcher from identifying some significant themes or patterns in students' behaviours.

3.3 Fieldwork and Empirical Study

3.3.1 Quasi-Experiment: Control and Experimental Groups

Gall et al. (1996), Mackey et al. (2005), Bryman (2012) and Cohen et al. (2011) highlight a number of issues involved in dealing with the inclusion of an experiment and control groups in a quasi-experimental design. The participants are subjected to different treatment conditions and thus should not be treated equally. The treatment group is likely to receive special training, while the control group receives either nothing or a conventional programme. In this research project, the experiment group will be trained to adopt the relatively new CBF technique in their writing sessions, using Criterion, while the control group will receive normal teaching sessions with TBF.

As a matter of fact, quasi-experimental research cannot fully control confounding variables because it does not fully control assignment of the subjects to a treatment (Ary, Jacobs, & Razavieh, 2002; Johnson & Christensen, 2004; Bryman, 2008; Cohen et al, 2011). In other words, this design is similar to the true experimental design except that research subjects are not randomly assigned to the control and experimental groups not intentionally but because of circumstances beyond control. The quasi-experimental design adopted in this study, pre-test and post-test with non-equivalent groups, will be outlined and discussed in the next section.

3.3.2 Pre-test and Post-test Design with Non-equivalent Groups

This design is probably the most widely used quasi-experimental design in educational studies and this design is as follows (Gal, Gall & Borg, 2005; Bryman, 2008; Cohen et al, 2011):

O1 X O2


O1 O2

O1 Both the experimental and control groups have been given a pre-test.

X The experimental group has been given the experimental treatment.

--- The broken line indicates that the experimental and control groups have not been formed randomly.

O2 Both the experimental and control groups have been given a post-test that measures the dependent variable.

Best and Kahn (2003) suggest that "a quasi-experimental design is justifiable in classroom experiments involving naturally assembled groups as intact classes." They argue that statistical analysis can be used with the pre-test to account for any group differences. However, they emphasise the importance of interpreting results cautiously and careful analysis of comparability of the educational backgrounds of participants. Similarly, Gall, Gall, and Borg (2005), suggest that "the pre-test scores in this design can be used to decide whether the two groups were initially equivalent on the pre-test variable," and they emphasise the necessity of the drawing experimental and control groups from similar classrooms. Further, they emphasise that the research report should give as much descriptive data as possible about the comparison groups, e.g., location, socioeconomic level of the participating schools/universities, teachers' experience level, and mean achievement scores of students in the different classrooms/schools/universities, to clarify the degree of similarity between the groups. If the comparison groups are judged to be similar, then the results of a pre-test post-test design with non-equivalent groups can carry nearly as much weight as the results from a true experimental design (Gall, Gall, and Borg, 2005).

Ary, Jacobs, and Razavieh (2002) also advocate this design as one of the most widely used quasi-experimental designs in educational research even though they highlight that the "selection bias" resulting from non-random assignment can seriously threaten the internal validity of this design. The application of the pre-test before starting the experiments provides a means of comparison. If there are no significant differences of the pre-test between the experimental and control groups, it is assumed that this selection is not a threat to the internal validity. However, if there are some differences, then an analysis of covariance (ANCOVA) can be utilised to statistically adjust the post-test scores for the pre-test differences. To maintain internal validity, various techniques need to be applied to the two comparison groups, e.g., they have to perform the same pre and post-test across the same period of time, and with the same teacher giving the instruction.

Significantly, there are some other 'threats' to internal validity that need to be avoided or reduced during this research such as; selection bias, maturation, instrumentation, regression and history (Johnson & Christensen, 2004).

To try to minimize the effects of some of the above five biases occurring in this study, there were several approaches that were used. Considering selection-bias, post-tests were made and analysed statistically to check if both groups had a close level at the beginning of the study. Regarding selection-maturation, both of the two groups were taught the same subject by the same teacher at the same day, one group met from 8 to 10 and another from 10 to 12. In terms of selection regression and history, both of the two groups were from the same populations, i.e., they were second year student English majors.

To summarise, in a typical school/university setting class schedules cannot be disrupted nor can classes be reorganised to facilitate experimental research. Consequently, it was necessary to use class groups as the students had already been grouped into classes (i.e., pre-existing intact classes). Thus, a pre- post-test design with non-equivalent groups was judged to be the accepted and valid quasi-experimental design for assessing quantitatively the effect of the experimental teaching interventions.

3.3.2 Qualitative Methods

So far, the previous sections have described the research as if there was just one accepted way of investigating the proposed questions. However, there is another paradigm, 'qualitative', that is very valuable for this investigation. For this study, the qualitative methods are in the interviews, the open-ended questions included in the pre- and post-questionnaire and field notes. The main advantage of open-ended questions is the freedom they give to the participants. Here, we get the participants ideas, thoughts in their own words, and these replies are often valuable (Oppenheim 1992). Interviews help to "follow up unexpected results, for example or to validate other methods, or to go deeper into the motivations of respondents and their reasons for responding as they do" (Kerlinger, 1970, cited in Cohen et al., 2011: 411).

On the other hand, field notes contain descriptive information which will be helpful for the researcher to return to the observed situations. The purpose behind conducting observations was to capture a realistic atmosphere of events, reactions, and behaviours that take place in the classroom. Marshall and Rossman (2006) state that observation "entails the systematic noting and recording of events, behaviours and artefacts (objects) in the social setting chosen for study" (p.98). They also point out that the rationale of observations is to determine the persistent patterns of behaviours and relationships among the participating students. Observations provided data about the students' behaviours toward writing in English.

3.3.3 Triangulation

Many specialists in education research, such as Bryman (2008), Cohen et al. (2011), Clough and Nutbrown (2007), Weir (2005), and Gillham (2000) consider triangulation as an essential step towards validating the results of a study. In this study, methodological triangulation will be done by having a number of different quantitative and qualitative data collection methods. As has been mentioned, triangulation helps minimise the drawbacks of employing a single-method research. Findings from different methods mutually reinforce each other. In the case of this research, methodological triangulation will be achieved by using different data collection methods: quantitative in the case of pre- and post-tests and the questionnaires; and qualitative as far as open-ended items of the questionnaires, individual interviews, written texts and field notes were concerned.

3.4 Overview of the Project Tools

Different data collection methods will be used to gather the necessary data during the research. Tools include a pre- and post- test written exams, pre- and post- questionnaire, individual interviews with members of the experimental group and the field notes. The pre- and post- tests will measure students' progress in writing in both groups. The pre- and post- questionnaire will help get a general idea of students' writing experience, computer familiarity and perceptions of TBF and CBF. The post-questionnaire will be used if their perceptions and use are going to change at the end of the experiment. The teacher's field notes will be used to record and examine how CBF is integrated in the classroom. Finally, the individual interviews will be used to give insights to the previous findings and examine students' use and perceptions of CBF and TBF.

3.5 Procedures of the Questionnaires

Bryman (2008), McDonough and McDonough (1997), Clough and Nutbrown (2007), Gillham (2000), and Cohen et al. (2011) among other experts believe that questionnaires are a very popular data collection method in educational research. There are several aspects that can lead to a researcher choosing questionnaires to collect data from students, which naturally apply to this research project, including: questionnaires tend to be more reliable as they are anonymous; they encourage greater honesty from respondents; they save the researcher's and participants' time and effort; and they can be used in small-scale and large scale issues (Cohen et al., 2011; Bryman, 2008; McDonough & McDonough, 1997). However, experts also point out that questionnaires also have some disadvantages. For instance, Mertens (1998) pointed out that questionnaires rely on individuals' self-reports of their knowledge, attitudes, or behaviours, thus the validity of information is subject to the honesty and perspective of the respondent. Cohen et al. (2011) also believe that questionnaires might have the following disadvantages: the percentage of returns is often too low; if only closed items are used they may lack coverage or authenticity; if only open items are used, respondents may be unwilling to write their answers.

It is therefore very important for researchers to strike a balance between the advantages and disadvantages. To address the lack of coverage and authenticity associated with closed questions, there will be open ended questions and a follow up individual interviews with some selected students, with less-structured questions and further opportunities to elaborate on answers to items in the questionnaire. This is expected to minimise any undesired negative effects including lack of coverage. Other suggestions will be taken from Cohen et al. (2011), who suggested that the researcher needs to pilot questionnaires and refine their content, wording, and length accordingly, and to make it appropriate to the targeted sample.

3.5.1 Selecting the Questions

This study utilised the structured type of questionnaires (Appendix II Questionnaire) due to the nature of data required for the present research, i.e. data regarding English writing attitude and motivation, recent writing and feedback experience, computer and internet familiarity, and TBF and CBF attitude, motivation and perceptions. The questionnaire mainly used structured responses, rather than using a semi-structured. However, some semi-structured questions were used to overcome some of the problems of the structured questionnaires like limiting the respondent's choices to the available answers. Therefore, I inserted some open-ended questions to allow the participants to respond freely. To start, I set the objectives of the questionnaires according to which relevant data would be collected in order to fulfil the main study goals. The questionnaire was used in this study to serve the following objectives:

To collect data about students' attitude and motivation in writing in English in general.

To collect data about students' most recent writing experience in terms of essays number and type of feedback.

To collect data about students' preferred place and method of writing.

To collect data about participants' computer and internet familiarity level.

To collect data about participants' TBF attitudes, motivation and perceptions.

To collect data about participants' CBF attitudes, motivation and perceptions.

The questionnaire has three main sections with forty four questions in total and was designed to achieve the objectives mentioned above. The first section, with twelve questions, was designed to collect data about the participants English writing in general. The first question was about the subject's gender so I can group participants according to their gender and use that afterwards in the between-group analysis to answer the research question about gender difference in performance on CBT. The second question was about the hometown of the participants to find out the representativeness of the participants in the wider context, i.e. the Saudi context. The third question investigates the participants schooling type, i.e. state or private, due to the fact that private schools provide computer instruction from year one for both male and female students, while state schools only provide that for male students from year 10. The fourth and fifth questions collected data related to the exposure and training on computers at any prior university educational level. All these questions aim at collecting data about participants' educational background which will provide data to answer the research question (RQ7) concerning any gender difference in performance on CBT. The second main section in questionnaire one, about participants' computer familiarity, with 14 questions, was difficult to construct due to the lack of a clear definition in the literature (see section 2.7) of the concept of 'computer familiarity'. Different studies have examined the computer familiarity from different perspectives. Some studies have operationalised the construct of computer familiarity as the amount of use of computers, the length of time of owning a computer, or previous computer courses taken by the students (Geissler & Horridge, 1993). Other studies have defined computer familiarity as having access to computers in different places (Geissler & Horridge, 1993). Indeed, Boo (1997) argued that a valid measure of current college students' level of computer familiarity is unavailable. Therefore, Taylor et al. (1999) tried to construct a comprehensive questionnaire for their study that encompassed all aspects of computer familiarity examined in the literature. To encompass the wider concept of computer familiarity, as in Taylor et al. (1999), and to construct a valid measure for the university students as in Boo (1997), most of the present study questionnaire items were adopted from these two different sources, i.e. Boo (1997) and Taylor et al. (1999), and some were added by the researcher himself. However, there were some questions in Taylor et al.'s scale that seemed irrelevant for the purpose of this study which are Q.3, Q.4, Q.8, Q.11, Q12, Q13 and Q14 (see appendix I for a copy of Taylor et al.'s and Boo's questionnaires). I adapted Taylor et al.'s (1999) remaining questions, and one question from Boo's scale, which is question 6 in my questionnaire, and I added more questions such as Q.7 C, Q.9, Q.10, Q.14, and Q.15. The reason behind adding these questions to my questionnaire is to make the scale more detailed and comprehensive. For instance, Q.7 C, about having access to computers at internet cafes, was inserted to add an important option to Q.7. Questions 9 and 10 give more details about the amount of computer use by the participants. Q14 was added to support Q.13 in my questionnaire, and Q.15 would provide further details about some keyboarding skills.

The last section in questionnaire one is about students' attitudes towards computers and their testing mode preference. There were eight items in this section that measure participants' computer attitudes, and one question to investigate their testing mode preference. The computer attitudes scale items were adapted from a wider computer scale called Computer Attitude Scale (CAS) developed and validated by Loyd & Gressard (1984) (see appendix N for a copy of CAS), and revalidated again by Bandalos & Benson in 1990 (see section 2.8 for more details). However, the wording of some questions, such as Q.21 (figuring out computer problems does not appeal to me), was mainly negative, and that confused the participants in the pilot study (see section 3.3.6 for implications for the main study). Therefore, I amended the wording of those questions to be affirmative, which resulted in clearer and more accurate statements. The last question in section three was about the participants' testing mode preference, and a justification for their choice. Before the questionnaire was administered, it was vital to administer the study questionnaires in Arabic to avoid problems that may occur due to the use of L2 (see appendix D & E for copy of the translated questionnaires). Therefore, after constructing the questionnaire, I translated it into Arabic. Then it was given to a professor of translation, a colleague at the English language centre at King Faisal University, to check the accuracy of translation. After that, I requested another colleague who is a professor of Arabic language at the Arabic language department to check the accuracy and clarity of the Arabic language used in translation.

3.5.2 Pre-Piloting Stage

3.5.3 Piloting Stage

The pilot study aims to achieve two main goals upon which the main study will have its future direction. First, it is always important to try out the study instruments before going to the real fieldwork and facing unwelcome troubles. Hitchcock and Hughes (1989) recommend that "it may be better for the teacher (in this study, the researcher) to develop a pilot study and uncover some of the problems in advance of the research proper." (p.198). It is also essential to reveal any difficulty that respondents might experience due to the study procedure, instructions, or instruments. Piloting will examine the validity and reliability of the study instruments as well as the clarity of the instructions given, and the appropriateness of the chosen instruments to fulfil the study objectives. I wanted to try out all the instruments used in this study before going for the main data collection. Furthermore, I needed to examine the instruments while still near the experts, my supervisors, so they can comment and give further feedback.

Second, the participants will always produce feedback that will guide the researcher to amend and modify the design of the main study in order to overcome any unexpected problems. I aimed at collecting as much feedback as I could in order to improve the instruments and the procedure for the main study design.

3.5.4 Administrating the Questionnaire

3.6 The Writing Pre- and Post-Tests

Writing tests should help yield essential data required for analysis into the effect of different feedback techniques. However, many experts in educational research (e.g. Cohen et al., 2011; Gall et al., 1996) stress the fact that the use of tests in research raises a number of ethical concerns. For instance, many researchers have reported that individuals may suffer from anxiety in testing situations. It is therefore the researcher's responsibility to elicit participants' best performance, while minimizing their anxiety if they plan to use a test as part of the data collection process. Scoring measures for writing assessments will follow recommendations by Weigle (2002), an analytic assessment-based rating procedure used by Lundstorm and Baker (2009), and the grading rubric used by the computer software used in this study which is Criterion, to ensure the reliability and validity of the rating practice. That includes defining the rating scale, and ensuring ratters use the scale suitably.

3.6.1 Selecting the Tests

3.6.2 Administrating the Tests

3.7 Content Analysis of Written Texts

3.7.1 Selection of Written Texts

3.7.2 Comparing of Written Texts

3.8 Individual Interviews

Interviews will be the last stage of data collection and are supposed to supplement and give an in-depth account of data already generated by the second questionnaire. Most research manuals mention that interviews and questionnaires are two very accepted methods for collecting data in educational research, and such extensive reviews of interviews give a clear idea of how they best function in this situation (e.g. Gillham, 2000; Cohen et al., 2011; Hollway & Jefferson, 2000). Seidman (1998) points out that the purpose of in-depth interviewing is to understand "the experience of other people and the meaning they make of the experience" (p.3). Interviews can be used to collect information, which is difficult, if not impossible, to obtain through the observations. For example, interviews can seek to obtain participants' personal views of a phenomenon or an event.

One important step towards developing the questions in the interviews is what Gillham (2000) calls 'trialling the interview questions,' which, despite many similarities, is different from 'piloting', a more advanced and mature level. In fact, trialling in a way resembles the pre-pilot study, in the sense that both were early stages in developing data collection methods for the inexperienced researcher. The interviews are semi-structured, one-to-one format to best meet the requirements of the study. Interviews will observe a more inductive logic, as opposed to deductive logic, whereby theories and cognitive principles would emerge from the data, or in other words moving from the specific to the general. Research methods literature suggests that inductive logic is more suitable for arguments based on experiences or observation as the case here (Gillham, 2000 and Cohen et al., 2011).

Interviews help to "follow up unexpected results, for example or to validate other methods, or to go deeper into the motivations of respondents and their reasons for responding as they do" (Kerlinger, 1970, cited in Cohen et al., 2005: 268).

Therefore, I used post-hoc interviews to verify and amplify the data collected from the post hoc questionnaire about testing mode preference after experiencing two different testing modes. It is a triangulated approach to increase the reliability and the validity of the study and the gathered data. It is also a way of overcoming drawbacks of each of these instruments and complementing one another (Brown, 2001). I used semistructured interviews for the following reasons:

1- The purpose of the interview is to verify, add information, and validate selected participants' responses in the second questionnaire.

2- Moreover, the nature of the collected data controls the type of the interview employed. Cohen et al., (2005) suggested that the more personal and informal data the interview seeks to find, the more open-ended and unstructured it should be.

Therefore, I prepared three questions (in Arabic) to initiate the interview with the selected participants. These questions are:

Q.1- Your preference in questionnaire one was.., would you please explain why?

Q.2- After the experiment, you changed your testing mode preference to…., would you please explain why?

Q.3- If the problems you mentioned with CBT are tackled, will you change your preference? (For those who initially preferred CBT, then changed to PBT)

After deciding on the type of interview to be used for the purpose of this research, I used the guidelines recommended by Brown (2001:80) to achieve the maximum out of this instrument. These guidelines are:

1- Understanding the general outline of the interview.

2- Being familiar with the interview instrument.

3- Following question wording exactly (see previous page for questions).

4- Recording answers verbatim using a very advanced MP3 recorder.

5- Using follow-up questions when necessary.

The interview was administered in Arabic so participants could respond freely and optimally to all interview questions. It was administered two days after the administration of the second questionnaire in one of the language labs. Due to the time limit and students' academic commitments, the interviews lasted for four days interviewing six participants every day during their one-hour break time. Each participant was interviewed separately, and for about 10 minutes.

3.8.1 Selecting the Questions

3.8.2. Administrating the Interviews

3.9 Teacher Field Notes

ESL classroom observation is important and may lead to changes in best practices in writing instruction. Van Lier (1988) suggests that observation for second language acquisition research is important because the L2 classroom is the place where second language development occurs.

As it is almost impossible to observe and remember all the details of classroom events, Patton (2002: 302-303) suggested that teacher researchers develop a systematic approach to note-taking as follows:

Field notes are descriptive.

Field notes contain what people say.

Field notes also contain the researcher's own feelings and reactions about what they observed.

Field notes include insights, interpretations, beginning analyses, and working hypotheses about what is happening in the setting.

In this respect, field notes contain descriptive information which will be helpful for the researcher to return to the observed situations. If there is a great lapse in time between the observed event and the writing up of the field notes, it is sometimes difficult to reconstruct the sequence of the action accurately (Hatch, 2002). In addition, observational skill needs to be improved with practice in order to discover more about the process of L2 teaching and learning. In reality, however, it was more or less demanding for me to take notes while engaging in teaching activities within the observation situation. It is important to note here that field notes provided the opportunity to understand what happened in the classroom and learn how to interpret the qualitative data collected during a classroom observation.

The purpose behind conducting observations was to capture a realistic atmosphere of events, reactions, and behaviours that take place in the classroom. Marshall and Rossman (2006) state that observation "entails the systematic noting and recording of events, behaviours and artefacts (objects) in the social setting chosen for study" (p.98). They also point out that the rationale of observations is to determine the persistent patterns of behaviours and relationships among the participating students. Observations provided valuable data about the students' behaviours toward writing in English.

The researcher/teacher kept a reflective journal at the end of each CBF session and periodically at the end of each week. In the reflective journal, the focus was placed on: 1) the classroom atmosphere, particularly, the connection between the instructor and students; 2) the significant issues that occurred in the class, such as problems or conflicts each student encountered, i.e. students' frustration of working on computers and the instructor as well as of operating computers; and 3) reflection as an instructor regarding students' performances in class as well as their development. Data collected through reflective journals helped the researcher to recollect what took place in each CBF session, what were the noticeable issues that occurred in each week, and how students developed academically, which contributed to the later data analysis in terms of discovering how participants' behaviours in CBF sessions were driven by their motives and how the context had influence on participants' participation in CBF sessions as well.

The researcher/instructor conducted more focused observation of CMPR in the computer lab. Stake (1995) purports that observations provide the researcher with a greater understanding of the case, and through observation, the researcher can keep "a good record of events to provide a relatively incontestable description for further analysis and ultimate reporting" (p.62). Since the researcher intended to discover each student participant's motive of participating in CMPR, which could only be observed in their actions (Lantolf & Thorne, 2006), as well as the influence the contexts cast on participants' actions, the researcher's focus during the observation was placed on: 1) the atmosphere and environment during the computer-mediated communication period; 2) interaction among ESL students during CMPR; 3) the interaction between ESL students and the instructor during CMPR; and 4) students' physical movements and facial expressions while operating a computer.

One sample observation protocol provided in Reed and Bergemann (2001) was used in the pilot study during the 2011/12semester. Based on the observation during the pilot study, the researcher made modifications in the protocol to more accurately address the research focus in the proposed study (see Appendix I for the revised observation protocol). The observation protocol was slightly modified during the data collection process to capture emergent issues such as participants' certain actions at certain point of time. Two modes of observation were conducted including the researcher's field notes-taking and video taping. When there was no one student asking questions, the researcher/instructor take field notes by using the observation protocol.

Due to the equipment arrangement in the computer lab, it was not feasible for the researcher to observe all participants at one site. In order to capture as much information as possible during CMPR sessions, the researcher spent equal time in observing each participant from the area that provided a clearer view of each participant's performances. However, due to the fact that students asked questions during the lab time and the instructor did not have plenty of time taking field notes, two video cameras were used during each CMPR session to record participants' beyond-screen behaviors. One video camera was placed in the front of the computer lab and the other one in the back area. After each observation, the researcher immediately reviewed the video recordings in order to fill the information that was captured in the video tapes but missing in the field notes, into the observation protocol. Once the protocol was updated, all observation information were be synthesized, which allowed the capturing of accurate information from the field.

In summary, the use of two modes of observation were beneficial to the research in the following three aspects: 1) collecting data through both videotaping and field notes taking allowed for the collection of thick data for the study (Lincoln & Guba, 1985); 2) the researcher's presence in the classroom since the start of the semester made less intrusive the later data collection process; and 3) the researcher's prolonged presence in the classroom and her role as technology trainer and facilitator in the computer lab helped build trust between the participants and the researcher, which benefits the collection of credible data (Lincoln & Cuba, 1985). Information obtained through lab observations were used to supplement the IM chat transcripts and the researcher's reflective journals, which not only helped triangulate findings, but portrayed a fuller picture of what ESL students underwent during CMPR.

3.9.1 Teacher Field Notes Technique

3.10 Choosing the Software

3.11.1 Software selection

For the purpose of this study, I did my best to find a suitable computer testing programme that would fulfil all the goals of this study and meet the requirements of the target context, if it is to be recommended in future. Certain criteria, based on my experience about the target context needs, were used in comparing and selecting the target system. These are:

1- A comprehensive system that supports different types of questions, i.e. objective, open-ended and short essays, as well as supporting all linguistic skills so that, if recommended, it meets not only the requirements of my study but also those of the target context .

2- A system that is easy to be learnt by both teachers and students, and easy to be taught and tutored by system administrators.

3- A system that can deliver assessments through different user interfaces, i.e. Windows-based or browser-based assessments.

4- A system that is computationally efficient and reliable, i.e. that does the job perfectly with minimum requirements on both the server and the clients.

5- A system that is scalable, i.e. supports all types of databases available in the market, such as MS Access, MS SQL, and Oracle, in order to accommodate the new students joining the university every year as well as the increasing number of administered assessments.

6- A system that enables delivery over the intranet, the internet, or a standalone delivery which can be used for high and low stakes testing sessions. This would help the university to accommodate more test-takers, particularly if administered over LAN.

7- A system that has a flexible interface design, as well as enabling the teachers and testers to customize the end-user interface.

8- A system that has the most comprehensive management system, i.e. the scheduling, analyzing and reporting systems. These criteria were selected not only for the purpose of this study, but also for the purpose of recommending the employed system to be used for many other types of tests in the target context, provided that the study results are encouraging and affirmative for the introduction of computer-based testing. After an extensive search in the market and on the internet, through which I examined the available computer testing systems, the researcher found the system that matches the previous criteria. The system is called 'Perception ©' which is designed and marketed by QuestionMark © (9). According to QuestionMark©, Perception© is a comprehensive, reliable and efficient computer testing system that has been in the market since 1995 as a complete package for computerized testing systems. It has also been implemented in several academic institutions, and used by many organizations including the University of Essex. The system is not free; however, it is not as expensive as other systems such as Blackboard © or WebCT © (10) that have either their own testing module, or can integrate third party assessment software. Perception© is licensed according to a one-off licensing scheme based on the number of servers and participants. It enables the test author to create 20 different types of questions, and to insert different types of multimedia materials. All of this can be done using a simple wizard in a Windows-based software component, or through a browser-based component. Editing and trying out the questions is as simple as browsing a regular website. In addition, also the flexibility of the system allows you to design whatever interface you need for your assessment by customizing the available templates or adding your own design. The system is very scalable, handling very small databases of tests and test items up to huge shared repositories. Perception© is very efficient in terms of delivery since test designers can deliver their assessment on paper, on CDs, over the local area network, or over the internet. It also has a very powerful management system that provides thorough assessment scheduling, comprehensive scores analysis and statistics, and very detailed reporting of results. All this can be achieved by easily setting up the system on a regular server and workstations with minimum requirements. However, there was some criticism of this system that was brought to my attention during the Language Testing Forum 2006 at Reading University. For instance, the limited support available from the vendor after purchase; however, this had not happened to me when I used the system.

Many CBF programs have been considered for the use in this project. The following table presents an idea about the CBF software that has been examined as potentials for this project:

Software name

What is it doing or claiming to do

Some References on the software


- focus on Style and


- address a wide range levels

- measure writing, track student progress

- instant feedback and scoring

- Students revise and resubmit their essays after feedback

- Students can write off line and paste it in criterion online later

- Online web based

- contains a handbook to help students

- teachers can add their own notes for SS's

- SS's can send questions to teachers

- Level of feedback, time and other features is controlled

- provides quantitative data

- measures:

• Grammar

• Usage

• Mechanics

• Style

• Organization & Development

- Attali, Y. (2004). Exploring the feedback and revision features of Criterion. Paper presented at the Paper presented at the National Council on Measurement in Education, San Diego, CA.

- Dikli, S. (2006). An Overview of Automated Scoring of Essays. The Journal of Technology, Learning, and Assessment (J.T.L.A), 5(1), 1-36.

- Burstein(2004). Automated Essay Evaluation: The Criterion Online Writing Service. AI Magazine Volume 25 Number 3 (2004).

Project Essay Grader

- older software

- ignoring the semantic aspect

- relies on style

- Analysis of surface linguistic features of a block of text.

- Essay is predominantly graded on the basis of writing quality, taking no account of content.

Page, E. B. (2003). Project Essay Grade: PEG. In M. D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 43-54). Mahwah, NJ: Lawrence Erlbaum Associates.

Intelligent Essay Assessor (IEA)

- Focus on content

- gives feedback on quality of content, grammar, style and mechanics.

- compares

content similarity between a student's essay and other essays on the same topic scored by human ratters to determine how closely they match

- can spot plagiarism

- web based

- Hearst, M. (2000). The debate on automated essay grading. IEEE Intelligent Systems, 15(5), 22-37, IEEE CS Press.

- Jerrams-Smith, J., Soh, V., & Callear D. (2001). Bridging gaps in computerized assessment of texts. Proceedings of the International Conference on Advanced Learning Technologies, 139-140, IEEE.


- the first essay-scoring tool that was based on artificial intelligence


- used for placement and not instruction

- uses Artificial intelligence (AI) and Natural Language Processing (NLP)

- uses pre-scored human material

- it grades focus and unity, organization,

development and elaboration, sentence structure, mechanics and conventions

- modelled on how the human brain acquires, stores, accesses and uses information

- learns from its mistakes

- capable of evaluating essay responses in multiple languages including English,

Spanish, Hebrew, Bahasa, Dutch, French, Portuguese, German, Italian, Arabic, and Japanese

- Elliot, S. (2003). IntelliMetric: from here to validity. In Mark D. Shermis and Jill C. Burstein (Eds.). Automated essay scoring: a cross disciplinary approach. Mahwah, NJ: Lawrence Erlbaum Associates.

- Rudner, L., Garcia, V., & Welch, C. (2005). An Evaluation of Intellimetric Essay Scoring System Using Responses to GMAT® AWA Prompts (GMAC Research report number RR-05-08).

MY Access!

- Focus on style and content

- web based

- the instructional version of IntelliMetric

- immediate scoring and diagnostic feedback

- uses Artificial intelligence (AI) and Natural Language Processing (NLP)

- assigns essay topics and provides feedback in English, Spanish, or Chinese.

- Students have two options in using the MY Access! Program. One option is writing on a topic assigned in English, Spanish, or Chinese and receiving feedback in the same language. Another option is writing an essay in English and receiving feedback either in the native language or in English

- multilevel feedback

- multilingual dictionary, thesaurus, and translator functions

- stores students portfolio

- Various teacher options

- Shermis, M. & Barrera, F. (2002). Exit assessments: Evaluating writing ability through Automated Essay Scoring (ERIC document reproduction service no ED 464 950).

- Shermis, M. D. & Burstein, J. (2003). Automated Essay Scoring: A cross disciplinary perspective. Mahwah, NJ: Lawrence Erlbaum Associates.

Bayesian Essay Test Scoring sYstem (BETSY)

- Focus on style and content

- classifies text based on trained material

- determine the most likely classification

of an essay into a four point nominal scale (e.g. extensive, essential, partial, unsatisfactory)

- a Windows-based program


- Rudner, L.M. & Liang, T. (2002). Automated essay scoring using Bayes' Theorem. The Journal of Technology, Learning and Assessment, 1(2), 3-21.

Conceptual Rater (C-Rater)

- designed for short written answers

- focus on content

- aims at scoring either correct or incorrect

- looks for specific written information on a specific domain

- doesn't require large texts input to grade answers

- uses an answer key

- uses Natural Language Processing (NLP)

- Burstein, J., Leacock, C., & Swartz, R. (2001). Automated evaluation of essay and short answers. In M. Danson (Ed.), Proceedings

of the Sixth International Computer Assisted Assessment Conference, Loughborough University, Loughborough,


Intelligent Essay Marking Systems (IEMS)

- IEMS is based on the Pattern Indexing Neural Network (the Indextron)

- The system can be used both as an assessment tools and for diagnostic and tutoring purposes in many content-based subjects.

- immediate feedback

- Ming, P.Y., Mikhailov, A.A., Kuan, T.L.: Intelligent Essay Marking System. In: Cheers, C. (ed.) Learners Together. NgeeANN Polytechnic, Singapore (2000)


- marking of free-text answer to open-ended questions

- employs NLP

- providing robust marking in the face of errors in spelling, typing, syntax and semantics.

- looks for specific content

- Feedback is typically provided as a mark

- Mitchell, T., Russel, T., Broomhead, P., & Aldridge N. (2002). Towards robust computerized marking of free-text responses. In M. Danson (Ed.), Proceedings of the Sixth International Computer Assisted Assessment Conference, Loughborough University, Loughborough, UK.

At present, though, the central force for the expansion and use of these CBF programmes comes from large international language testing organizations which are increasingly delivering products such as IELTS and TOEFL electronically across the world. Their aim is to be able to correctly and economically rank essays automatically (Valenti, Neri & Cucchiarelli, 2003). Among the top recognized of these CBF systems are the Criterion e-rater developed by the Educational Testing Services (ETS) and MY Access! developed by Vantage Learning. The Criterion e-rater (Burstein, 2003; Burstein, Chodorow & Leacock, 2004), for example, scans a student text and provides a holistic score with real-time CBF on grammar, usage, style, organization and development.

After careful considerations the CBF that will be used in this project is going to be Criterion. The decision was based on that the software gives more feedback than other CBF programs and on its ease of use by both teachers and students. Attali (2004) reports a large-scale study based on Criterion, which, as well as a holistic essay score, provides feedback on grammar, usage, mechanics, and style.

3.11.2 Software pre-piloting

3.11.3 Software Pilot test

3.11.2 Administrating the software

Many researchers have pointed out that the novelty of using computers can act as a motivational force that may prompt students to write. Other researchers, however, have warned us about anxiety problems on the part of some students who are faced with a machine that they do not know how to operate (Pennington, 1996, p. 26). That is why, in order to avoid the degree of fear and anxiety that the use of computers may produce in new users, the subjects in this study had some practice before actually typing the compositions to be e-mailed to the US.

3.12 Data Collection and Analysis

3.12.1 Questionnaire

For analysing and processing the questionnaire's closed-ended questions SPSS 17.0 was used. SPSS was used to get percentages, means and reliability values from a descriptive point of view, in addition to other quantitative measures including parametric and nonparametric tests. However for the open ended questions in the questionnaire, NVivo 9 was used to organize the data and record immerging themes for further analysis.

3.12.2 Pre- and post-tests

3.12.3 Field Notes

3.12.4 Written Texts

3.12.5 Individual Interviews

I transcribed/ translated the audio-tapes as soon as I finished interviewing the students (Appendix M). Each transcribed interview was reviewed and studied by itself to establish holistically the interpretive framework for each interviewee regarding his/her attitudes about and practices of the process writing approach. I totalled and recorded themes, repeated words, patterns, and positive or negative attitudes toward writing for each interviewee, centring on the outlines in the interview questions. Recurring themes identified when an interviewee was repeated the same words, phrases, or sentences several times as well as negative or positive reactions to each aspect of the writing process. Then the recurring themes for each interviewee compared with those of the other interviews. I categorized the recurring themes that ran across the interviews questions (Appendix N).


For a long time, qualitative research has been criticized for the lack of systematic and transparent methods and procedures of data analysis. Aiming to systematize the data management and analysis of qualitative research, Miles and Huberman (1994) define that qualitative data analysis consists of three subprocesses: data reduction, data display, and conclusion drawing and verification. Data reduction, which is also emphasized by other qualitative researchers such as Creswell (1998), refers to the process of condensing obtained data through summarizing, coding, finding themes and clustering. Data display means a compressed organization of information for further conclusion drawing, which can be achieved through writing structured summaries, synopsis, vignettes, or drawing matrices or diagrams (Miles & Huberman, 1994). The third process, conclusion drawing and verification involve the researcher's interpreting and building connections between 3.12.4 Written Texts the condensed and well-organized data as well as testing and confirming the findings. Techniques in this process include clustering, counting, compare and contrast, triangulation, negative cases, and member checking (Lincoln & Guba, 1985; Miles & Huberman, 1994). Miles and Huberman also emphasize that these three subprocesses unfold throughout the study design, data collection and analysis. In addition, data reduction, data display, and conclusion drawing and verification are undertaken in an iterative mode rather than a linear sequence during data collection and data analysis. In other words, unlike in a quantitative study, the boundary between data collection and analysis in a case study is blurred so that data can be condensed, displayed in matrices or networks, and then, interpreted in a way to draw conclusions any time during the study. For the data analysis of the current study, the researcher went through the aforementioned three processes. According to Miles and Huberman (1994), data reduction was an ongoing process throughout the study design, data collection and analysis in which the researcher decided which data chunk to code and / or discard. In this study, data reduction started with initial selection and summery of important information, which is called initial coding (Miles & Huberman, 1994). During the initial coding, the researcher took an overall review of all data and tentatively clustered data that were relevant to certain constructs, which varied in different questions, into distinct groups. Then, a higher level of coding called pattern coding (Miles & Huberman, 1994) were conducted to identify themes, configurations and initial explanation based on the first-level coding. To name the coded patterns, the researcher used both existing themes that came from literature, a deductive coding technique, and labels or categories emerging from the field data, an inductive