'Research design is theâ€¦who, what, where, when, why, and how ofâ€¦an investigation' (Hagan 2000:68). How a study is designed influences how a hypothesis is addressed, whether findings resolve the research question in some way, and whether findings are reliable, valid and replicable.
Many types of research design exist but space does not permit their exploration in this paper. Accordingly, to enable some level of description, I will only evaluate two designs - experimental and cross-sectional - in terms of how choice of design influences research findings and conclusions.
Three main criteria are used to evaluate research findings - reliability, replicability and validity. Reliable findings are repeatable and consistent (for example measures devised for particular concepts such as poverty, racial prejudice) (Bryman 2008). Replicable studies are deemed reliable so procedures used to arrive at findings should be detailed.
'Research designs areâ€¦major way of controlling for invalidity in research orâ€¦resolvingâ€¦causality problemâ€¦means of excluding rival causal factors' (Hagan 2000).
Get your grade
or your money back
using our Essay Writing Service!
There are four measures of validity (Bryman 2008):
Measurement - whether a measure of a concept is stable.
Internal - if the conclusion that a causal relationship exists between key variables (independent variable, for example unemployment rate, causes change in the dependent variable, for example crime rate) is proven to be reliable when controlling for intervening variables which could causally affect the outcome.
External - findings are generalisable to populations 'beyondâ€¦specific research concept' (2008:33).
Ecological - findings bear relation to 'what happens in people's everyday lives' (2008:33). Qualitative research designs are generally deemed more ecologically valid due to their 'naturalistic' nature (2008:34).
The experimental research design ("ERD") is deemed a guideline in measuring robustness of conclusions produced by using other research designs. ERDs use randomised control trials ("RCT") where possible - random assignment of research subjects 'from a similar population' (Hagan 2000:84) to experimental and control groups. The experimental group is treated with the independent variable (factor that the researcher hypothesises causes the variation in the dependent variable). RCTs eliminate selection bias - the researcher can assume the groups are equal in all main respects save the independent variable. Sampling is an 'integral part of any research design' (Crow 2008:43). Valid findings require that samples be representative of the population being studied. ERDs can be laboratory or field experiments. I will examine the latter.
Rosenthal and Jacobson (1968) used an ERD to study the impact of self-fulfilling prophecies - whether teacher's expectations ("TE") could influence academic performance ("AP"). Research was done over eighteen months in lower class locality American schools with high percentages of ethnicity. All students took the 'Harvard Test of Inflected Acquisition' (Hock 1999:94) to identify the more academically gifted (it was actually the 'Test of General Ability'(1966:115)). Teachers were deceived in this regard to allow for creation of genuine expectations necessary for the experiment to succeed. A RCT assigned students to treatment and control groups. The independent variable (TE) must come prior to change in the dependent variable (AP) to be causal. At the beginning of the year, teachers received a top ten list of students (lists actually compiled using RCTs) identified as 'academic bloomers' (1966:116). Students were retested eight months later - the experimental childrens' AP had improved significantly more than the control group's. Higher TE were found to cause academic improvement. The independent variable is all that differentiates the two groups. Therefore the groups are taken to be equally affected by intervening factors - including history  , testing  and maturation  - which could otherwise be argued to contribute to the change in AP. Omission of a control group from the design would affect internal validity of findings as it, together with the pre-test, safeguards against ambiguity regarding direction of causal influence.
In many situations it is inappropriate for the design to use a RCT. Non-random sampling is used where groups selected are matched as closely as possible (Ostrom, Parks, Whitacker 1973 - Posttest Only Group Design) - selection bias may influence findings in these cases.
Always on Time
Marked to Standard
Methods used within a design influence findings. This design employed IQ tests which are criticised as inadequate measures of intelligence because they could contain racial and cultural bias (Kamin 1974) - significant here as schools studied were in lower class localities with high ethnicity percentages. Schools in more affluent areas with lower ethnicity levels could have been sampled also to improve generalisability of findings. The measurement validity of findings may have been improved had more reliable tests of AP been used. Pre-tests influence the ecological validity of findings as they are absent in real life situations. These points could argue conclusions partly result from the research design chosen. However, to forego the pre-test means causal conclusions cannot be drawn.
ERDs may have reactive effects on findings, especially regarding generalisability - awareness of the experiment would undoubtedly influence peoples' response to it. This risk unavoidably accompanies all designs. Truly covert research, as in this case, may avoid it but raises ethical concerns in the absence of informed consent. Neither teachers nor students appear to have known of the experiment. Bentham and Mill argue that issues like invasion of privacy 'should be outweighed byâ€¦beneficial consequences forâ€¦society as a whole' (Crow 2008:51). An ethical trade off was made in the name of valid findings. However, if some teachers suspected an experiment were taking place, external and internal validity of findings could be challenged - firstly, behaviour may be altered and secondly, genuine higher expectations may not be created. Therefore results may not be entirely accurate.
The ERD seems a good option here as despite its noted disadvantages, resulting findings can be argued to be valid and objective. If the design included qualitative methods - interviewing teachers and children before and after tests - resulting findings may be different. It may include an outsider in the classroom observing teacher-child interactions. Subjects would be aware of the experiment and of their behaviour as a result. However, qualitative data could contribute to an understanding of how children perceive teachers and their personal differential treatment and vice versa. Deductions could be made regarding how such perceptions influence children's drive for academic improvement.
This experiment is very well but it is often difficult to obtain individuals or conditions where variables are manipulable. An ERD would be inappropriate for many studies. Longitudinal or ethnographic designs have stronger external and ecological validity but internal validity of findings can be weaker than in ERDs or other more quantitatively focused designs. In choosing a suitable design the researcher must consider how their choice may influence findings.
Cross-sectional research designs ("CSRD") are non experimental involving collection of information at one specific time regarding questions about variables like actions, attitudes and characteristics of subjects. Large samples are typical so researchers may examine effectively the many differences between individual groups within the population - 'to detect patterns of association' (Bryman 2008:44). The larger the sample, the more generalisable the results - one can gauge better the views of the whole population. Due to the nature of subjects non-random sampling is generally used.
Subjects can include people, organisations, towns etcetera.
'Identifying the time order of effects....is critical for developing a causal analysis but can beâ€¦insurmountable problem withâ€¦[CSRD]' (Bachman and Schutt 2007:157-8).
As information is collected at one time, variables are not time ordered. This is a reason why, although findings of relationships between variables can be made and sometimes generate causal inferences, causal conclusions generally cannot be drawn. Controlling for all intervening variables is difficult with a CSRD which necessarily affects findings. Sampson and Raudenbush (1999) conducted research in Chicago neighbourhoods using a CSRD to explore whether a relationship existed between 'visible public social and physical disorder' (Bachman and Schutt 2007:158) and crime rate. Findings lacked internal validity because their CSRD 'could not establish directly that variation in crime rate occurred after variation in informal social control' (2007:158) - other factors could have contributed to the reduced crime rate. This problem can be remedied somewhat by using statistical controls. When levels of social control were controlled for here, it was found that visible disorder itself did not alter crime rate.
CSRD findings have stronger internal validity where characteristics like age or race are the independent variable which by their nature are fixed at a point in time before variation in the dependent variable; or, where measures taken are based on going through old records to obtain information on earlier cases relevant to the current research. However, use of retrospective CSRDs (archival data) can affect findings. Lo et al (2008), in using arrest records obtained only from Ohio, 'may have missed important incidents' (Bachman and Schutt 2007:158) which would necessarily have negatively influenced their findings' generalisability.
This Essay is
a Student's Work
This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.Examples of our work
Research designs are chosen according to their appropriateness regarding a hypothesis. CSRDs are commonly used in social research as variables are typically non-manipulable. Blaxter's (1990) Health and Lifestyles study illustrates how CSRDs can influence internal validity of conclusions. Information was collected from open-ended questions about health asked of a random sample of 9000 people, in England, Scotland and Wales. A relationship was found to exist between smoking and diet. However, it was not concluded whether smoking was caused by diet, whether diet caused one to smoke or whether one smoked despite eating healthily. Therefore, findings resulting from the use of CSRDs can be vague - such causal associations would 'rarely haveâ€¦internal validity of those deriving from experimental designs' (Bryman 2008:46). The sample was randomly selected - findings can be argued as generalisable. Structured interviews and self-completion questionnaires were used which could be said negatively affected the naturalness of the social setting of research subjects (Bryman 2008) and therefore the ecological validity of findings. A longitudinal study following subjects from childhood to adulthood may have provided more causal conclusions regarding a smoking-diet relationship. To make the study worthwhile a number of issues, including smoking, could be examined. Changes in diet over time could be studied in terms of those who took up smoking and those who did not. Longitudinal studies are far more expensive to conduct than CSRDs and findings may be affected by mortality issues  .
The fact that variables (like gender, race, age) are generally non-manipulable is problematic regarding research projects. In an ideal research world variables would be manipulable - ERDs could be used in all research projects thereby eliminating many of the issues associated with CSRDs and other designs like longitudinal and ethnographic. Blaxter may have arrived at internally valid conclusions if the variables in her study were manipulated, for example, diets of a random sample of non-smokers were established, a RCT randomly assigned them to experimental and control groups, cigarettes being the independent variable. Diets of both groups would be analysed for change after a period. Intervening variables would be controlled for - one could conclude changes in the experimental group's diet resulted from their new smoking habit. This type of research, for obvious reasons, would be inherently unethical and also unreasonable (Bryman 2008).
The CSRD in this case is, and CSRDs in general are replicable as the researcher usually details procedures used in designing and carrying out research. Therefore one can argue CSRD findings can be proven reliable by replicating the study.
'Researchers always wonder whether theyâ€¦omittedâ€¦relevant variables fromâ€¦controls, whetherâ€¦experimental results would differ ifâ€¦experimentâ€¦conducted in another settingâ€¦whether theyâ€¦overlookedâ€¦critical historical event' (Bachman and Schutt 2007:169).
In reviewing research findings one must note that research designs used necessarily and unavoidably have influences on aspects of conclusions and their potential shortcomings should always be considered. As already noted, the effectiveness of different research designs is measured against the experimental research design model, a model which itself generates issues that influence findings. All designs are different, therefore choice of a particular design will always influence outcome. Researchers must balance resources available with the design that best suits the research proposal - trade offs may have to be made. Above all, researchers must, in choosing a research design, safeguard against threats to validity, reliability and replicability as much as possible to ensure findings are not spurious. Ethical issues should also be considered and should only be waived in exceptional circumstances.