Comparison between true experiment and quasi experiment

Experiments are basically conducted to determine the cause and effect of a treatment, program or any other implementation. It is done to find out if something works; a test to see if a different method would be better than the previous. They are usually done to find a new way to improve the present situation at hand. For example, in hospitals, a particular group of patients with stomach cancer are divided into two subgroups, one to stick with the current medication and the other to be given a new medication. Their progress are tracked by researchers for a few weeks and recorded to analyse which group of patients has shown better health improvement. If the new medication was proven to be a successful treatment, it would quickly be of replacement to the previous.

Most experiments are conducted in a highly controlled environment such as the laboratory whereby a random sample of test participants has been selected prior. They are usually conducted as a comparison test between at least two groups of participants, a treatment group and a controlled group. The controlled group will be those of a standardised condition while the treatment group are those who will receive the treatment (E.g. new program, medication, etc). All experiments would have to have a degree or validity and reliability to ascertain its effectiveness and genuinity.

A controlled experiment is done in a laboratory and is usually conducted to satisfy a knowledge gain without any immediate purposes that affects the current conditions (Shaunessy, et. al, 2006). It is therefore the researcher’s task to decide on an experimental design which suits best his experiment. He will be required to take into consideration the internal and external validity of independent and its dependent variables that would have a causal effect on his participants and situation.

As such, true experiments conducted in a natural setting or a field experiment out of the laboratory would be done to test the external validity of these controlled laboratory experiments. This may be in hospitals, institutions or businesses. They are conducted in such a way that the results will be of a certain impact to the group of people concerned. The results will then be the deciding factor if the program implementation should be of immediate effect. In a natural setting, the researcher has a much lesser degree of control over the external validity of his participants due to unknown confounding factors that may unknowingly affect the experiment. True experiments in a natural setting are commonly conducted to assess ‘social’ issues and have a more practical direction.

As discussed by Campbell & Stanley (1966; Shaughnessy, et. al., 2006; Jackson, 2003), factors such as history, maturation, testing, bias, instrumentation, regression, subject attrition and selection would have to be looked at while conducting any experiments. These are confounds that may be threats to the internal validity of any experiment. Campbell & Stanley (1966) informed that internal validity is the most basic of what is required in an experiment.

Shaughnessy, et. al (2006) further explains that true experiments require the manipulation of an independent variable with treatment and comparison condition. With a randomised participant selection, there would also be a high degree of control. A high degree of control is the researcher’s ability to take control of participant randomised assignment, choice of dependent variables, situation and systematic manipulation of independent variable. Effectiveness of the experiment would be determined by the differences of the independent variables between the comparison groups.

It is known that there would be a definite level of difference in independent variables when comparing laboratory and true experiments due to confounding factors. In such circumstances, the researcher can implement the pretest-posttest control group design. In pre-test, is to measure the equivalency of both groups to assess their similarities and group them according to their most common dependent variables. Whilst a post-test is measured to analyse the differences of the independent variable at the end of the experiment. The researcher will analyse the data collected and determine the effectiveness of the experiment.

In natural settings, there will be a lack in degree of control. The researcher would be unable to control confounding factors that may affect his end results. As such he may alternate to a quasi-experimental design to limit threats to the internal validity of his experiment. The merriam-webster (2011) online dictionary defines the adjective ‘quasi’ as “having some resemblance usually by possession of certain attributes”. Therefore a quasi-experimental design in itself is an experiment which holds some similar characteristics to true experiments with an exception of random selection. It is often applied to case studies and when conducting true experiments are not feasible. It reduces time and resources needed for experimentation.

In quasi experiments, its test participants are pre-selected and the researcher has to sometimes ‘make-do’ with the given set of participants. The participants may also have been arbitrarily divided by the host institution or company. The lack of control in the experiment is purely based on convenience to reduce disruptions on the ongoing activities at the host institution.

The confounding factors affecting the internal validity as mentioned previously can be demonstrated through the following example. Similar to William Trochim, 2000, “Keep me in school” social experiment, a study conducted in a higher educational institution over a period of four years. The test participants are two groups of college freshmen and have been arbitrarily divided according to the administrative data provided. The independent variable is a new study method introduced to review its effectiveness over the current study method already in place. Its theoretical hypothesis is show that there is indeed a mark improvement in the treatment group’s grades over the four years of study in comparison to the controlled group.

In history effect, the students’ grades can be affected by extra help outside the school curriculum, with external educators implementing different methods of teaching. As the experiment runs for four years, this will take into account the maturation and testing effect. The test participants would probably be more familiarised with the school and examination system as time goes by. The anxiety of a new school system would naturally fade away. In testing, they generally improve after the initial test also due to familiarisation. This would come into effect especially in pre-test and post-test design. With pre-test, the test participants would gradually become aware of objective of the experiment and change their behaviour.

As mentioned by Trochim, (2000), instrumentation effects occur when the experiment results are affected if the test participants have done a pre-test assessment and were measured by different methods throughout the course of study. With subject attrition, there is the risk of test participant dropping out of college or transferring to another college. This will thus affect the end result as Jackson (2003) explains that if the number of dropouts does not correspond between the treatment and controlled group, the post-test difference would be too great making the experiment insignificant.

Campbell & Stanley (1966; Shaugnessy, et, al, 2006) informed that for a statistical regression threat to happen is when the test participants were selected based on their “extreme” scores through pre-test. This will change when a re-assessment is conducted as the test participants may score differently on a subsequent test due to regression to the mean. The some test participants may have had previously scored highly or lowly due to chance. Selection threat happens when there are inconsistencies in between test participant groups. This usually affects multiple group experimental design in association with the single group threats to the internal validity which are selection-history, selection-testing, selection-maturation, selection-instrumentation, selection attrition and selection-regression.

Apart from the previously mentioned threats, Shaughnessy, et, al. (2006) discusses that there are other factors that may be of threats to internal validity. They are diffusion of treatment, experimenter expectancy effects and novelty effects. Diffusion of treatment is the contamination of information to the other group whereby there is communication between participants of different groups. In our example, this will be unavoidable as the participants attend the same institution and probably live in close proximity to each other sharing the same daily activities.

Experimenter expectancy effects happens when the researcher knowingly or unknowing bias the results of the experiment. This will lead to errors in result interpretation of observation, identification, recording and behaviour of test participants. A novelty effect on the other hand is the effect caused by test participants themselves. When the group of test participants have been informed that they are to participate in an experimental project, their behaviour may sometimes affect the results. The knowledge of being interested in would cause some participants to be overly anxious or enthusiastic thus not showing their true behaviour. This effect is also known as the Hawthorne effect based on the study of productivity and work conditions at the Hawthorne plant of Western Electric Company in Illinois between 1942 and 1932 by Roethlisberger, 1977.

With threats to internal validity, we would also have to take into consideration the threats to external validity. These are the generalised relationship of the experiment with the situation at hand such as how representative the sample is, the conditions of experiment, treatment implementation or the end results. Jackson (2003) explains that to resolve this is the ability to replicate the experiment with other randomly selected group of test participants using a similar setting. The researcher may also choose to alternate the manipulation of the independent and dependent variables with concept replication.

In replication of an experiment, when manipulating the independent variable, the dependent variables would need to say constant similar to the experiment the researcher is replicating vice versa. Threats to external validity are almost similar to internal validity whereby the history, test participants behaviour, experimental settings or experimenter expectancies can affect the result outcomes. Shadish, Cook & Campbell (2002) describes it is known as the causal relationship of interactions between the variables.

Given this, true experiments may be accurately statistically analysed, however they may not be representative of the real world conditions. They are commonly impractical and expensive to undertake. Whilst during experiments, the researcher would be unable to determine how genuine are the results provided by his test participants under the experimental conditions set. This is due to the high degree of control and criteria implemented by the researcher set during the course of the experiment (Shaughnessy, et. al, 2006; Jackson, 2003). Nevertheless, even though randomly assigned, we can still ascertain its validity and reliability. They may not represent the true social population but its results can be a good starting point to theoretically assess the hypotheses.

Laboratory experiments can be brought out as true experiments in a natural setting to test its validity and reliability with regards to threats to its internal validity as tested in a controlled settings. It is the researcher’s prerogative to take note of all the external factors throughout the course of the experiment and analyse its statistical data. An experiment set at a natural setting would be more representative of a real world situation.

The choice to conduct a quasi experiment is when there is a lack in the ability of random assignment for a true experiment. Even with a limitation on sampling, this experimental design does at least resemble the social population. As such, a non-equivalent group design can be implemented. According to Trochim (2006), this is one of the most frequently used experimental designs in social research. Threats to its internal validity are highly affected by confounding factors unable to be controlled by the researcher similar to true experimental designs.

The main difference of a quasi-experiment with a true experiment is the lack of randomised participant selection. It is almost similar to true experiments yet lacks the degree of control over its internal validity. Pre-test and post-test assessments are conducted to provide plausible evidences to support the experiment outcomes. Thus the researcher must go in-depth in finding out every possible factor that would be a threat to the internal validity of the experiment for causal claims.

In conclusion, it is up to the researcher to consider the various factors in designing an experiment. They would need to take into consideration the many confounding variables encapsulating the independent variable to be tested on. No doubt that a randomised sampling population would be the best group to experiment on but to what extend and how does its surrounding environment and influences affect the resulting outcome. As such, in selecting an experimental design would need to fit the treatment and conditions.