The Likert Scale and Manipulation Checks in Research

The independent variable is the status and the dependent variables are rule compliance and hierarchy-enhancing beliefs.

The study collects the data through an online survey, uses a likert scale, and incorporated manipulation checks into its experimental design. It is a between subject design. Its weakness lies with its lowered external validity.

Firstly, the survey was conducted online. Hence, the computerized system would be able to manage the order in which question were shown, controlling the questions in which it was being answered (Wan, 2014). This also eliminates the need for researchers to code their answer into a format that can be analysed by statistical program which could sometimes results in mistakes when transferring the data from paper to software. It is also impersonal and can help decrease the effect of social desirability as compared to a scenario in which the survey was administered by the experimenter (Wan, 2014).

Secondly, the study uses a likert scale. Dependent variables are measured with the use of a likert scale instead of a simple dichotomous categorical answer. This allows the researcher to measure the degree in which a particular construct is felt rather than it just being felt or not. This is important as the hypothesis wants to measure not just the presence of rule compliance and hierarchy-enhancing beliefs, but the level (“lower” vs “stronger”) of it.

Items on the measurement scale were being reverse coded and can help to counter acquiescent bias (Wan, 2014). The effect of acquiescent bias can be measure on a bipolar scale such as what was used for measuring the hierarchy-enhancing beliefs, from a “strongly disagree” to a “strongly agree” (Hinz, Michalski, Schwarz, & Herzberg, 2007). However, the scale testing the rule compliance construct is unipolar. It also uses multiple item scale. This helps to average out participant random error within a scale, making it more reliable (Wan, 2014).

The scale that was used to measure the hierarchy enhancing beliefs was used in past research to measure the same construct, increasing the confidence we have that they are more valid than other measurements of hierarchy enhancing beliefs (White & McBurney, 2013).

While the odd point likert scale has its advantages, it also has drawbacks. By doing using an odd-point scale, researchers did not force participants to take a side. Participants could clearly state that they feel neutral about an item rather than choosing one mild directional response over the other when they do not really feel that way. This will enable the scale to reflect what the participants really feel. However, the interpretation of mid-point is entirely up to the participants as there are only two anchors on the extreme end of the 5-point scale. One participant’s interpretation of the mid-point might not be the same as another participant. For example, one participant might choose the mid-point if they do not have an opinion about it, but another participant could also choose the mid-point if they do not want to think about it or just want to finish the survey faster. This could affect the reliability and validity of the instrument and result. The researchers could have put an anchor for the mid-point as well so that all participants can see the mid-point in the same way.

The participants could also use the mid-point as a way of avoiding decision. This is especially so since the scale measures hierarchy-enhancing beliefs and rule compliance, both of which could inflict a certain extent of social desirability on the participants. Furthermore, the scale was put on a survey which essentially is a self-reported measure and they can choose whichever choice they feel like (Wan, 2014). This will make the effect of social desirability even stronger. The scores measured might not be a representation of what they actually feel, but a reflection of what they feel is morally right.

Thirdly, a manipulation check was done as they asked participants to “describe what happened and how they felt in the situation”. This can help provide evidence for construct validity of manipulation and help make sure that status have indeed been manipulated (White & McBurney, 2013). This will help ensure that the status manipulation has been properly executed, and the results had been obtained in the presence of successful manipulation and not due to chance. This is especially important for participant manipulation as the manipulation might not have the intended effect on the participant as it is subjected to the participant’s interpretation (Gravetter & Forzano, 2011).

Fourthly, as participants are only being manipulated to experience one condition (either a high status or low status condition), the study uses a between subject design. Researchers can be sure that the scores will not be influenced by factors like sensitization of knowing what is being measured and as a result, the possibility of having a good-subject tendency (Wan, 2014). Participants will also not experience the effects of order effect such as carryover of perceived status as compared to a study that uses a within subject design. However, as the individuals cannot act as their own control as they can in a within subject design, the study is subjected to individual differences such as family upbringing, emotions and personality, all of which can influence the result (Wan, 2014). Individual differences can produce high variability in the scores (Gravetter & Forzano, 2011). They can also become confounding variables (University of Arizona, 2006). The study limited the confounding through randomization (University of Arizona, 2006). The participants were randomly assigned to either of the two conditions. This helps to reduce or eliminate the systematic differences between the two groups, achieving group equivalence. Hence, the internal validity of the study, which is important for the study as it is trying to establish relationships between two variables, is increased (White & McBurney, 2013).

Furthermore, two dependent variables are being measured in one study. The effect of manipulation might have worn off or be diminished by the time participants answered the second survey on hierarchy enhancing beliefs. The order in which the surveys were given and filled up might have been important in the degree to which participants agree or disagree with the items in the survey.

However, the external validity might have been lowered as the study might have a biased sampling frame and the method of selection of participants was not mentioned.

Firstly, the study could have a biased sampling frame. Unless the market research firm has a wide reach of participants that spread across the country, they had only sampled from a small set of cities and towns in the whole of America. By systematically excluding all US working adults who lives in other parts of United States of America (USA), valid generalizations about the entire American population could not be drawn from this data and results (Wan, 2014). This is assuming that the population that the study is investigating is the whole of USA. The assumption was made as the hypothesis did not specify the population it will be investigating. By stating “people”, it could mean people around the world, people in America, or people in a specific state.

Secondly, random selection of the sample was not mentioned. Hence, it might not be representative of the population, lowering the external validity of the study (Wan, 2014). However, this might not be that important as the goal of the study was to establish a relationship between the two variables and not to generate a result that could be generalized across the population. Since, there might be limited resources, it is wise to only choose one validity; internal validity was more important than external validity in this study that investigates the relationship of variables (Wan, 2014). This weakness will not affect the conclusion that much.

In conclusion, the use of online survey, likert scale, and manipulation checks had made the study better. However, the reduced external validity might make it difficult for the results of this study to be generalized.

• Tan Jun Xuan

References:

Gravetter, F., & Forzano, L.-A. (2011). Research Methods for the Behavioural Sciences. United States of America: Cengage Learning.

Hinz, A., Michalski, D., Schwarz, R., & Herzberg, P. Y. (2007). The acquiescence effect in responding to a questionnaire. Psychosocial Med.

University of Arizona (2006). Ch8 Between-subjects design [Powerpoint slides]. Retrieved from: www.u.arizona.edu/~dusana/psych290Bpresession06/notes/Ch8%20Between-subjects%20desing.ppt

Wan, C. (2014). HP1100 Sem1 1415 Week 2 Slides [Powerpoint slides]. Retrieved from: ntulearn.ntu.edu.sg

Wan, C. (2014). HP1100 Sem1 1415 Week 3 Slides [Powerpoint slides]. Retrieved from: ntulearn.ntu.edu.sg

Wan, C. (2014). HP1100 Sem1 1415 Week 4 Slides [Powerpoint slides]. Retrieved from: ntulearn.ntu.edu.sg

White, T. L., & McBurney, D. H. (2013). Research Methods. United States of America: Wadsworth, Cengage Learning.