ontological and epistemological assumptions of the survey method

Inquiry paradigms, the basic belief systems or world views of the researcher frame the course of research and its outcomes. According to Guba (1990), paradigms can be characterised through their: ontology (What is reality?), epistemology (How do you know something?) and methodology (How do go about finding out?). Qualitative and quantitative research methods are underpinned by different ontological or epistemological assumptions; these are assumptions that are made about the nature of social reality and how we acquire these assumptions to be true, respectively.

Wand and Weber (1993:220) refer to ontology as “a branch of philosophy concerned with articulating the nature and structure of the world”. Ontology primarily addresses the question ‘what exists?” and attempts to comprehend the kinds and structures of objects, properties, events, processes and relationships in every area of reality (Welty &Smith, 2001). There are three ontological paradigms; Positivism, interpretivism and critical theory. In the MRC-CFAS survey, a positivist approach is taken, with the assumption that Alzheimer’s disease exists as a distinct condition in a world populated by human beings who have their own thoughts, interpretations and meanings. Further to this the paper states, “it is reasonable to anticipate di¬€erences in potentially associated syndromes such as dementia and cognitive impairment within England and Wales” – the MRC-CFAS assume that these syndromes may be quantified and can be generalised. There is an objective reality. The researcher makes the realist assumption that sampled units’ beliefs, thoughts and desires are real and not simply social constructs. As a result of these beliefs, the researcher can hold their own view as to what ‘cognitive function’ consists of and the meanings behind it based on personal reflection about the nature of cognitive function. They are also able to propose a research question that examines the prevalence of dementia as a real entity.

Underlying ontology is epistemology. Epistemology is concerned with deciding what knowledge is valid and appropriate within our ‘reality’. Asking the questions; ‘What are its sources?’, ‘What is its structure, and what are its limits?’ and ‘How can we use our reason and other resources to acquire knowledge’. As the MRC-CFAS study is grounded in positivism, it is assumed an objective reality can be acquired by the researcher, if he or she uses the correct methods and applies those methods in a correct manner, such as in physical science (Cohen and Crabtree 2003). “This tradition may therefore be characterized in terms of the prediction, explanation of the behaviour of phenomena and the pursuit of objectivity” (May 1997). Indeed we see that the researcher makes ‘a posteriori ‘justifications on the method (The Canadian Study of Health and Aging, 1994) to find causality, effects, and explanations through the evidence based they have accumulated from previously published research. The positivist paradigm enables the researchers to compare their claims and ascertain truth. The survey method uses repeatable surveying, to eliminate subjective bias, in which the researcher is independent and detached from the unit being measured. The first interviews (prevalence screen) established level of cognitive performance and baseline risk factors on all individuals .A 20% subsample for diagnostic assessment was selected to include name cases plus randomly selected individuals and they were followed up by sequential observations on an annual or biannual basis. These individuals were selected on the basis of cognitive function. The interviews were designed to be sensitive to dementia but were lacking in specificity. After 2 years, 80% of the participants who had not been identified as potential cases of dementia at the first interview were retested. A further 20% subsample was assessed one month later and again followed up on an annual or biannual basis. Those that were identified as possible new cases of dementia then had a more detailed assessment. There is a determined criteria for the knowledge, with quantifiable scales by constructing a fixed instrument (a set of questions). They do not allow the questions to emerge and change, so the object can be researched without being influenced by the a priori judgements of the researcher. Any possible researcher influence can be anticipated, detected, and controlled.

Concisely define: (a) non-response bias, (b) sampling error, (c) validity and (d) reliability.

(5% marks each)

a) In surveys with low response rates, non-response bias can be a major concern. Non-response refers to the failure to obtain observations due to unit or item non response. Unit non response is when the sampled unit that is contacted may fail to respond, for example, they are unwilling or unable to participate in the survey. Item non response results as consequence of failure to answer all the survey questions “e.g., leaving just one item on a questionnaire blank, or responding to some questions by saying, “I don’t know,” while providing a valid response to other questions” (Berg, 2005, Kish 1965:532). As non-response increases, the potential for a biased sample increase and the significance of the conclusions drawn become weaker. Non-response is most likely when there are multiple stages or components of response, e.g. screener interviews, multiple respondents associated with a case, or more than one waves of data collection (Bose 2001).

b) Sampling error is the variability among the sample and the population. A sample is expected to mirror the population from which it comes; however, there is no guarantee that any sample will be precisely representative of the population. While the level of nonresponse does not necessarily translate to bias, large differences in the response rates of subgroups serve as indicators that potential biases may exist. Weights are then calculated based on the proportions in each sub-group and applied to the respondents to reflect the total sample population. By increasing the sample size, the sampling error can be minimized (Erzberger, 1998).

c) The concept of validity is traditionally defined as “the degree to which a test measures what it claims, or purports, to be measuring” (Brown, 1996, p. 231) allowing a measurement or evaluation of an objective reality. It is an objective measure of accuracy and refers the degree to which an observed result can be relied upon and not attributed to random error in sampling and measurement. There are 4 types of validity: internal, external, construct and conclusion validity

d) Reliability stems from the positivist scientific tradition and refers to the reproducibility or consistency of a measurement. For example, when measuring height it is expected that the scale will measure the same result on each measurement of the same object.

Identify and describe the different types of non-response bias in the study. (10% marks)

Attrition can be defined as the loss of relevant individuals occurring after definition of the population to be included in a study (Matthews et al, 2004). The study used geographical area as a sampling frame. Recruitment for all centres except Gwynedd was through the Family Health Service Authorities (FHSA) with the population being derived from general practitioner registration, inclusive of institutionalised individuals. A loss of subjects to recruitment and baseline data collection is often referred to as ‘Unit non-response’. Unit non-response takes place when a randomly sampled individual cannot be contacted, the unit has an inability to respond, there is a lack of co-operation (refusal), lack of interest (salience) or alternatively, there is dropout due to unit death. Of the sample frame (n=123,691), the study defines the eligible population as n=20,234, this was all those aged >64years on a specified date and residing in a named geographical area. Deaths and emigrations were flagged. Population based samples stratified to ages 65-74 years and 75 and above were taken to achieve the 2,500 interviews at each centre. The population that was screened at baseline was n=13,009. This accounted for an 80% response rate of the available population. There was considerable variability in the response rates across the centres. This may because of initial contact method, GP approval etc. As with any longitudinal study, the results obtained by the MRC-CFAS study may be affected by attrition and salience.

Item non-response occurs when certain questions in a survey are not answered by an eligible respondent, leaving gaps in the acquired data. This may be due to the action of the sample member (e.g. refusal to answer, or inability to understand the question posed); the action of an interviewer (e.g. failure to ask a question that should have been asked, or failure to record the answer adequately); or the survey design (e.g. poor routeing instruction). At baseline, there was a 97% completion rate (n=12,555) on the MMSE test. MMSE item non-response (missing variables) was linked to physically or sensory frailty. This suggests that subjects with dementia are less likely to comply and therefore cognitive compromise is a predictor of non-response. At wave one, the assessment interview on a 20% sub-sample of the respondents, was biased towards the cognitively frail.

What steps have the researchers taken to limit non-response bias? (10% marks)

The full audit trail, to wave 2 interview, of the 13,009 individuals who completed the wave 1 interview is shown in table 1. Non response bias was limited through the fieldwork screening interview taking place in the respondent’s place of residence or the institution in which the resided, using portable computers with software customised centrally by the MRC Biostatistics Unit. This makes the survey more accessible to those that may be unable to attend a community setting. Hox (1994) noted that face-to-face surveys typically yield higher response. Investigation into the characteristics of those lost to follow-up after initial enrolment was not reported. Respondents were asked whether they were willing to be interviewed and if they accepted, participants were selected to form a subsample (20%) skewed towards information-rich participants displaying cognitive impairment. This is referred to as being selected through “snowball” or “chain sampling” (Patton, 1990), which “identifies cases of interest from people who know people who know people who know what cases are information-rich, that is, good examples for study, good interview subjects” (Patton, 1990, p. 182). Using volunteer groups minimises the longitudinal dropout. The MRC Biostatistics software was capable of providing a ‘priority mode,’ consisting of a very short subset of questions if the respondent was confused or demented or too frail to answer the complete set. This could be invoked automatically or manually by the lay interviewer. Proxy informants were also utilised where the interview was not possible with the named participant, due to e.g. extreme confusion or frailty. Item non response was accounted for in the coding strategy. No assumptions were made about the likely answer and the full score was recorded as missing. `Don’t know’ and `no answer ‘ were assigned separate codes, but were recoded to zero for the analyses.

How representative are the estimates of the prevalence of cognitive impairment reported in this paper? (10%marks).

The MRC-CFAS study explored the cognitive impairment of respondents that were recruited from an age stratified sample of 6 UK sites. There was no heterogeneity social background, residential status (community or institutional) or health status there is across sites the study, so conclusions drawn from these variables are possible to generalise. The six centres were chosen to reflect the social economic variability across England and Wales, however it would be impossible to conclude that these centres would be as reflective as a nationally drawn sample. The sample frame was inclusive of institutionalised subjects who were more likely to show cognitive impairment. There is a high response rates at each stage over the geographical sites (mean=80%), with participation increasing with age.

Critically discuss the reliability (20% marks) and validity (20% marks) of the data-collection instruments used to assess the cognitive status of study participants.

Structured interviews aid reliability by presenting all subjects with a standardized stimulus. Lack of reliability may arise from deviations between interviewer (such as prompting) or deviations of the instrument measurement. The testing procedure may be called into question as the administration and scoring of the tests can vary between different individuals. Sampling is imperative to external validity: a researcher must be able to claim that the participants included in the study are representative of the overarching population from which the sample is derived. If the measured sample is representative of the general population you can automatically generalise your results back to the population and to other contexts. However, proxy screening interviews were conducted where an interview was not possible with the named participant – this brings into question the impact of patient-proxy bias. As such, the researcher did not remain explicitly independent. Construct validity seeks agreement between a theoretical concept and makes comparison to the perceived ‘gold standard’. The construct validity of the MMSE has been well studied (Jefferson et al, 2002:1) and validated in a number of population studies (Folstein, 1975). Furthermore, the method used by the MRC CFA mimics the methodology of the Canadian Study of Health and Aging (1994). Conclusion validity is the degree to which conclusions we reach about relationships in our data are reasonable. There is inherent bias in using the MMSE test as it will not detect subtle memory losses, particularly in in certain unit groups resulting in Type 1 error. Type I error concludes there is a relationship when in fact there is not. This may be due to the emphasis placed on language items and a lack of items assessing visual-spatial ability. Indeed, the MMSE instrument relies heavily on verbal response and reading and writing. Therefore, patients that are hearing and visually impaired, intubated, have low English literacy, or those with other communication disorders may perform poorly even when cognitively intact. Adjustment must be made in light of social status and education level, otherwise those with low education level may show poor results despite not having cognitive impairment and those with high education and pronounced cognitive results may show favourable results that ultimately mask the problem. It is also not clear how one deals with answers that are “near misses.”

Angen, MJ. (2000). Evaluating interpretive inquiry: Reviewing the validity debate and opening the dialogue. Qualitative Health Research. 10(3) pp. 378-395.

Burell, G. & Morgan, G. (1979). Sociological paradigms and organizational analysis. London: Heinemann.

Cohen D, Crabtree B. “Qualitative Research Guidelines Project.” July 2006. http://www.qualres.org/HomePosi-3515.html. Assessed 13/12/10

Creswell, JW. (1998). Qualitative inquiry and research design: Choosing among five traditions. Thousand Oaks: Sage Publications.

Folstein MF, Folstein SE, McHugh PR. Mini mental state: A practical method for grading the cognitive state of patients for the clinician. J Psych Res 12:189-198, 1995.

Guba, E. G., & Lincoln, Y. S. (1994). Competing paradigms in qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (pp. 105-117). Thousand Oaks, CA: Sage.

Hirst G. 1989. Ontological assumptions in knowledge representation. In Proceedings of the first international conference on Principles of knowledge representation and reasoning, Ronald J. Brachman, Hector J. Levesque, and Raymond Reiter (Eds.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 157-169.

Hox, J.J. and de Leeuw, E.D. (1994). A Comparison of Nonresponse in Mail, Telephone, and Face-to-face Surveys: Applying Multilevel Models to Meta-analysis. Quality and Quantity, 28, 329-344.

Israel, GD. (1992) Determining Sample Size. Program Evaluation and Organizational Development, IFAS, University of Florida. PEOD-6. November.

Israel, GD. (1992) Sampling The Evidence Of Extension Program Impact. Program Evaluation and Organizational Development, IFAS, University of Florida. PEOD-5. October.

Kish, L (1965) Survey Sampling. New York: John Wiley and Sons, Inc.

Matthews F E, Chatfield M, Freeman C, McCracken C, Brayne C and MRC CFAS, (2004) Attrition and bias in the MRC cognitive function and ageing study: an epidemiological investigation, BMC Public Health, 4:12

Mattingly S,Sheikh K (1981) Investigating non-response bias in mail surveys, Journal of Epidemiology and Community Health;35:293-296

May T. (1997) Social research. Issues, methods and process, 2nd ed. Buckingham: Open University Press,

Miaoulis, G. Michener RD, (1976) An Introduction to Sampling. Dubuque, Iowa: Kendall/Hunt Publishing Company

MRC CFAS Medical Research Council Cognitive Function and Ageing Study Group. (1998) ‘MRC CFAS Medical Research Council Cognitive Function and Ageing Study Group: Cognitive function and dementia in six areas of England and Wales: the distribution of MMSE and prevalence of GMS organicity level in the MRC CFA study’, Psychological Medicine, 28, pp. 319-335.

Popper, K. (1972). Objective Knowledge. Oxford: Oxford University Press.

Swepson, P. (1995) Action research: understanding its philosophy can improve your practice [On line]. Available at http://www.scu.edu.au/schools/gcm/ar/arp/philos.html

Wand, Y. Weber, R. 1993. On the ontological expressiveness of information systems analysis and design grammars. Eur. J. Inf. Syst. 3, 217-237.

Welty, C. and Smith, B. (eds.) 2001 Formal Ontology and Information Systems, NewYork: ACM Pres