Validity and Reliability of Systematic Review

Abstract

Systematic review with meta-analysis are considered more objective than other types of reviews such as traditional reviews because it involve the application of scientific strategies in ways that limit the bias but the interpretation of the systematic process like any other type of research is subject to bias and this articles will illustrate the sources of bias in every step of conducting a systematic review and what is its types and ways.

Keywords: Systematic review, Bias, Meta-analysis

Introduction

A systematic review is an overview of many studies that used clear and reproducible methods while a meta-analysis is a mathematical synthesis of the results of two or more primary studies that address the same hypothesis in the same way.

Get Help With Your Essay

If you need assistance with writing your essay, our professional essay writing service is here to help!
Find out more about our Essay Writing Service

Systematic reviews are very popular so about 2500 new English language systematic reviews are indexed in Medline annually (Mother D, Tetzlaff J, Tricco a, et al). While the number is impressive ,the quality of their reporting is not always ideal which lead to some kind of biased results and thus shrink their usefulness.

Although meta-analysis can increase the precision of a result, it is important to ensure that the methods used for the review were valid and reliable. (Greenhangh. 1997)

Speaking generally,there are two sources -at least- can generate bias in systematic reviews: the risk of bias in the included studies which it can exaggerate the results of a treatment’s effectiveness by 18% (Pidal J, Hrobjartsson A, Jorgensen KJ, et al) and the review itself as it has a little control over the reporting of RTC’s but it can apply considrable control over conducting and reporting the review, thereby minimizing the bias of review itself.

In this article we will try to spot the sources of bias in every step of conducting a systematic review and what is its types and ways and after that we will talk in details about each factor might cause a bias including publication bias, time lag bias, citation bias, the influence of external funding on the validity of systematic review and outcome reporting bias.

Assessing the Quality of a Systematic Review

General Tips (step-by-step)

Fundamentally, the quality of a systematic review and the reliability of its result are contingent on both the quality of the included studies and the quality of the methodology used to produce the systematic review.

The first most important step in conducting systematic review is proposing a clear, specific,focused and concise question which will guide the review process after.

Searching for articles to be included can be retrieved by electronic databases, searching by hands through appropriate journals and by contacting researchers in the area of interest. To avoid the bias in the retrieval of articles the search strategy specified in the protocol must include as much details as possible. In most cases this amounts of to a list of keywords and how they will be combined for use in electronic search engines. Some knowledge of the capability of each subject specific database is important at this point, as some databases operate a thesaurus search system and others operate on the basis of keywords only.

Next step, selective inclusion studies may bias the results of systematic reviews if selected based on report characteristics which called ‘Biased inclusion criteria’ and low methodological quality of studies included in a systematic review is another important source of bias (Strerne JAC, Egger M, Smith GD 2001) and inclusion of data from sources other than randomized trials reduces the reliability of the conclusions of a systematic review on issues of prevention and treatment, so they should be thoroughly considered and properly defined to avoid ambiguity and to inform the validity of the review. As protocol availability may decrease the biased post-hoc changes to methods and selective outcome reporting, this information should be included in the review protocol to minimize this bias.

Even if the study has high internal validity, it may not to be generalizable(high external validity). There is often a trad-off between internal and external validity. To decide about the generalizability of the study is to explore whether the study population appears to be representative of the population to which you wish to apply the results and even in similar populations, differences in the settings and in culture or other contextual factors, should also be considered.(Petticrew M, Robert, H 2008)

Reviewing the results of a number of studies of course itself provides a est of generalizability, if the results have been replicated in several settings with different population, then this gives an indication of whether the results are transferable. If the number of studies is large enough, it can suggest the range of effect sizes to be expected in different settings. Generalizability is not often assessed separately in systematic reviews, though consideration of the issue is included in some critical appraisal checklists. (Deeks J, Dinnes J, D’Amico R, Sowden A, Sakarovitch C. 2003)

The risk of bias of a particular study is a key component in the assessment of studies that affect the validity of the results of a systematic review. Therefore, reducing the risk of bias assessment can be completed by using scales, checklists and every individual component should be reported for each study.(Sandrson S, Tatt ID, Higgins JP 2007).

As the protocol developing, all the outcomes derived from the included studies should be considered and the outcome of primary importance should be differentiated from the secondary outcomes as recent surveys have showed that the outcomes selectively reported in final reports were significantly more likely to be statistically significant than those omitted (Chan AW, Hrobjartsson A, Haahr MT, et al 2004). Therefore, if a review does not identify important variables clearly, the review risks being subject to bias.

So, the reviewer might select statistically significant variables and ignore the ones were initially important by the reviewer.

When it comes to analysing data, the analysing method is determined by the review question and the type of data collected and it should include a narrative synthesis for describing the results and risk of bias. The next step is usually determining if statistical synthesis is appropriate to apply or not. Indeed, such forced analysis might in the axiom ‘garbage in garbage out’, providing useless results which it will discussed Later.

When the results of the analysis are ready, there are many different way to represent them but sufficient details should be presented to dtermine the potential threats to validity.

As conclusion in the review, the reviewer should discuss the risk of bias, strength, limitation, weakness and applicability of the evidence for each main outcome to ensure that clinicians have all the information to interpret the results. A table outlining the users’ guides to the Medical Literature highlight critical appraisal questions for systematic reviews and meta-analyses may help to reduce the bias in every step when conducting a systematic review. (Table 1)

1. Did the overview address a focused clinical question?
2. Were the criteria used to select articles for inclusion both defined and appropriate?
3. What is the likelihood that relevant studies were missed?
4. Was the validity of the included studies assessed?
5. Were the assessment reproducible?
6. How precise were the results of the overview?

In assessing the value the review, it is important to consider the following question:

1. Can the results be applied to my patients, and will the results help me care for my patients?
2. Are the benefits worth the harms and costs?

Garbage in – garbage out?

The quality of component trails is important as an example if the raw material is not that quality, then the findings of reviews may also be the same. So what we put in exactly what we get out. Clearly, the studies included in systematic reviews should ideally be of high methodological quality and free of bias as possible.

The biases that threaten the validity of clinical trials are relate to systematic differences in the patients’ characteristics at baseline (selection bias), unequal provision of care apart from the treatment under evaluation (performance bias), biased assessment of outcomes (detection bias) , and bias due to exclusion of patients after they have been allocated to treatment groups (attrition bias).(Altman 1991)

Some reviews produced discordant results precisely because the authors chose to ignore the quality of component trails. The same reviewers were considerably more thorough in their attempt to identify all-relevant trails,Independent of publication statue or language of publication.

Although the quality of component trails happened to be more important in this particular situation, the dissemination of findings from clinical trials is known to be biased, and a comprehensive literature search is an essential intergradient of high-quality reviews. (Eddger,M. Dickersin, K. Smith,G, S 2001)

Putting the light on dissemination of research findings, Scherer et al. showed that only about half of abstracts presented at conferences are later published in full.

The fact that sustainable proportion of studies remains unpublished after the study had been completed must be a concern as a large information remains hidden from reviewers. Making things worse, the dissemination of research findings is not a random process, rather it is strongly influenced by the nature and direction of results. (Eddger,M. Dickersin, K. Smith,G, S 2001)

Type of reporting bias : Definition

Publication bias : The publication or non-puplication of research findings, depending on the nature and direction of results

Time lag bias : The rapid or delayed publication of research findings, depending on the nature and direction of results

Duplicate publication bias : The multiple or singular publication of research findings, depending on the nature and direction of results

Citation bias : The citation or non-citation of research findings, depending on the nature and direction of results

Language bias : The publication of research findings in particular language, depending on the nature and direction of results

Outcome reporting bias : The selecting reporting of some outcomes but not others, depending on the nature and direction of results

Publication Bias

In a 1979 article on “the ‘file drawer problem’ and tolerance for null results” Rosenthal said, where “the journals are filled with the 5 per cent of the studies that show type I errors, while the file drawers back at the lab are filled with the 95 per cent of the studies that show non significant results. (Rosenthal R. 1979). The file drawer problem has long been recognized in the social sciences: as a review of psychology journals found that of 294 studies published in 1970s, 97% rejected the null hypothesis at the 5% level.(Sterling TD.1980)

It is thus possible that studies which suggest a beneficial treatment effect are published, while an equal mass of data pointing the other way remains unpublished. In this situation, a systematic review of the published trails could identify a spurious beneficial treatment effect, or miss an important adverse effect of a treatment. In the field of cancer chemotherapy such publication has been demonstrated by comparing the result from studies identified in a literature search with those contained in an international trials registry (see figure 1.2). (Simes RJ. 1986)

Time lag bias

Published studies continued to appear many years after approval by the ethic committee. Among proposals submitted to the Royal Prince Alfred Hospital Ethics Committee in Sydney, 85% of studies with significant results as compared to 65% of studies with null results had been published after 10 years (Stern JM, Simes RJ. 1997). The average time to publication was 4.8 years for studies with significant results comparing to 8.0 years for studies with null results. In fact, the time lag was attributable to differences in the time from completion to publication.(Eddger,M. Dickersin, K. Smith,G, S 2001)

These findings indicate that time lag bias may be introduced in systematic reviews even when most or all trails will eventually be published. Trails with positive results will dominate the literature and introduce bias for several years until the negative results finally appear.

The influence of external funding and commercial interests

Many systematic reviews are funded by organizations such as pharmaceutical companies. As in the design of randomized trials, the design of systematic reviews can be influenced (particularly through manipulation of inclusion and exclusion criteria) to select a particular set of studies. As a result, such systematic reviews may present a biased viewpoint. Careful assessment of the quality of the systematic review should reveal the flaws in their design. Another way in which bias can be introduced is through biased interpretation of the results of a systematic review funded by industry or authored by investigators who are influenced by industry.(Crowther, MA. Cook, DJ 2007)

External funding was associated with publication independently of the statistical significance of the results. Funding by government agencies was significantly associated with publication in three cohorts of proposals submitted to ethics committees whereas pharmaceutical industry sponsored studies were less likely to be published in two studies. Indeed, the pharmaceutical industry tends to discourage the publication of negative studies which it has funded. (Eddger,M. Dickersin, K. Smith,G, S 2001).

Duplicate publication bias

Once a list of articles is obtained, they should be reviewed by two or more individuals and compared with a list of pre-developed inclusion and exclusion criteria The production of multiple publications from single studies can lead to bias in a number of ways. Two or more systematic reviews on the same topic may arrive at different conclusion, which raise concern about validity. Studies with significant results are more likely to lead to multiple publications, which makes it more likely that they will be located and included in a meta-analysis.

Moher and Johansen and Gotzsche described the difficulties caused by redundancy and the “disaggregation” of medical research when results from multicentre trail are presented in several publications. It may be impossible for reviewers to determine whether two papers represent duplicate publications of one trail or two separate trails, since example exist where two articals reporting the same trail do not share a single common author. (Eddger,M. Dickersin, K. Smith,G, S 2001).

Citation bias

The perusal of the reference lists of articles is used to identify additional articles that may be relevant. The problem with this approach is that the act of citing previous work is far from objective and retrieving literature by scanning references lists for many possible motivations such as decoration and showing up-to-dateness and knowledge may thus produce a biased sample of studies. (Eddger,M. Dickersin, K. Smith,G, S 2001)

Language bias

Language bias still evident in many reviews (Hearther, 2009). Reviewers are often exclusively based on trails published in English. For example, among 36 meta-analyses reported in leading English-language general medicine journals from 1991 to 1993, 26 had restricted their search to studies in English language. Reviewers in other countries will published their work in local journals as well as English language journal if their results are positive while negative results will just be published in local journals.

This is demonstrated for the German language literature when comparing articles published by the same author, 63% of trails published in English had produced significant results as compared to 35% of trails published in German. Thus bias could be introduced in meta-analyses exclusively based on English-language reports.(Figure 1.3) (Eddger,M. Dickersin, K. Smith,G, S 2001)

Outcome reporting bias

Reporting the outcome can be influenced by the results: the outcome with the most favorable findings will generally be reported. (Eddger,M. Dickersin, K. Smith,G, S 2001).

The future of unbiased, systematic reviewing

Reporting biases is potentially serious problem for systematic review. While the Cochrane Collaboration has a simple aim -help people to make well informed dicisions about healthcare-, there are many challenges that must be met to achieve this aim.

Ethical and social challenges include finding ways to continue to build on enthusiasm while avoiding duplication and minimizing bias, to ensure sustainability and to accommodate diversity.

Logistical challenges include finding ways to identify efficiently trails and manage criticisms and updates of reviews.

Methodological challenges include developing sound guidelines for deciding what types of studies to include in reviews, effective ways of communicating the results of reviews and summarizing the strength of evidence for specific effects. (Eddger,M. Dickersin, K. Smith,G, S 2001).

Conclusion and Summary points

In summary, There are numerous ways in which bias can be introduced in reviews and meta-analysis of controlled clinical trials. All these biases are more likely to affect small studies therefore, their results need large treatment effect to be significant. On the other side, the large studies invest more money and time that means they are more likely to be high methodological quality and published even if their results are negative. Bias in a systematic review may become evident through an association between the size of the treatment effect and study size. Reliability and validity often not established within quality assessment instrument (Heather, 2009).If the methodological quality of trials is inadequate the findings of reviews of this materials may also be compromised. Publication bias can distort findings because trials with statically significant results are more likely to be published, and without delay, than trials without significant results. Among published trails, those with significant results are more likely to get published in English, more likely to be cited, and more likely to be published more than once which means that they will also be more likely to identified and included in reviews. The choice of the outcome that is reported can be influenced by the results. The outcome with the most favorable findings will generally be reported, which may introduced bias. Criteria for inclusion of studies into a review may be influenced by knowledge of the results of the set of potential studies. The definition of eligibility criteria for trails to be included, a comprehensive search for such tails, and an assessment of their methodological quality are central to systematic reviews. Systematic reviews are thus more likely to avoid bias than traditional, narrative reviews.(Eddger,M. Dickersin, K. Smith,G, S 2001)

Nevertheless, the systematic review is a powerful research methodology which answers question on the the basis of good evidence and provides researchers with a valuable, impartial, comprehensive and up-to-date summary of the work conducted in a specific area.

References

• Altman DG. Practical statistics for medical research. London: Chapman and Hall, 1991.
• Chan AW, Hrobjartsson A, Haahr MT, et al. Emperical evidence for selective reporting of outcomes in randomized trails. JAMA 2004;291:2457-2465.
• Crowther, MA. Cook, DJ Trails and tribulation of systematic reviews and meta-analyses. Hematology 2007:493-497
• Deeks J, Dinnes J, D’Amico R, Sowden A, Sakarovitch C. Evaluating non-randomized intervention studies. Health technology assessment 2005;7
• Egger M, Smith GD., Altman DG: Systematic reviews in health care. London: BMJ 2001.
• Greenhalgh T. How to read a paper: Papers that summarise other papers (systematic reviews and meta-analyses) BMJ 1997;315:672-675 http://www.bmj.com/cgi/content/full/315/7109/672
• Mother D, Tetzlaff J, Tricco a, et al. Epidemiology and reporting characteristics of systematic reviews. PloS Medicine 2007;4:e78.
• Petticrew M, Robert, H Systematic Reviews in the social sciences, a practical guide. Blackwell publishing 2008
• Pidal J, Hrobjartsson A, Jorgensen KJ, et al. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2007;36:847-857.
• Rosenthal R. The ‘file drawer problem’ an tolerance for null results. Psychological Bulletin 1979;86:638-641.
• Sandrson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidmiol 2007;36:666-676.
• Simes RJ. Confronting publication bias: a cohort design for meta-analysis. Stat Med 1987;6:11-29
• Stern JM, Simes RJ. Publication bias:evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315:640-5
• Sterling TD. Publication bias and meta-analysis. Evaluation Education 1980;4:22-24.
• Strerne JAC, Egger M, Smith GD . Investigating and dealing with publication and other biases in meta-analysis 2001;323:101-105