Validierung eines Tools zur Beurteilung des Risk of Bias in symptomevaluierenden Studien
Loading...
Files
Date
Authors
Publisher
Abstract
Background
Systematic reviews aim to bring together all available evidence on a specific
research question and thus to present reliable findings. The Validity of these results
directly depends on the validity of the primary studies’ data. Therefore, assessing
study quality respectively risk of bias is an important step in conducting a
systematic review. As part of the project “Systematic Reviews of Symptom-
Evaluating Studies”, a tool for assessing risk of bias in symptom-evaluating studies
has been developed at the Institute for General Medicine at the University of
Marburg.
Studies of symptoms investigate a defined symptom regarding its prevalence in a
specific setting, the underlying conditions and the prognosis of the affected patients.
The aim of this dissertation is to validate the new Risk of Bias Tool.
Methods
The Risk of Bias Tool consists of four domains with a total of sixteen signaling
questions/ items, which are answered with “yes”, “no”, or “unclear”. The items
refer to either the appraisal of the risk of bias (internal validity), the risk of
variation (external validity/ applicability to the review’s research question) or the
reporting quality of the primary studies (information).
In order to test the tool’s reliability, two reviewers independently rated the quality
of 101 studies using the Risk of Bias Tool. Each team initially rated a few studies
during a training phase. Afterwards, uncertainties were discussed and resolved.
Interrater agreement was calculated as percentage agreement p0. Cohen’s Kappa
coefficient was additionally calculated to account for agreement by chance.
A hypothesis-testing approach was used to test the tool’s validity. Assuming that
the presence of systematic error leads to biased results, it was hypothesized that
studies with a high risk of bias according to the tool would show different results
– such as higher prevalences – compared to studies with a low risk of bias.
I² and tau² were calculated as measures of heterogeneity. When comparing the
results of all studies, regardless of their assigned risk of bias, to the results of only
those studies judged to have low risk of bias, the latter should be more
homogeneous, meaning they should yield lower values for I² and tau².Results
Interrater agreement for the entire tool was 73.5 %. Kappa amounted to 0.56
(moderate agreement). In the training phase, an agreement of 65.9 % (κ =0.45) was
achieved, and in the following routine phase, 79.0 % (κ = 0.64, good agreement).
For the individual items, values ranged from 58.8 % to 100 % with corresponding
kappa values ranging from 0.00 to 0.74.
For the heterogeneity analysis, only a few sufficiently comparable studies could be
identified. In five out of nine cases considered, a reduction in heterogeneity was
observed in the study groups with low risk of bias.
Discussion
The interrater agreement results are comparable to those of other risk of bias tools.
They show that an initial training phase with close communication may lead to
improved reliability. However, the results also show that despite those measures,
only a certain level of agreement is achieved. For this reason, the Risk of Bias Tool
does not give a definite judgment on the quality of a study but serves as guideline
for a structured quality assessment and as a basis for discussion among the review
authors.
When comparing risk of bias and heterogeneity, the expected effect could be
observed in several cases, especially where a larger number of comparable studies
was available.
Conclusion
The Risk of Bias Tool allows for structured assessment of the risk of bias in
symptom-evaluating studies. A trial run with approximately ten studies is
recommended to gain confidence in using the tool and to uncover and resolve any
ambiguities.
Review
Metadata
License
This item has been published with the following license: In Copyright