Context dependence of bias in randomized controlled trials: a Bayesian meta-epidemiological study from the BFREE project

Abstract

Whether the magnitude of bias in randomized controlled trials (RCTs) varies systematically across study contexts remains unclear. We investigated how outcome type, intervention type, and comparator type influence bias-related inflation of treatment effect estimates in RCTs. Meta-analyses from the ROBES study were used. Analyses were conducted within six subgroups defined by outcome type (subjective vs. objective), intervention type (pharmacological vs. nonpharmacological), and comparator type (active vs. inactive). To further assess context dependence, secondary analyses stratified these subgroups into all two-way and three-way combinations of outcome, intervention, and comparator types. For each analysis, we fitted Bayesian hierarchical models to estimate ratios of odds ratios (RORs), quantifying the degree of effect estimate inflation in RCTs judged to be at high or unclear risk of bias due to inadequate sequence generation, allocation concealment, or blinding. Models assumes random interventions effects within meta-analysis. Subgroup differences, expressed as ratio of RORs (RRORs), and interactions, expressed as ratio of RRORs (RRRORs) were also estimated. For sequence generation and allocation concealment, there was strong evidence of greater bias in pharmacological than nonpharmacological interventions (RROR 1.14, 95% CrI 1.00–1.32, and RROR 1.16, 95% CrI 1.01–1.34, respectively). For sequence generation and allocation concealment, there was strong evidence of greater bias in inactive than active comparators overall (RROR 1.14, 95% CrI 0.99–1.32, and RROR 1.22, 95% CrI 1.08–1.39, respectively). No contrast showed strong evidence of greater bias in either subjective or objective outcomes for sequence generation, allocation concealment, or blinding. However, intervention and comparator type appeared to differentiate bias mainly among objective outcomes. Strong evidence of interaction was observed in the sequence generation domain, between comparator type and intervention type (RRROR 1.30, 95% CrI 0.96–1.78), indicating that their effects on bias were mutually dependent. Bias associated with methodological flaws in RCTs appears highly context-dependent and may be driven mainly by intervention and comparator type when randomization is inadequate. Future studies should therefore account for the joint influence of outcome, intervention, and comparator type when interpreting bias. Our results may help refine risk-of-bias assessment tools and will inform future work on bias-adjusted meta-analysis within the BFREE Project.