Background: We are undertaking a Cochrane review evaluating the effect of supplementary search methods compared to bibliographic searching to identify randomised studies.
Cochrane requires an appraisal of the risk of bias of studies included in their systematic reviews. The standard tool recommended by Cochrane (RoB 2) covers interventions evaluated in randomised studies. This ‘type’ of study does not align with the information retrieval studies we anticipate finding (comparative case studies), so we designed our own risk of bias tool specific to information retrieval (IR) studies.

Methods: We undertook a review of the following risk of bias tools identified via the Latitudes website.(1) The aim of the review was to determine the applicability of these tools to IR studies:
• RoB 2
• EPOC – suggested risk of bias criteria for EPOC reviews [proposed criteria for RCTs, non-RCT, and controlled before-after]
• ROBIS
• Newcastle-Ottawa Quality Assessment Scale Case Control Studies
• Scale for the Assessment of Narrative Review Articles (SANRA)

This review was then supplemented by a methodological review by Tomlinson and colleagues who evaluated common challenges and suggestions for risk of bias tool development.(2)

Results: Our proposed tool has three domains: review of study protocol, the comparison, and outcomes. We provide guidance on how to judge risk of bias, adopting the signalling questions from the ROBIS tool.

Conclusions: This is the first tool to appraise risk of bias in IR studies. We will report the tool in full for the first time, providing a worked example of the tool and findings.

References;

1. LATITUDES Network. LATITUDES Network. 2017. URL: https://www.latitudes-network.org/about/ (Accessed September 2024).

2. Tomlinson E, Cooper C, Davenport C, Rutjes AWS, Leeflang M, Mallett S, et al. Common challenges and suggestions for risk of bias tool development: a systematic review of methodological studies. Journal of Clinical Epidemiology 2024;171111370