Background: Systematic reviews (SRs) can be retrieved in several ways. One approach is a search filter designed to retrieve SRs in bibliographic databases such as PubMed. Another utilizes machine learning (ML) to sort articles into two mutually exclusive classes in online platforms such as DistillerSR.
Objective: To test the performance of a binary ML classifier designed to identify SRs in DistillerSR against search filters designed to retrieve SRs in PubMed.
Methods: Umbrella reviews will be identified, included SRs will be extracted to create a reference set. Each SR in the reference set will be verified as indexed in PubMed. Two SR search filters will be tested: a narrow SR filter and a broad SR filter. A binary ML classifier developed by DistillerSR to identify SRs will be used, informed by the reference set and a seed of non SR articles. The number of SRs from the reference set retrieved by the two search filters, and the number of SRs from the reference set correctly identified as a SR by the ML classifier will each be recorded discretely.
Results: Relative recall, precision and F1 score will be calculated for each set. Recommendations will be made on when to apply a search filter or ML classifier to a search.
Conclusions: Different approaches to limiting search results to specific study designs can influence the overall search strategy. The objectives of the project and resource requirements need to be considered when deciding on the approach.