Introduction: Research Information Services at Canada’s Drug Agency has adopted a test-and-learn program to evaluate the performance of automation or Artificial Intelligence (AI) tools for informational retrieval tasks in health technology assessment. Methods: We identified 51 promising tools using a targeted internet search, evaluated these tools with standardized criteria, tested 3 highly ranked tools against usual practice search approaches, developed implementation recommendations, and initiated a pilot to build custom AI tools for targeted search tasks with the potential for large efficiency gains. Results: Our program built an evaluation instrument for automated or AI search tools for use by any evidence synthesis producers. Comparative tool rankings allowed us to identify promising tools for further testing against usual practice. From our testing, we learned that available tools had low sensitivity for database searching but could assist with specific tasks (e.g., keyword generation for strategy development) for modest efficiency gains. Discussion: Our ongoing work builds on these findings to target AI app use for grey literature searching where we anticipate larger efficiency gains than with database searching. We are building custom AI tools to implement recommendations and to semi-automate established search approaches using checklists for grey literature.