Implementation of machine learning in evidence syntheses in the Cluster for Reviews and Health Technology Assessments: Final report 2020-2021

Abstract

Key Messages: Machine learning (ML) has the potential to increase the efficiency of evidence syntheses. During 2020- 2021, a team in the division for Health Services at the Norwegian Institute of Public Health, tested and documented pros and cons of using ML in various phases of the conduct of various evidence syntheses, and built employees’ competence in using ML. This report describes the work undertaken by the ML team, project results and lessons learned. The ML team focused attention on ML functions and systems available within EPPI Reviewer: Priority screening, Custom and Pre-built classifiers, RobotReviewer to assess Risk of Bias, Automatic text clustering, and Microsoft Academic Graph (MAG). We implemented ML functions across 19 project teams and trained 23 employees. We found that utilizing ML in our reviews increased speed, with no identified threats to methodological quality. Screening time was reduced by 60-90% in all projects. Automated study categorization – while applicable to a smaller range of projects – reduced manual time in this phase by 60-70%. ML can, and should, change usual project workflows. The review process can become less linear and more cyclical, and several tasks can be conducted in parallel. However, workflow changes are not insignificant for those involved, and future ML work would benefit from a structured approach to both change management and innovation diffusion. The report concludes with lessons learned and experiences gained. They shaped our proposals for future ML strategies, covering capacity-building, innovative activities, evaluation of effect, and workflow optimization.