Machine learning in systematic reviews: comparing automated text clustering with Lingo3G and human researcher categorization in a rapid review

Abstract

Systematic reviews are resource-intensive. The machine learning tools beingdeveloped mostly focus on the study identification process, but tools to assistin analysis and categorization are also needed. One possibility is to useunsupervised automatic text clustering, in which each study is automaticallyassigned to one or more meaningful clusters. Our main aim was to assess theusefulness of an automated clustering method, Lingo3G, in categorizing stud-ies in a simplified rapid review, then compare performance (precision andrecall) of this method compared to manual categorization. We randomlyassigned all 128 studies in a review to be coded by a human researcher blindedto cluster assignment (mimicking two independent researchers) or by a humanresearcher non-blinded to cluster assignment (mimicking one researcherchecking another's work). We compared time use, precision and recall of man-ual categorization versus automated clustering. Automated clustering andmanual categorization organized studies by population and intervention/con-text. Automated clustering failed to identify two manually identified categoriesbut identified one additional category not identified by the human researcher.We estimate that automated clustering has similar precision to both blindedand non-blinded researchers (e.g., 88% vs. 89%), but higher recall (e.g., 89%vs. 84%). Manual categorization required 49% more time than automated clus-tering. Using a specific clustering algorithm, automated clustering can be help-ful with categorization of and identifying patterns across studies in simplersystematic reviews. We found that the clustering was sensitive enough togroup studies according to linguistic differences that often corresponded to themanual categories