Comparative evaluation of the reliability and validity of three data extraction programs: UnGraph, GraphClick, and DigitizeIt

Abstract

The use of evidence-based practices in education has been gaining a lot of attention in recent years. Researchers often use meta-analyses to identify evidence-based practices. To conduct meta-analyses of studies employing single-subject experimental research (SSER) designs for the purpose of identifying evidence base for a practice, a necessary step is to obtain raw data from published graphs. One method for obtaining raw data from published SSER graphs is the use of computer programs specifically designed to extract data from graphs. Purpose of the present study was to examine the reliability and validity of three data extraction programs, Ungraph, GraphClick, and DigitizeIt, using 60 graphs obtained from 15 SSER studies focused on a practice. Three coders extracted data from the graphs using the three programs. Values extracted by each coder were compared to (a) each other (reliability) and (b) values reported in the original articles in which the graphs were obtained from (validity). Results showed that raw data from SSER graphs can be obtained reliably using all three data extraction programs and values obtained using the three programs are highly valid. These results suggest that researchers can use data extracted using these programs with a high level of confidences while concluding meta-analyses of studies employing SSER designs. Authors make recommendations for improving the accuracy of data extraction using the three programs. (C) 2015 Elsevier Ltd. All rights reserved.