This document reports on a project aimed at answering the question of whether or not propensity score modelling can replicate the results from randomized controlled trials for criminal justice evaluations.
In this document, the authors report on a research project to determine whether or not propensity score modelling (PSM) can replicate the results of randomized controlled trials (RCTs) for criminal justice evaluations. For the research project, the authors set out to assess the reliability and validity of seven PSM methods in replicating the results of 10 criminal justice RCT experiments. In their research, the authors focused on the following five different PSM techniques: one-to-one matching, with and without a caliper; one-to-many matching, with and without a caliper; inverse probability of the treatment weighting (IPTW); stratified weighting scheme; and optimal pairs matching. The researchers gathered the datasets of 10 publicly available and restricted RCT studies from the National Archive of Criminal Justice Data (NACJD), introduced an artificial selection bias into the treatment groups of the investigations, and then used each PSM technique to remove the selection bias. The researchers then compared the results generated from the PSM methods to those derived from the original RCT experiments and meta-analyzed the findings across all studies to reveal the true reliability and validity of PSM in relation to RCTs using criminal justice data. Results indicated that there is sufficient support for the use of PSM in criminal justice research, and the authors note that their research demonstrated that those seven PSM methods can be an effective means for estimating reliable and valid simulation of RCT experiments.