Experimental Evaluation of Algorithm-Assisted Human Decision-Making: Application to Pretrial Public Safety Assessment

Presented by Kosuke Imai, Professor of Government and of Statistics, Harvard University, Institute for Quantitative Social Science

Friday, September 9
11:00 a.m. ET
School of Business, Room 106

Despite an increasing reliance on fully-automated algorithmic decision-making in our day-to-day lives, human beings still make highly consequential decisions. As frequently seen in business, healthcare, and public policy, recommendations produced by algorithms are provided to human decision-makers to guide their decisions. While there exists a fast-growing literature evaluating the bias and fairness of such algorithmic recommendations, an overlooked question is whether they help humans make better decisions. We develop a general statistical methodology for experimentally evaluating the causal impacts of algorithmic recommendations on human decisions. We also show how to examine whether algorithmic recommendations improve the fairness of human decisions and derive the optimal decision rules under various settings. We apply the proposed methodology to preliminary data from the first-ever randomized controlled trial that evaluates the pretrial Public Safety Assessment (PSA) in the criminal justice system. A goal of the PSA is to help judges decide which arrested individuals should be released. On the basis of the preliminary data available, we find that providing the PSA to the judge has little overall impact on the judge’s decisions and subsequent arrestee behavior. Our analysis, however, yields some potentially suggestive evidence that the PSA may help avoid unnecessarily harsh decisions for female arrestees regardless of their risk levels while it encourages the judge to make stricter decisions for male arrestees who are deemed to be risky. In terms of fairness, the PSA appears to increase an existing gender difference while having little effect on any racial differences in judges’ decisions. Finally, we find that the PSA’s recommendations might be unnecessarily severe unless the cost of a new crime is sufficiently high.

Speaker Bio:

Kosuke Imai is a professor in the Department of Government and the Department of Statistics at Harvard University. He is also an affiliate of the Institute for Quantitative Social Science. Before moving to Harvard in 2018, Imai taught at Princeton University for 15 years where he was the founding director of the Program in Statistics and Machine Learning. In addition, Imai served as the President of the Society for Political Methodology from 2017 to 2019 and was elected fellow in 2017. He has been Professor of Visiting Status in the Faculty of Law and Graduate Schools of Law and Politics at the University of Tokyo.
After obtaining a B.A. in Liberal Arts from the University of Tokyo (1998), Imai received an A.M. in Statistics (2002) and a Ph.D. in political science (2003) from Harvard University. He specializes in the development of statistical methods and machine learning algorithms and their applications to social science research. His areas of expertise include causal inference, computational social science, program evaluation, and survey methodology. His substantive applications range from the randomized evaluation of Mexican and Indian national health insurance programs to the assessment of pretrial public safety assessment in the United States criminal justice system.

Imai is the author of a widely used undergraduate introductory statistics textbook for social scientists, Quantitative Social Science: An Introduction (2017, 2021) and Quantitative Social Science: An Introduction in tidyverse (2022). He has published more than seventy peer-refereed journal articles in political science, statistics, and other fields, and authored over twenty opensource software packages. Imai has been recognized as a highly cited researcher in the cross-field category by Clarivate Analytics since 2018. He has won several awards including the Miyake Award (2006), the Warren Miller Prize (2008), the Pi Sigma Alpha Award (2013), the Stanley Kelley, Jr. Teaching Award (2013), the Statistical Software Award (2015, 2021), the Excellence in Mentoring Award (2021), and was the inaugural recipient of Society of Political Methodology’s Emerging Scholar Award (2011). Imai’s research has been supported by National Science Foundation grants as well as grants from other government agencies and private organizations.