Test Validation for a Crisis: Five Practical Heuristics for the Best and Worst of Times

Presented by Dr. Andrew Ho, Harvard University

Friday, January 28
3 p.m. ET

The COVID-19 pandemic has raised debate about the place of education and testing in a hierarchy of needs. What do tests tell us that other measures do not? Is testing worth the time? Do tests expose or exacerbate inequality? The academic consensus in the open-access AERA/APA/NCME Standards has not seemed to help proponents and critics of tests reach common ground. I propose five heuristics for test validation and demonstrate their usefulness for navigating test policy and test use in a time of crisis: 1) A “four quadrants” framework for purposes of educational tests. 2) The “Five Cs,” a mnemonic for the five types of validity evidence in the Standards. 3) “RTQ,” a mantra reminding test users to read items. 4) The “3 Ws,” a user-first perspective on testing. And 5) the “Two A’s Tradeoff” between Assets and Accountability. I illustrate application of these heuristics to the challenge of reporting aggregate-level test scores when populations and testing conditions change as they have over the pandemic (e.g., An, Ho, & Davis, in press; Ho, 2021). I define and discuss these heuristics in the hope that they increase consensus and improve test use in the best and worst of times.

Speaker Bio:

Dr. Andrew Ho is the Charles William Eliot Professor of Education at the Harvard Graduate School of Education. He is a psychometrician whose research aims to improve the design, use, and interpretation of test scores in educational policy and practice. Dr. Ho is known for his research documenting the misuse of proficiency-based statistics in state and federal policy analysis. He has also clarified properties of student growth models for both technical and general audiences. His scholarship advocates for designing evaluative metrics to achieve multiple criteria: metrics must be accurate, but also transparent to target audiences and resistant to inflation under high stakes. Dr. Ho is a director of the Carnegie Foundation for the Advancement of Teaching and has served on the governing boards for NCME and NAEP. He has chaired the research committee for the Vice Provost for Advances in Learning at Harvard, which governed research on “massive open online courses”. He holds a Ph.D. in Educational Psychology and M.S. in Statistics from Stanford University. Before graduate school, he taught middle school creative writing in his hometown of Honolulu, Hawaii, and high school Physics and AP Physics in Ojai, California.