Harnessing Impact Evaluations for All Levels of Policymaking
As America continues to grapple with the COVID-19 pandemic, public leaders will need to decide what initiatives to incorporate in efforts to rebuild the economy. Successful initiatives will deliver broad benefits; impact evaluations conducted by economists and social scientists can guide local, state, and national decisionmakers to those choices.
First and foremost, using evaluations to get the maximum impact from national initiatives will be essential. This requires measuring how much a given initiative contributes to the country’s rising economic tide on a nationally representative basis (rather than in a smattering of places most convenient for investigation) since federal funds will be spent and results will be needed for the entire nation.
This principle—assess the effectiveness of government interventions in a representative set of sites to get the most reliable national policy guidance—applies in many contexts. In a recent presentation to officials at the U.S. Department of Labor, 2 Westat researchers studying how to make evaluations stronger and more meaningful reinforced this principle.
“If a rigorous impact evaluation lacks a well-defined target population of individuals whom the policy seeks to assist and neglects to have a representative sample from that population, findings will be misleading to policymakers,” explains Westat’s Stephen Bell, Ph.D., and Rob Olsen, Ph.D.
Improving National Policy Guidance
Focusing on workforce policy studies, Dr. Bell, a Westat Vice President and Senior Statistical Fellow, and Dr. Olsen, an Associate Director of Education Studies, discussed ways to increase “external validity” to make findings more generalizable and, thus, improve the reliability of findings informing national policy. “This will require evaluators to select a nationally representative sample of sites and to incentivize site participation,” says Dr. Olsen.
Over the past 10 years, Drs. Bell and Olsen, teaming with Drs. Elizabeth Stuart and Larry Orr of Johns Hopkins University, have examined multisite impact evaluations to interpret how well their findings represent the nation or individual states and localities seeking policy guidance. Their pioneering research—including recommendations and innovative tools for making findings more representative of the target population—has been published in such outlets as the Journal of Policy Analysis and Management and Educational Evaluation and Policy Analysis.
Steps Toward More Generalizable Findings
To support national generalizability, Drs. Bell and Olsen recommend 4 steps that impact evaluations can take to better inform national policy decisions:
- Identify the population which the studied intervention hopes to affect.
- Select a representative sample of sites where that population lives.
- Use creative study design features and financial incentives to encourage site participation.
- Choose replacement sites wisely when some sites opt out.
These steps play a crucial role for evaluations of labor policy and in other policy areas such as education. See, for example, Education Impact Evaluations: Finding Ways to Enhance Reliability.
Extrapolating Impact Evidence Locally
States and localities also need sound evidence about program effectiveness. “Few social program evaluations in the United States are specific to a single state or metropolitan area,” says Dr. Bell. “Even in large-scale national evaluations, sample sizes for individual political subunits are rarely large enough for separate analysis, so the push for ‘evidence-based’ policymaking doesn’t work well for, say, the State of Ohio or the City of San Antonio.”
“To draw localized measures of impact from multisite evaluations—for workforce programs and for social policy generally—statistical adjustments need to be made,” says Dr. Olsen. “We are striving to learn how well such adjustments can work.”
What Does the Future Hold?
“The team at Westat and Johns Hopkins intends to be the catalyst for ensuring that external validity becomes as high a priority as internal validity in social policy impact evaluations,” says Dr. Olsen. “We believe that already existing methods for improving external validity should and can be comprehensively adopted over the next 5 to 10 years.” “But that’s not enough,” adds Dr. Bell. “Our impact evaluation team pushes innovative approaches designed from the get-go to deliver reliable policy guidance to all levels of government.”
Our impact evaluation team pushes innovative approaches designed from the get-go to deliver reliable policy guidance to all levels of government.
- Stephen Bell, Ph.D., a Westat Vice President and Senior Statistical Fellow