Harnessing Impact Evaluations for All Levels of Policymaking

As America continues to grapple with the COVID-19 pandemic, public leaders will need to decide what initiatives to incorporate in efforts to rebuild the economy. Successful initiatives will deliver broad benefits; impact evaluations conducted by economists and social scientists can guide local, state, and national decisionmakers to those choices.

First and foremost, using evaluations to get the maximum impact from national initiatives will be essential. This requires measuring how much a given initiative contributes to the country’s rising economic tide on a nationally representative basis (rather than in a smattering of places most convenient for investigation) since federal funds will be spent and results will be needed for the entire nation.

This principle—assess the effectiveness of government interventions in a representative set of sites to get the most reliable national policy guidance—applies in many contexts. In a recent presentation to officials at the U.S. Department of Labor, 2 Westat researchers studying how to make evaluations stronger and more meaningful reinforced this principle.

“If a rigorous impact evaluation lacks a well-defined target population of individuals whom the policy seeks to assist and neglects to have a representative sample from that population, findings will be misleading to policymakers,” explains Westat’s Stephen Bell, Ph.D., and Rob Olsen, Ph.D.

Improving National Policy Guidance

Focusing on workforce policy studies, Dr. Bell, a Westat Vice President and Senior Statistical Fellow, and Dr. Olsen, an Associate Director of Education Studies, discussed ways to increase “external validity” to make findings more generalizable and, thus, improve the reliability of findings informing national policy. “This will require evaluators to select a nationally representative sample of sites and to incentivize site participation,” says Dr. Olsen.

Over the past 10 years, Drs. Bell and Olsen, teaming with Drs. Elizabeth Stuart and Larry Orr of Johns Hopkins University, have examined multisite impact evaluations to interpret how well their findings represent the nation or individual states and localities seeking policy guidance. Their pioneering research—including recommendations and innovative tools for making findings more representative of the target population—has been published in such outlets as the Journal of Policy Analysis and Management and Educational Evaluation and Policy Analysis.

Steps Toward More Generalizable Findings

To support national generalizability, Drs. Bell and Olsen recommend 4 steps that impact evaluations can take to better inform national policy decisions:

  1. Identify the population which the studied intervention hopes to affect.
  2. Select a representative sample of sites where that population lives.
  3. Use creative study design features and financial incentives to encourage site participation.
  4. Choose replacement sites wisely when some sites opt out.

These steps play a crucial role for evaluations of labor policy and in other policy areas such as education. See, for example, Education Impact Evaluations: Finding Ways to Enhance Reliability.

Extrapolating Impact Evidence Locally

States and localities also need sound evidence about program effectiveness. “Few social program evaluations in the United States are specific to a single state or metropolitan area,” says Dr. Bell. “Even in large-scale national evaluations, sample sizes for individual political subunits are rarely large enough for separate analysis, so the push for ‘evidence-based’ policymaking doesn’t work well for, say, the State of Ohio or the City of San Antonio.”

“To draw localized measures of impact from multisite evaluations—for workforce programs and for social policy generally—statistical adjustments need to be made,” says Dr. Olsen. “We are striving to learn how well such adjustments can work.”

What Does the Future Hold?

“The team at Westat and Johns Hopkins intends to be the catalyst for ensuring that external validity becomes as high a priority as internal validity in social policy impact evaluations,” says Dr. Olsen. “We believe that already existing methods for improving external validity should and can be comprehensively adopted over the next 5 to 10 years.” “But that’s not enough,” adds Dr. Bell. “Our impact evaluation team pushes innovative approaches designed from the get-go to deliver reliable policy guidance to all levels of government.”

Our impact evaluation team pushes innovative approaches designed from the get-go to deliver reliable policy guidance to all levels of government.

- Stephen Bell, Ph.D., a Westat Vice President and Senior Statistical Fellow

Want to work with us?
You’ll be in great company.

About Us Careers

Westat Employees.
Westat Employee.
Centers for Disease Control and Prevention
Centers for Medicare & Medicaid Services
Substance Abuse and Mental Health Services Administration
The National Institutes of Health
AAA Foundation for Traffic Safety
The Johns Hopkins University
University of Maryland Baltimore Campus
University of Denver
U.S. Department of Veterans Affairs
U.S. Department of Transportation
U.S. Department of Justice
U.S. Department of Health and Human Services
U.S. Department of Education
U.S. Department of Agriculture
Toyota
The Verizon Foundation
Texas Education Agency
Baltimore Metropolitan Council
Teach for America
Social Security Administration
SiriusXM
Robert Wood Johnson Foundation
Organization for Economic Cooperation and Development
NYC Mayor’s Office for Economic Opportunity
National Science Foundation
Michigan Department of Health and Human Services
Maryland Cancer Registry
Internal Revenue Service
Georgia Department of Transportation
DC Public Schools
ClearWay Minnesota
Chicago Metropolitan Agency for Planning
University of Michigan
Explore Our Clients

Please wait...

Forbes 2020 The Best Employers for Diversity Powered by Statista Copyright 2019 Forbes Media LLC. Used with permissionForbes 2019: The Best Employers for Women Powered by Statista Copyright 2019 Forbes Media LLC. Used with permission

Westat is an Equal Opportunity Employer and does not discriminate on the basis of race, creed, color, religion, sex, national origin, age, veteran status, disability, marital status, sexual orientation, citizenship status, genetic information, gender identity, or any other protected status under applicable law. Notices to Employees & Applicants.