Education Impact Evaluations: Finding Ways to Enhance Reliability

Rigorous education impact evaluations help determine the effect of policies, programs, and interventions on student outcomes. In many cases, the ideal study would be a randomized controlled trial conducted in a nationally representative sample. But randomized trials are rarely conducted in nationally representative samples.

“Fully representative samples are not realistic since schools aren’t usually required to participate,” notes Rob Olsen, Ph.D., a Westat Associate Director for Education Studies. “Also, the schools and students who do participate may be atypical, so the study findings are not guaranteed to reliably measure the program’s effect nationwide.”

Producing reliable evidence is even more challenging for education studies designed to inform policy decisions on the local level because educational programs often work better in some places than others. “We can’t predict where the program will work and where it won’t,” Dr. Olsen adds.

Producing More Reliable Findings

Westat is studying ways to generate generalizable or externally valid results to ensure more population-specific findings. Dr. Olsen works with a team of experts at Westat—including Stephen Bell, Ph.D., a Westat Vice President—and Johns Hopkins University to provide recommendations for generalizability to be used in future evaluations of education programs and other interventions.

Producing generalizable results first requires defining the evaluation’s target population—the individuals who would be affected by the key policy decisions. Dr. Olsen says the next steps are to

  1. Identify a sampling frame of potentially eligible schools.
  2. Select a representative sample. Random sampling is recommended, but systematic sampling approaches may be sufficient.
  3. Choose replacement schools when schools selected for the sample decline to participate.
  4. Apply statistical methods that account for differences between the students and schools that participate in the study and the target population.

Focusing on Generalizability

Westat is conducting a simulation study to test promising methods for selecting schools for rigorous education evaluations in the real world where schools are not required to participate. The study will assess how much more reliable evaluation findings would be if researchers attempted to select a representative sample instead of a convenience sample. Westat will disseminate evidence-based site selection methods to help future studies produce more generalizable findings.

In another study, Westat will explore how well evidence from multisite impact studies can inform local policy decisions. This research will examine the potential for local decisionmakers to improve youth outcomes by adopting programs that are effective in other sites.

“These studies will test possible solutions to these challenges to help the field produce more reliable evidence for education policymakers,” says Dr. Olsen. “We’re excited to do this important work.”

 

Without rigorous studies, education policymakers would lack the evidence they need to make good decisions. We just need to be honest about the limitations of these studies and work hard to address them.

- Rob Olsen, Ph.D., Westat Associate Director, Education Studies

Want to work with us?
You’ll be in great company.

About Us Careers

Westat Employees.
Westat Employee.
Centers for Disease Control and Prevention
Centers for Medicare & Medicaid Services
Substance Abuse and Mental Health Services Administration
The National Institutes of Health
AAA Foundation for Traffic Safety
The Johns Hopkins University
University of Maryland Baltimore Campus
University of Denver
U.S. Department of Veterans Affairs
U.S. Department of Transportation
U.S. Department of Justice
U.S. Department of Health and Human Services
U.S. Department of Education
U.S. Department of Agriculture
Toyota
The Verizon Foundation
Texas Education Agency
Baltimore Metropolitan Council
Teach for America
Social Security Administration
SiriusXM
Robert Wood Johnson Foundation
Organization for Economic Cooperation and Development
NYC Mayor’s Office for Economic Opportunity
National Science Foundation
Michigan Department of Health and Human Services
Maryland Cancer Registry
Internal Revenue Service
Georgia Department of Transportation
DC Public Schools
ClearWay Minnesota
Chicago Metropolitan Agency for Planning
University of Michigan
Explore Our Clients

Please wait...

Forbes 2020 The Best Employers for Diversity Powered by Statista Copyright 2019 Forbes Media LLC. Used with permissionForbes 2019: The Best Employers for Women Powered by Statista Copyright 2019 Forbes Media LLC. Used with permission

Westat is an Equal Opportunity Employer and does not discriminate on the basis of race, creed, color, religion, sex, national origin, age, veteran status, disability, marital status, sexual orientation, citizenship status, genetic information, gender identity, or any other protected status under applicable law. Notices to Employees & Applicants.