Education Impact Evaluations: Finding Ways to Enhance Reliability

Rigorous education impact evaluations help determine the effect of policies, programs, and interventions on student outcomes. In many cases, the ideal study would be a randomized controlled trial conducted in a nationally representative sample. But randomized trials are rarely conducted in nationally representative samples.

“Fully representative samples are not realistic since schools aren’t usually required to participate,” notes Rob Olsen, Ph.D., a Westat Associate Director for Education Studies. “Also, the schools and students who do participate may be atypical, so the study findings are not guaranteed to reliably measure the program’s effect nationwide.”

Producing reliable evidence is even more challenging for education studies designed to inform policy decisions on the local level because educational programs often work better in some places than others. “We can’t predict where the program will work and where it won’t,” Dr. Olsen adds.

Producing More Reliable Findings

Westat is studying ways to generate generalizable or externally valid results to ensure more population-specific findings. Dr. Olsen works with a team of experts at Westat—including Stephen Bell, Ph.D., a Westat Vice President—and Johns Hopkins University to provide recommendations for generalizability to be used in future evaluations of education programs and other interventions.

Producing generalizable results first requires defining the evaluation’s target population—the individuals who would be affected by the key policy decisions. Dr. Olsen says the next steps are to

  1. Identify a sampling frame of potentially eligible schools.
  2. Select a representative sample. Random sampling is recommended, but systematic sampling approaches may be sufficient.
  3. Choose replacement schools when schools selected for the sample decline to participate.
  4. Apply statistical methods that account for differences between the students and schools that participate in the study and the target population.

Focusing on Generalizability

Westat is conducting a simulation study to test promising methods for selecting schools for rigorous education evaluations in the real world where schools are not required to participate. The study will assess how much more reliable evaluation findings would be if researchers attempted to select a representative sample instead of a convenience sample. Westat will disseminate evidence-based site selection methods to help future studies produce more generalizable findings.

In another study, Westat will explore how well evidence from multisite impact studies can inform local policy decisions. This research will examine the potential for local decisionmakers to improve youth outcomes by adopting programs that are effective in other sites.

“These studies will test possible solutions to these challenges to help the field produce more reliable evidence for education policymakers,” says Dr. Olsen. “We’re excited to do this important work.”

 

Without rigorous studies, education policymakers would lack the evidence they need to make good decisions. We just need to be honest about the limitations of these studies and work hard to address them.

- Rob Olsen, Ph.D., Westat Associate Director, Education Studies

Want to work with us?
You’ll be in great company.

About Us Careers

Westat Employees.
Westat Employee.
AAA Foundation for Traffic Safety
Baltimore Metropolitan Council
Centers for Disease Control and Prevention
Centers for Medicare & Medicaid Services
Chicago Metropolitan Agency for Planning
DC Public Schools
Georgia Department of Transportation
Internal Revenue Service
Leadership Montgomery
Maryland Cancer Registry
Michigan Department of Health and Human Services
National Science Foundation
NYC Mayor’s Office for Economic Opportunity
Organization for Economic Cooperation and Development
Robert Wood Johnson Foundation
SiriusXM
Social Security Administration
Substance Abuse and Mental Health Services Administration
Teach for America
Texas Education Agency
The Johns Hopkins University
The National Institutes of Health
The Verizon Foundation
Toyota
U.S. Department of Agriculture
U.S. Department of Education
U.S. Department of Health and Human Services
U.S. Department of Justice
U.S. Department of Transportation
U.S. Department of Veterans Affairs
University of Maryland Baltimore Campus
University of Michigan
University System of Maryland
Explore Our Clients

Please wait...

Forbes 2021 The Best Employers for Women Powered by StatistaForbes 2021 The Best Employers for Diversity Powered by StatistaForbes 2021: America's Best Midsize Employer Powered by Statista Forbes 2021 The Best Employers for Veterans Powered by Statista

Westat is an Equal Opportunity Employer and does not discriminate on the basis of race, creed, color, religion, sex, national origin, age, veteran status, disability, marital status, sexual orientation, citizenship status, genetic information, gender identity or expression, or any other protected status under applicable law. Notices to Employees & Applicants.