Education Impact Evaluations: Finding Ways to Enhance Reliability
Rigorous education impact evaluations help determine the effect of policies, programs, and interventions on student outcomes. In many cases, the ideal study would be a randomized controlled trial conducted in a nationally representative sample. But randomized trials are rarely conducted in nationally representative samples.
“Fully representative samples are not realistic since schools aren’t usually required to participate,” notes Rob Olsen, Ph.D., a Westat Associate Director for Education Studies. “Also, the schools and students who do participate may be atypical, so the study findings are not guaranteed to reliably measure the program’s effect nationwide.”
Producing reliable evidence is even more challenging for education studies designed to inform policy decisions on the local level because educational programs often work better in some places than others. “We can’t predict where the program will work and where it won’t,” Dr. Olsen adds.
Producing More Reliable Findings
Westat is studying ways to generate generalizable or externally valid results to ensure more population-specific findings. Dr. Olsen works with a team of experts at Westat—including Stephen Bell, Ph.D., a Westat Vice President—and Johns Hopkins University to provide recommendations for generalizability to be used in future evaluations of education programs and other interventions.
Producing generalizable results first requires defining the evaluation’s target population—the individuals who would be affected by the key policy decisions. Dr. Olsen says the next steps are to
- Identify a sampling frame of potentially eligible schools.
- Select a representative sample. Random sampling is recommended, but systematic sampling approaches may be sufficient.
- Choose replacement schools when schools selected for the sample decline to participate.
- Apply statistical methods that account for differences between the students and schools that participate in the study and the target population.
Focusing on Generalizability
Westat is conducting a simulation study to test promising methods for selecting schools for rigorous education evaluations in the real world where schools are not required to participate. The study will assess how much more reliable evaluation findings would be if researchers attempted to select a representative sample instead of a convenience sample. Westat will disseminate evidence-based site selection methods to help future studies produce more generalizable findings.
In another study, Westat will explore how well evidence from multisite impact studies can inform local policy decisions. This research will examine the potential for local decisionmakers to improve youth outcomes by adopting programs that are effective in other sites.
“These studies will test possible solutions to these challenges to help the field produce more reliable evidence for education policymakers,” says Dr. Olsen. “We’re excited to do this important work.”
Without rigorous studies, education policymakers would lack the evidence they need to make good decisions. We just need to be honest about the limitations of these studies and work hard to address them.
- Rob Olsen, Ph.D., Westat Associate Director, Education Studies