Conducting National Surveys in the Age of COVID-19

Conducting data collection for national surveys during a crisis is not unusual. Hurricanes, earthquakes, wildfires, civil unrest, and other emergency situations have posed challenges to data collection in geographically localized areas at different times. But how do researchers tackle the challenges of conducting a nationally representative survey during a health pandemic? “The situation with this COVID-19 pandemic is especially unpredictable,” notes Leyla Mohadjer, Ph.D., Westat Vice President and Associate Director for Statistics and Evaluation Sciences. “So, we continually plan for different scenarios and quick turnaround strategies that rely on the fundamentals of statistical research and survey methodology.”

Proactive planning is key to conducting survey research especially at this time. Westat is in ongoing communication with our clients on strategies to move forward on in-person data collection. “We’re continually taking the temperature of projects as we move forward and continually planning for different contingencies depending on the expected timing of returning to the field,” explains Dr. Mohadjer.

Mode Changes in Surveys

Many surveys that were conducted in person have transitioned to telephone or web surveys. “We’re studying any mode changes to see if there is a mode change effect. If so, we have developed plans for adjusting the final samples to minimize the impact of mode change. Also, for some surveys, combining data from the pre- and post-COVID period may not be justifiable,” cautions Dr. Mohadjer. “This is true for surveys where the pre-COVID and post-COVID respondents are coming from different parts of the country. In some cases, the statistics of interest, like employment status, changed drastically from pre- to post-COVID. We have developed special sampling procedures that create a bridge between the pre- and post-COVID samples to best deal with these kinds of situations.”

Response Rates Are Key

Getting high response rates is critical no matter how the survey is conducted, but the U.S. Census Bureau and the Bureau of Labor Statistics have already raised concerns about lower response rates for their large surveys. “We need to prepare for a similar impact on response rates in surveys we conduct,” notes Dr. Mohadjer. “In some cases, it is a matter of working with survey methods specialists to devise ways to improve response rates. We are also mitigating this risk by doing extensive analyses to assess bias due to nonresponse, and to adjust the resulting sample to reduce bias in outcome statistics by applying various sample weighting adjustments, like benchmarking.”

Lower response rates not only impact the representativeness of the sample but also result in a smaller sample size for the survey. “One way we to mitigate this,” Dr. Mohadjer explains, “is to split the selected sample into a main sample and many smaller random samples, with the main sample being released to the field first. Throughout the course of data collection, Westat monitors data collection activity in real time and is prepared to release new samples if need be. We can implement this seamless process because of our integrated network of survey statisticians, survey operations staff, field data collection oversight, and analysis staff. If the original sample released to data collectors is not providing enough responses,” she continues, “we’ll release another subsample of different households, maintaining random representativeness of the survey at the same time as aiming to achieve the desired sample size.”

Dealing with Unpredictable Shifts

Part of the planning for surveying during a pandemic is to also manage potential pauses in data collection either in local hot spots or across the country. “We have to be cognizant and flexible as different parts of the country are affected at different times, and how we can reduce bias in samples that we’re collecting,” notes Dr. Mohadjer. Westat has established procedures to minimize disruption and risk by developing data collection plans with quick turnarounds, restarting data collection quickly after a pause, or changing the mode of data collection if needed.

Focus on Quality Data Collection

“At Westat, we put priority on the quality of the data collected during the pandemic, and also prepare for correcting for any potential bias through sample adjustment and weighting,” continues Dr. Mohadjer. “We share the methods that are working effectively across different data collection projects, and this benefits all Westat projects and clients.”

Statisticians also harness other tools, like data linkage, where data sources are linked to augment or replace missing data as a result of the pandemic, and data science techniques such as machine learning to quickly gather large amount of data and conduct various analyses either using paradata or external data sources for various needs.

The rapidly evolving nature of the COVID-19 pandemic requires researchers to quickly adapt and be nimble in meeting the challenges of sampling and data collection. While the challenges may appear daunting, innovative uses of statistical tools and techniques, including adaptive survey methods, are instrumental in achieving the quality and integrity of the data.