All Projects

How can coding of RECS ESS data be automated and accelerated?

Using natural language processing to match open-ended energy suppliers in the Residential Energy Consumption Survey, Energy Supplier Survey (RECS ESS)

The Energy Information Administration (EIA) administers the Residential Energy Consumption Survey (RECS), a nationwide survey of energy-related characteristics, consumption, and expenditures for U.S. homes. Westat conducts the followup component for EIA, the RECS Energy Supplier Survey (ESS). Data from RECS ESS allow for broader comparisons across sectors, as well as projections of future consumption trends.

In this current RECS ESS cycle, there are 20,000+ open-ended entries about energy suppliers in natural gas, electricity, fuel oil, and propane reported by the households. Traditionally, human coders had to manually clean up typos and verify the existence and accuracy of the respondents' input by looking up a reference list of all the energy suppliers in the United States and relying on internet resources. It was a labor-intensive and time-consuming effort.

To speed the process as well as enhance efficiencies, Westat used the power of artificial intelligence (AI) capabilities.

To automate this process, Westat used natural language processing, more specifically, string-matching technique—finding strings of data that closely match a pattern, to calculate the editing distance between the respondents' inputs and all the lookup texts in the reference list.

We then developed an algorithm to identify matched pairs of 1 input and 1 lookup supplier based on the editing distance and some data patterns we discovered during the proof-of-concept stage.

If a pair cannot be detected, for each input, the algorithm will suggest the top 10 reference suppliers ranked by distance for human review.

The end product is a Python program. Using natural language processing, data entry was significantly accelerated. The first batch of 2021 respondent inputs of 4,000+ entries was processed in less than 30 minutes. It detected 48% matched pairs with 100% precision, and passed 56% of data to human review.

Want to work with us?
You’ll be in great company.

About Us Careers

Westat Employees.
Westat Employee.
AAA Foundation for Traffic Safety
Baltimore Metropolitan Council
Centers for Disease Control and Prevention
Centers for Medicare & Medicaid Services
Chicago Metropolitan Agency for Planning
DC Public Schools
Georgia Department of Transportation
Internal Revenue Service
Leadership Montgomery
Maryland Cancer Registry
Michigan Department of Health and Human Services
National Science Foundation
NYC Mayor’s Office for Economic Opportunity
Organization for Economic Cooperation and Development
Robert Wood Johnson Foundation
Social Security Administration
Substance Abuse and Mental Health Services Administration
Teach for America
Texas Education Agency
The Johns Hopkins University
The National Institutes of Health
The Verizon Foundation
U.S. Department of Agriculture
U.S. Department of Education
U.S. Department of Health and Human Services
U.S. Department of Justice
U.S. Department of Transportation
U.S. Department of Veterans Affairs
University of Maryland Baltimore Campus
University of Michigan
University System of Maryland
Explore Our Clients

Please wait...

Forbes 2021 The Best Employers for Women Powered by StatistaForbes 2021 The Best Employers for Diversity Powered by StatistaForbes 2021: America's Best Midsize Employer Powered by Statista Forbes 2021 The Best Employers for Veterans Powered by Statista

Westat is an Equal Opportunity Employer and does not discriminate on the basis of race, creed, color, religion, sex, national origin, age, veteran status, disability, marital status, sexual orientation, citizenship status, genetic information, gender identity or expression, or any other protected status under applicable law. Notices to Employees & Applicants.