Can Machine Learning Automate the Creation of Data Catalogs?

Many research and evaluation questions can be answered using extant data from surveys, program evaluations, and administrative data. Data catalogs help support the use of extant data by providing researchers with an overview of different datasets and information about how to access them. Developing and maintaining data catalogs requires considerable time and resources to ensure they are thorough and accurate.

Our team from Westat and the American Institutes for Research (AIR), on behalf of the U.S. Department of Labor (DOL), explored the potential of automated machine learning (ML) algorithms to create and maintain data catalogs for employment and training outcomes. We conducted a literature review, piloted a manual data catalog assembly process, and consulted with a technical working group of computer science experts.

While current ML innovations present challenges for automating data catalog development, there are promising solutions. Data sources vary in their structures and the amount of metadata available, making it difficult to develop an automation program. Moreover, employing automations for even a portion of the data catalog development process would require large investments in staff and computing resources. However, artificial intelligence (AI) is a rapidly evolving field, and literature suggests there may be many opportunities to support automated data collection in the future, including the potential use of generative AI. Federal agencies might explore and use existing tools to meet their needs.

Our brief, Explorations in Data Innovations: Can Machine Learning Support Data Catalog Development? (PDF) details the data catalog development process, options for automation, and recommendations for future explorations to eventually harness artificial intelligence. When using machine learning to produce public-facing products, Federal agencies may need to use a mix of staff with different skill sets, including data scientists, website developers, experts in cloud computing, and subject matter experts.

Allison Hyra, PhD, Westat Associate Vice President of Social Policy and Economics Research and the project director for this work, summed up the study: “This was a great collaboration with the staff of DOL’s Chief Evaluation Office (CEO) not only because we explored the extent to which new technology can support researchers, but because CEO wanted to share our lessons learned despite finding automated data catalogs are currently infeasible. By engaging the field and identifying next steps, Westat, AIR, and CEO are contributing to realizing a future where we can further democratize access to datasets that answer pressing employment and training information needs.”