Strategies and Tools For Curating Large Datasets

Researchers have diverse data publishing needs, depending on the size, scope, and nature of their data. The Federated Research Data Repository (FRDR) in Canada is a generalist repository designed to curate and publish large open datasets. But this poses a challenge: How do we curate large (sometimes very large) datasets from a variety of disciplines, ensuring they are well-documented and can be broadly re-used? How do we review many thousands of files efficiently? The Curation Services team will walk you through the development workflow, use cases, and experiences and challenges curating large datasets in a high-performance computing environment. We will showcase a suite of open source tools we have developed to systematically record metadata from data files commonly deposited in FRDR, and will finally discuss further challenges and directions.