A New Look At Metadata

When first coined around 1970, the term metadata was defined as data about data. This sufficed until the Dublin Core Metadata Initiative expanded the notion to include the description of any resource, around 1995. Metadata encompassed both text documents and more structured approaches. DDI Codebook was designed with both perspectives in mind. New definitions focused on the descriptive nature of metadata, preserving the feature that anything can be described.

As the desirability of data integration and the application of the FAIR principles to data and metadata management have been demonstrated, designing and populating highly structured metadata is becoming more necessary. Now new standards, such as DDI-CDI, have a more structured design in mind.

These efforts are tied closely to achieving more automation around the many uses of data. This has increased the disparity between metadata that is highly structured and that which is just in the form of textual documents. Machines can interpret and operate with highly structured metadata directly. Machine-driven processes to process, integrate, and analyze data are enabled and enhanced through the use of highly structured metadata. Text documentation on the other hand requires humans to interpret the content or the application of ML engines to translate the text into a more structured format. LLMs mostly produce human-readable output.

This paper will address the issue of the differences between structured metadata and text-based documentation. We carefully define the differences and describe needs in treating the two approaches separately. A similarity between interpreting text documentation and qualitative data analysis is explored. The application of ML techniques is similarly explored. The promise of LLMs driven by both a highly structured core and a corpus of text documentation is looked at. The limitations of highly structured metadata are discussed.