Refactoring data delivery: The case study of the new tools for census flow data at UKDS

This paper presents the API and user interface for disseminating census flow data developed as part of the UK Data Service (UKDS) strategy to modernise its supported software. It explores the development challenges and the diverse technologies employed, contextualised within the broader scope of data services. The new tools were in the final stages of development at the time of writing.

We opted to construct these tools from the ground up, a decision influenced by the complex nature of the underlying data, which necessitated high levels of flexibility and adaptability. This paper will critically evaluate the decision-making process, weighing the advantages and disadvantages of developing in-house solutions versus the trend of relying on external, often commercial, platforms. Third-party solutions frequently compromise functionality and the ability to tailor to specific requirements, especially when dealing with highly complex data. The presentation will showcase some functionalities of these new tools, highlighting ongoing enhancements, including integrating AI and machine-learning technologies. We will also discuss the advantages of the design principle to separate the user interface from the backend API. This approach improves user experience and promotes better interoperability.

While acknowledging that budget and overall resource constraints are a common hurdle in such initiatives, this case study provides insights into the feasible options available to data services striving to deliver robust and comprehensive data to their users. The insights and experience shared are intended to contribute to the dynamic relationship between data services and information science, especially when dealing with data related to the social sciences. By offering practical examples and lessons learned, we aim to inform those enhancing data accessibility, utility, and distribution in a rapidly evolving digital environment, where data services often face challenges in keeping pace with technological advances.