Data Cleaning With Llms: An Evaluation of Consistency

Data wrangling can be a time consuming and tedious task. Recently, researchers have explored LLMs to enhance the automation and quality in data preparation and data cleaning for analysis (Akella et al 2025, Liu et al 2024, Zhang et al 2024). These studies generally focus on adjusting prompts and evaluating accuracy for the best given prompt in relationship to a given model. However, in the context of data science—despite its effect on reproducibility—the consistency of the model output has been less studied.

Building on existing literature about LLM consistency (Kuhn et al, 2023, Lin et al 2023, Qiu & Miikkulainen 2024, Wu et al 2025) we conduct a small experiment using the case study of data cleaning. We investigate the variation (rather than just the accuracy) in responses generated by an LLM. Using a carefully curated, messy dataset and a single prompt we iteratively ask the model to generate a ‘clean’ csv file and evaluate the outputs for similarity and consistency. We discuss challenges in evaluation, relevance of the variations, and implications for using LLMs for data cleaning.

References: Akella et al 2025 [https://aclanthology.org/2025.naacl-industry.70/] Kuhn et al, 2023 [https://arxiv.org/abs/2302.09664] Lin et al 2023 [https://arxiv.org/abs/2305.19187] Liu et al 2024 [https://doi.org/10.1145/3691620.3695267] Qiu & Miikkulainen 2024 [https://arxiv.org/abs/2405.13845] Wu et al 2025 [https://arxiv.org/pdf/2505.23799] Zhang et al 2024 [https://ieeexplore.ieee.org/abstract/document/10555056