Figure 3: DatologyAI data curation also yields significant training efficiency wins across retrieval tasks. We visualize retrieval performance for ViT-B/32 models trained over varying numbers of image-text pairs.

Train Smaller: Improved Data Curation Unlocks Inference Compute Savings

Classification

Across a range of popular classification benchmarks, we find that DatologyAI models obtain an order of magnitude improvement in inference costs. Figure 1 visualizes how DatologyAI ViT-B/32 models outperform MetaCLIP and DFN ViT-B/16 models across ImageNet 1k and ImageNet v2, resulting in a greater than 60% reduction in inference costs. Moreover, DatologyAI ViT-B/16 models also outperform MetaCLIP ViT-L/14 models, implying a greater than 70% reduction in inference cost.

Retrieval

The inference time efficiency gains also translate to retrieval benchmarks. Figure 1 visualizes DatologyAI ViT-B/32 exceeding the retrieval performance of ViT-B/16 MetaCLIP and DFN models across Flickr and MSCOCO — corresponding to a 60% reduction in inference cost. We also see similar trends with our ViT-B/16 models, which match or exceed the performance of MetaCLIP and DFN trained ViT-L/14 models.

Key Scientific Improvements

The DatologyAI data curation pipelines consists of a suite of distinct algorithm families. Throughout this work, we have built upon the initial version of multimodal data curation, described in depth in our prior blog post. Following from the prior blog post, we continue to develop two separate data curation pipelines, one optimized for classification and a second tailored for retrieval.

Our model quality gains are primarily driven by a series of scientific and engineering advances which improve the overall quality and scalability of our data curation pipelines:

• Improved target distribution matching: Web-scale datasets are inherently vast and heterogeneous, with significant variation in both quality and relevance across subsets. Consequently, modern data curation workflows often prioritize the identification of high-quality, domain-relevant subsets within these large corpora (Gadre et al, 2023, Wang et al, 2024). In our work, we found that identifying high-quality, relevant subsets led to substantial performance gains in CLIP training. However, the multimodal nature of CLIP introduces a challenge: should quality and relevance be defined over the image features, text features, or both? Our experiments suggest that the optimal strategy is task-dependent. For example, retrieval tasks benefited more from aligning distributions along text representations, whereas classification tasks showed greater improvements when alignment was based on image features.

• Improved Synthetic Data: Leveraging synthetic data effectively at scale is not a plug-and-play solution. Small distributional mismatches can degrade performance, and mode collapse can severely limit model diversity and capability. While our original blog post we had seen large improvements from synthetic data for retrieval tasks, these benefits did not extend to classification tasks. In this work, we generalized our synthetic data pipeline to also benefit classification evals, identifying various properties of synthetically generated image captions which negatively impacted the performance on classification tasks. As a concrete example, synthetically generated image captions tended to be significantly longer than natural captions — this induces a large distribution mismatch with (typically short) templates used during classification evaluation (Radford et al., 2021). These insights helped us improve our synthetic data generation pipelines for classification tasks.

• Predictive Metrics for Curation Quality: A significant challenge when iterating on data curation pipelines is the need to train and evaluate models in order to ensure that data curation interventions are in fact yielding significant improvements. Indeed, practitioners may need to run many thousands of slow and expensive experiments to verify complex data curation pipelines and carefully select associated hyper-parameters. To tackle this challenge, we developed predictive metrics to approximate the effect of data curation interventions without needing to train and evaluate models. These cheap-to-compute metrics enabled rapid iteration and informed the selection of numerous hyper-parameters within our data curation algorithms. They also helped to identify beneficial curation steps that may not have been intuitively incorporated into our CLIP data pipelines. In total, we conducted over one thousand distinct experiments and ablations prior to finalizing the pipeline.

For further details about the DatologyAI multimodal data curation pipeline, please see our in-depth blog post here.

Experimental details

Our objective is to highlight the large-scale improvements which can be obtained exclusively by improving data curation. As such, we did not modify architecture, optimizer, objective functions, etc, and focused entirely on standard CLIP models and training procedures.

Our model evaluation follows the same setup as described in our prior blog post. We evaluate models across 27 tasks which are divided into 24 classification tasks, which require models to correctly identify a category label for an image, and 3 retrieval tasks, which require a model to identify the most relevant image or text for a given query. Throughout this blog post we have focused primarily on ImageNet 1k, ImageNet v2, MSCOCO and Flickr as these are evaluations which are frequently reported — we provide a full breakdown of our performance in the Appendix.

Our primary results focused on training CLIP ViT-B/32 models as described in Radford et al 2021. We relied on the open source OpenCLIP training repository (Cherti et al 2024, release 2.24.0) and used default architectures and hyper-parameters. To complement the results of our ViT-B/32 models, we also report performance across ViT-B/16 and ViT-L/14 models. Following our prior blog post, we continue to develop two separate data curation recipes which are tailored for classification and retrieval tasks, respectively. This choice is motivated by the various distinct applications of CLIP models which are frequently deployed independently.

Data

All experiments presented in this blog post leverage the DataComp multimodal benchmark dataset (Gadre et al., 2023). While our original blog post focused on the DataComp Large filtering track, in this work we further extended our data pool-size to the Extra-Large filtering track. This expands our pool-size by a factor of 10, going from 1B in our original blog post to 10B image-text pairs. We note that the while the Extra-Large filtering track indexes 12.8B image-text pairs, we were only able to obtain 10B pairs, which we primarily attribute to dead links. Our curation refined this data down to approximately 1B high quality image-text pairs.

Summary

Our previous blog post introduced a sophisticated data curation pipeline yielding improved CLIP model quality, faster training and reduced inference costs. We’ve built on these results, advancing our curation in several key areas, ranging from improved target distribution matching and synthetic data workflows to more robust engineering workflows. As a result, CLIP models trained with DatologyAI curated data achieve SoTA performance across a range of evaluations at a fraction of the compute cost.

Our journey is just beginning. We’re actively improving our multimodal data curation pipeline and associated research. Our nexts steps will be to focus on data curation vision-language models (VLMs).

As we continue to innovate, we invite collaboration and feedback from the broader community. If you’re an enterprise AI company seeking faster, better, or more cost-effective training solutions for multimodal and text models, we’d love to connect. Get in touch here and join us in shaping the future of AI data curation.

If you are interested in pushing the boundaries of what is possible with data curation, we are actively recruiting across a range of roles. We are particularly excited about researchers and engineers who have experience doing data research, building research tooling, translating science into products, and building scalable data products.

Contributors

Core Contributors

Ricardo Pio Monti   •   Haoli Yin   •   Amro Abbas   •   Alvin Deng   •   Josh Wills

Contributors

Cody Blakeney   •   Paul Burstein   •   Aldo Carranza   •   Parth Doshi   •   Pratyush Maini   •   Kaleigh Mentzer   •   Luke Merrick   •   Fan Pan   •   David Schwab   •   Jack Urbanek   •   Zhengping Wang

Leadership and Advising

Matthew Leavitt   •   Bogdan Gaza   •   Ari Morcos

For attribution in academic contexts, please cite this work as

Monti et al., “CLIP Gets a Data Upgrade: Outperforming SoTA with Improved Data Curation Only”, 2025.

@techreport{monti_clip_upgrade_2025,
  title = {{CLIP} {Gets} a {Data} {Upgrade}: {Outperforming} {SoTA} with {Improved} {Data} {Curation} {Only}},
  url = {https://blog.datologyai.com/multimodal-plus-blogpost},
  institution = {DatologyAI},
  author = {DatologyAI Team},
  month = jun,
  year = {2025},
}

Appendix