DatologyAI

Two weeks ago, we at DatologyAI released our first results demonstrating massive gains from data curation on contrastive image-text models. Today, I'm incredibly excited to share our newest results, applying our curation pipeline to LLMs. It's absolutely astonishing to see what a small, incredibly talented group of individuals can accomplish, and boy have we cooked! Starting with an exact-deduplicated version of Red Pajama v1 as our baseline and by manipulating only the training data for the model: Train Faster -- Training on our curated data reached the same baseline performance 7.7x faster, meaning results cost dramatically less to obtain and drastically improving iteration speed. Train Better -- Push the frontier of what's possible with a given budget, improving performance by 8.5 absolute percentage points (60.5% Datology vs. 52.0% RPJv1). This isn't just because of Red Pajama: compared to the strongest publicly curated datasets, DataComp-LM and FineWeb-Edu, improve performance by 4.4% and 6.1%, respectively. Train Smaller -- Better data enables you to train smaller models. Reduce cost per query at inference by 2.1x while simultaneously increasing performance over the baseline by 5.7%. As with our image-text results, we present these results both at a high-level (https://lnkd.in/g_hMR5Tx) and with an extremely meaty technical deep-dive for all of you who want the nitty-gritty details (https://lnkd.in/gY5tpq3s). We are just getting started on our journey and are so excited about what's in store. Are you training or customizing your own text models and want to improve performance, training efficiency, and inference efficiency through better data? Get in touch (https://lnkd.in/gSGckr6s)! Are you a data-obsessed researcher, engineer, or somewhere in between who wants to push the bounds of what's possible with better data? We're hiring Members of Technical Staff across a number of roles (https://lnkd.in/gHCwPk8e).

Models are what they eat: high quality data lead to high quality models, enabling faster training of better models with fewer parameters. However, identifying and curating high quality data at scale, automatically, is an incredibly challenging problem requiring deep expertise. Our goal at DatologyAI is to make state of the art data curation accessible to anyone who wants to train a model, and we’ve been hard at work realizing this vision over the last year. On a personal note, I am so proud of the incredible work our small, but mighty team has accomplished, and today, I’m incredibly excited to share our first set of results at DatologyAI! We focused on contrastive models (ala CLIP) trained on the large-scale DataComp dataset, and the results we’ve been able to achieve have exceeded our already high expectations! Train Faster - Training on DatologyAI’s optimized dataset, we were able to reach the same performance with up to ~98% less compute, meaning that models cost dramatically less to train and train dramatically faster! Train Better - Models trained on our optimized data for the same compute budget achieve up to 13 absolute percentage points better performance relative to models trained on raw data. Train Smaller - Train models with >60% fewer parameters to better performance by training on our curated data. Check out our high-level blog post here (https://shorturl.at/jkYqk), and if you’re interested in all the nitty, gritty details, check out our technical deep dive here (https://shorturl.at/Mt0k9). We are so excited about these results, and we are just getting started! Stay tuned for more exciting results on text models coming very soon!

Models are what they eat: high quality data lead to high quality models, enabling faster training of better models with fewer parameters. However, identifying and curating high quality data at scale, automatically, is an incredibly challenging problem requiring deep expertise. Our goal at DatologyAI is to make state of the art data curation accessible to anyone who wants to train a model, and we’ve been hard at work realizing this vision over the last year. On a personal note, I am so proud of the incredible work our small, but mighty team has accomplished, and today, I’m incredibly excited to share our first set of results at DatologyAI! We focused on contrastive models (ala CLIP) trained on the large-scale DataComp dataset, and the results we’ve been able to achieve have exceeded our already high expectations! Train Faster - Training on DatologyAI’s optimized dataset, we were able to reach the same performance with up to ~98% less compute, meaning that models cost dramatically less to train and train dramatically faster! Train Better - Models trained on our optimized data for the same compute budget achieve up to 13 absolute percentage points better performance relative to models trained on raw data. Train Smaller - Train models with >60% fewer parameters to better performance by training on our curated data. Check out our high-level blog post here (https://shorturl.at/jkYqk), and if you’re interested in all the nitty, gritty details, check out our technical deep dive here (https://shorturl.at/Mt0k9). We are so excited about these results, and we are just getting started! Stay tuned for more exciting results on text models coming very soon!

Models are what they eat: high quality data lead to high quality models, enabling faster training of better models with fewer parameters. However, identifying and curating high quality data at scale, automatically, is an incredibly challenging problem requiring deep expertise. Our goal at DatologyAI is to make state of the art data curation accessible to anyone who wants to train a model, and we’ve been hard at work realizing this vision over the last year. On a personal note, I am so proud of the incredible work our small, but mighty team has accomplished, and today, I’m incredibly excited to share our first set of results at DatologyAI! We focused on contrastive models (ala CLIP) trained on the large-scale DataComp dataset, and the results we’ve been able to achieve have exceeded our already high expectations! Train Faster - Training on DatologyAI’s optimized dataset, we were able to reach the same performance with up to ~98% less compute, meaning that models cost dramatically less to train and train dramatically faster! Train Better - Models trained on our optimized data for the same compute budget achieve up to 13 absolute percentage points better performance relative to models trained on raw data. Train Smaller - Train models with >60% fewer parameters to better performance by training on our curated data. Check out our high-level blog post here (https://shorturl.at/jkYqk), and if you’re interested in all the nitty, gritty details, check out our technical deep dive here (https://shorturl.at/Mt0k9). We are so excited about these results, and we are just getting started! Stay tuned for more exciting results on text models coming very soon!