Introduction
Generative AI runs on data. A large language model learns everything it knows from a vast body of text, so before any training can start that text has to be gathered and prepared, and preparing it is one of the hardest and least visible parts of building the model. This material is a practical, self-paced introduction to how that preparation is done, written for practitioners who can already code but are not data specialists; plan on about an hour to read it, longer if you stop to work the exercises. Specifically we treat data processing: the work of turning raw text into something a model can learn from. It is one neighbor in a larger family that includes data management and data governance. Most of that data is training data, but the same care applies to the smaller evaluation and fine-tuning sets a model also needs.
What follows comes from our experience training large language models at Fraunhofer IAIS, across several projects: OpenGPT-X, OpenEuroLLM, and SOOFI. OpenGPT-X produced Teuken, a family of open models trained on all 24 official languages of the European Union. The lessons are drawn from building real training data and generalized so they transfer to any large-scale text pipeline.
You should be comfortable reading code: the examples are short Python snippets, there to make an idea concrete, never for you to write or run. A general sense of what a language model is (text in, next-token prediction out) is enough on the modeling side. No prior experience with data pipelines, web crawling, or natural-language processing is assumed; building that understanding is the point of this material.
Learning objectives
By the end of this material you will be able to:
- Explain why data processing matters for what a model can do.
- Define the basic concepts of data processing, starting with data quality and filtering (Part 0).
- Trace how raw text, whether curated or scraped from the open web, becomes clean, uniform training data (Part 1).
- Reason about why scale makes compute the binding constraint rather than logic alone (Part 2).
- Distinguish the two things a pipeline produces at once, the data and the traceable record of how it was made, and say why the record matters as much as the data (Parts 1–2).
- Explain why quality signals are proxies for judgments too expensive to run directly, and why good text is approximated rather than measured (Parts 3–4).
Each part opens with its own, more specific objectives.
In this material you will meet short “predict, then reveal” checks and small reasoning tasks that mark where it is your turn to think.
Why this matters
A language model is a product of its training data. It has no knowledge of the world beyond the text it was shown, so the size, the languages, and the quality of that text set the ceiling on what the model can do. Get the data wrong and no amount of clever training recovers it. This is also where transparency is thinnest: how a model’s data was processed is rarely reported in any detail, which is part of why the work is worth teaching.
None of that text arrives ready to use. It has to be found at the scale of hundreds of terabytes, cleaned, labeled, scored for quality, and deduplicated. You may never run a pipeline this size yourself, but the lesson still applies: the datasets you download and train on were shaped by exactly these steps, and knowing how they were made tells you what you can trust them for.
Two questions run underneath all of it and shape the rest of the material: where does the enormous amount of suitable text a model needs come from, and what does suitable even mean?
How this is organized
Part 0 fixes the basic vocabulary. From there the material follows the data through four parts. Part 1 builds the pipeline: how raw text, whether curated or web, becomes a single clean training format. Part 2 turns to scale, where compute cost becomes the binding constraint. Part 3 confronts the question the pipeline leaves open: how to judge whether text is any good. Part 4 answers it as far as it can be answered: every quality signal is a cheap proxy for a judgment too costly to run directly, and what makes text good is approximated, never defined outright. A short close returns to the question we started from, and what the four parts have answered.
Part 0 Basic concepts
Before following data through the pipeline, it helps to fix a little vocabulary. If you are already comfortable with these terms, skip ahead to Part 1.
- Place data processing among its neighbors, data management and data governance.
- Name the kinds of data a model uses: training, evaluation, and fine-tuning.
- Define data quality, and explain why it has no single fixed definition for training text.
- Describe what a filter does, and how it differs from the signal it acts on.
- Distinguish wall-clock time from compute time, the two budgets every large run lives within.
Where data processing sits
Three terms sit close together and are easy to confuse. They name different jobs over the same data. Definitions vary between frameworks, but a useful working split is:
- Data governance sets the rules: who may use which data, for what, under which policies, and who is accountable. It is about decisions and oversight.
- Data management carries those rules out across the data’s life: storing, cataloging, securing, and keeping it available and correct.
- Data processing is one step inside management: transforming raw data into a usable form. For us, that means turning messy raw text into clean training data.
Put simply, governance decides what is allowed and required, management keeps the data, and processing transforms it. This material lives in the last of the three. It touches the other two only where they constrain the processing, for example when a record must stay traceable so a fault can be assigned later.
Kinds of data
Not all data plays the same role. The bulk is training data, the text a model learns from and the subject of this material. Smaller sets serve other purposes: evaluation data measures how well a trained model performs, and fine-tuning data adapts it to a particular task or style. Evaluation and fine-tuning sets are far smaller in volume than the training corpus, though the same quality concerns apply to all three.
Data quality
There is no single number that captures whether text is “good.” The closest standard definition comes from ISO/IEC 25012, which describes data quality as the degree to which the characteristics of data satisfy stated and implied needs under specified conditions (International Organization for Standardization 2008). An older and more intuitive phrasing calls quality fitness for use: data is high quality when it serves the purpose you have for it.
That phrasing carries a warning for everything that follows. Quality is defined against a use, and the use here is training a capable language model, so what counts as “good text” depends on a downstream outcome no filter can observe directly. Parts 3 and 4 return to this.
Filters and signals
Two words appear throughout the pipeline and are worth separating now.
A signal measures something about a piece of text: the share of characters that are symbols, the average number of words per line, a language-detection confidence, a model’s quality score. A signal is a measurement.
A filter is the decision that acts on a signal. It sets a threshold and keeps or drops each document accordingly. A signal tells you a document is 60% symbols; a filter is the rule “drop anything above 30%.”
Keeping the two ideas distinct matters later, when we ask what a filter can and cannot know. A filter is only ever as good as the signal beneath it, and a signal measures the surface of the text; what the words actually mean stays out of its reach.
Compute and time
A web-scale pipeline runs on shared high-performance computing (HPC) clusters, where two different budgets govern what is possible. They are easy to conflate, so it is worth separating them now.
Compute time is the total machine work a run consumes, measured in CPU hours and GPU hours: one processor busy for one hour is one core-hour. It is what the cluster bills, and what a project’s compute budget is fixed in.
Wall-clock time is the elapsed calendar time from start to finish. It is set by the longest chain of steps that must run in order, and by how much of the work can run in parallel. The same ten thousand core-hours can finish in an afternoon on enough machines or stretch over months on few.
The two come apart constantly: a stage can be cheap in compute yet slow on the clock, or the reverse. A real project is handed both a fixed compute budget and a deadline, so the two have to be managed together. Part 2 follows a concrete run where this split decides the shape of the whole pipeline.
With the vocabulary in place, Part 1 follows raw text through the pipeline that turns it into training data.
Part 1 The pipeline
How raw text becomes training data. We follow two paths, curated and web, from their very different starting points to a single shared output.
- Define curated data by the editorial process behind it, and say why curated and web sources take different paths through the pipeline.
- Trace a single document from raw input to the shared JSONL record, naming what each step cleans and what metadata it records.
- Explain why a pipeline produces two things at once, the cleaned text and the record of how it was made, and why that record is what makes work at scale possible.
Not all text is equal
A model learns from whatever text it is shown, so the composition of the data (its sources, its domains, its quality) shapes how the model behaves (Gao et al. 2020; Penedo et al. 2024).
Raw text from the internet is uneven:
- clean prose sits next to navigation menus, cookie banners, and boilerplate
- the same page can appear thousands of times
- some text is spam or machine-generated filler
More text is not automatically better text. The job of a data pipeline is to turn a large, messy pile of raw text into a smaller body of training data that is actually worth learning from.
The fork: curated vs web
That job starts by splitting the raw material by where it came from: two kinds of source need two kinds of handling.
Curated sources are collections that already passed through a human curation process before we ever saw them: a book has an editor, an encyclopedia has reviewers, parliamentary proceedings have an official transcriber. That editorial care, someone whose job was to make the text correct and coherent, is what makes them relatively clean, relatively small, and high value per token. They take the light path, precisely because most of the quality work was already done upstream.
Web data comes from large crawls of the open internet. It is enormous and cheap to obtain, but mostly low value. It takes the heavy path: it needs aggressive cleaning before it is usable (Penedo et al. 2024).
The next two sections follow each path from its raw input to the shared JSON record they both produce.
Recap
Two very different inputs, curated and web, now share one shape: clean text paired with the metadata that records where it came from and how it was scored. That pairing is the real result of Part 1, and it is two outputs at once, both of which have to hold: the quality of the data, and the traceability of how it was produced.
So far we have treated each step as a matter of logic: what to keep, what to drop, what to record. Part 2 changes the question. At the true size of these corpora, every one of those steps is also a compute decision, and the traceability we just built is what makes the whole effort survivable.
Part 2 Working at scale
When you treat the pipeline as pure logic, every step has a clean answer: keep this, drop that, record the rest. Put it at the true size of a web corpus and a second constraint appears behind every one of those answers, a price measured in machine time. This part is about what changes once compute is the thing in short supply, because some of the answers from Part 1 stop being affordable and others stop being possible.
- Define compute time and wall-clock time as two separate budgets.
- Reframe a 97% removal rate as the expected outcome at web scale.
- Explain why deduplication is the biggest compute cost yet not the slowest stage, and why that fixes its place in the pipeline.
- Identify ordering, storage, and lineage as the load-bearing engineering decisions that emerge at this scale.
Scale changes the problem
The web side of Teuken’s data started from 60 Common Crawl dumps, spanning late 2014 to early 2023. Together they held about 173 billion documents, roughly 703 terabytes of raw text, around 2,7 billion documents per dump. Numbers this large change what the steps from Part 1 mean in practice.
Take filtering. On average, 96,94% of web documents are removed during filtering (Brandizzi et al. 2024). The introduction set out why a number like that is a sign of success rather than failure: at web scale the input is known to be mostly navigation, boilerplate, and spam, so aggressive removal is the job working as intended. What the headline figure hides is how much still survives. Roughly 3% of 173 billion documents is on the order of five billion documents of far cleaner text, a large corpus in its own right.
Numbers this size are hard to feel, so start from a landmark and work outward. The founding of Rome was about 2.800 years ago, already deep history. Yet reading only the five billion documents that survive filtering, at a normal pace and never stopping, would take some 35.000 years, more than ten times as far back as Rome, as old as the first cave paintings. And the full raw collection of 173 billion documents would take on the order of 1,2 million years, about four times longer than our species has existed.
A web corpus already holds far more text than any model needs, so the binding constraint becomes machine time. From here on the work is rationed by compute, and that budget is spent very unevenly across the steps.
Every decision is a compute decision
Once a step has a price, ordering it is an optimization. The pipeline runs its cheapest checks first so the expensive ones only ever see documents that already survived. A prefilter that costs almost nothing per document removes the obvious junk before any costly scoring or comparison runs, which means the expensive stages process a far smaller pile.
Scale also turns two more concerns into load-bearing engineering decisions:
- Lineage and versioning. Each stage writes its output in the same JSONL format and saves intermediate results, so any version can be compared with the one before it and a fault can be traced to the stage that introduced it. This is the traceability from Part 1.
- Storage and movement. Those intermediate results are copied to remote storage to guard against loss, which costs disk; and moving hundreds of terabytes between a processing cluster and a training cluster is slow and expensive enough that keeping the data where the model will train is itself a design choice.
None of this is visible in the final corpus, yet it is what keeps a multi-month pipeline from becoming unrepeatable.
Take the failure from Part 1’s metadata exercise, the corpus that turns out weak in one language (Part 1). If the pipeline saved its outputs after language detection and after later filtering, the two versions can be compared and the loss traced to the stage that introduced it. If those intermediate snapshots were discarded to save storage, diagnosing the failure turns into rerunning months of work.
Poolside’s model factory describes an industrial version of exactly this idea: treating data processing as a versioned, monitored production line rather than a one-off script.
Deduplication is the expensive step
The clearest example of a logical step that is really a compute decision is deduplication. Of all the CPU time spent processing the web data, deduplication consumed about 80,8%, with filtering at 16,7% and format conversion at 2,5% (Brandizzi et al. 2024). Put in human terms, the three stages together come to about 273.952 CPU hours, close to 31 years of work for a single CPU core. Deduplication is around 25 of those years on its own. A cluster runs it in months by splitting the work across many cores, but the raw cost is the size of three decades on one machine.
That single number sets the pipeline order. Because comparing documents for similarity is so costly, you run it last, on the small fraction that survived everything else, never on the full raw pile. Order the stages the other way and you would pay the most expensive step on the 97% you were about to discard.
Deduplication is the most expensive stage and runs last; filtering removes about 97% of documents and runs earlier. Suppose an engineer, wanting “the cleanest data first,” reorders the pipeline to deduplicate before filtering. The output documents are identical either way. So what exactly does the reordering cost, and roughly by how much?
The factor that drives the cost is the number of documents deduplication has to compare: more documents means more comparisons, so the cost rises with the size of the input pile. Since filtering removes about 97% of documents, the “dedup first” order makes the most expensive stage process roughly 30 times as many documents for the same final corpus. Put in the same single-core terms as above: the ~25 core-years deduplication already costs would balloon to on the order of 750 core-years to produce an identical output. That is the whole reason the cheap, decisive cuts come first and the costly comparison comes last.
Compute time versus wall-clock time
Total CPU time measures the work; how long the calendar takes is a separate question. Every stage already spreads across thousands of cores, so they all parallelize equally well. What separates them, and sets the calendar, is throughput: how much data each stage clears per day, which depends on how much work a single document costs it.
Deduplication makes the point. It is the largest compute cost of all, four-fifths of the budget from the previous section, yet it moves through the data quickly: the cluster clears about 20 terabytes of raw data a day. The reason is the kind of work it does. Deduplication compares documents with a simple whitespace tokenizer and numeric similarity algorithms, cheap operations per document, so even though there are an enormous number of comparisons, each one is fast.
Computing the quality signals is the opposite case. Every document is read in full: content analysis and tokenization, the heavy lifting of the pipeline, run on each one, so the stage clears only about 6 terabytes a day. Downloading the raw data is slow for a different reason: its speed is set by network bandwidth from the source, not by how many cores you add, and it manages a similar 6 terabytes a day. These two stages, three to four times slower than the rest, set the pace, because the whole pipeline can finish no faster than its slowest stage.
This is the gap between compute and time. Pushing roughly 725 terabytes of raw web text through the pipeline, held to about 6 terabytes a day at the two slow stages, takes on the order of 116 days, close to four months, even on a large cluster.
And wall-clock time, not total compute, is the budget that actually binds. It is tied to the project’s lifetime: a project is allotted a compute window, and the corpus has to be ready inside it. Four months of wall time inside a one-year project is a very different proposition from the same corpus needing two years. Total core-years is what you pay for; wall-clock time is what you live by.
Total compute time and wall-clock time are two separate budgets, and a real project is handed both at once: a fixed compute allocation and a deadline. Wall-clock time, tied to the project’s lifetime, is the one that ultimately binds, so the job is to fit the corpus inside the deadline without spending more than the compute you were given.
Recap
Scale does not add new steps so much as it re-prices the ones we already have. Once every decision carries a cost in machine time, the question behind each step shifts from what is correct? to what is affordable? That shift is the lesson of this part: at web scale, preparing data becomes an exercise in economics as much as logic. The order of the pipeline, the reach of a step like deduplication, even whether a step is possible at all, are settled by the compute budget rather than by correctness alone.
Both parts so far have rested on one assumption: that cheap, surface-level signals can separate good text from bad. Part 3 puts that assumption to the test.
Part 3 The quality question
Part 1 built a pipeline that asks one question of every document: is this well-formed? The cheap surface checks answered it by counting characters and lines, and Part 2 showed that they scale to hundreds of terabytes precisely because they are so cheap. This part asks the question those checks cannot reach: of the documents that are well-formed, which ones are actually worth learning from? That is the quality question, and it is where the easy answers end.
- Distinguish a document’s form (is it well-formed?) from its value (is it worth learning from?).
- Explain why surface heuristics, for all their speed and transparency, cannot see content value.
- Compare an LLM judge against a surface heuristic in real terms: what it gains in judgment, and what it costs in money and hardware.
Asking a model to judge
If the missing instrument has to read meaning, the obvious candidate is a language model: hand it the full document and ask it to score the value the heuristics could not see. Scoring value means choosing which kind of value to measure. The one put to work at web scale is educational value: how much genuine, learnable information a document carries, rated on a small scale such as 0 to 5 (Penedo et al. 2024). Because the model actually reads the text, it can tell the explainer from the spam in a way no counting rule can.
HuggingFace’s FineWeb blog post walks through building exactly such a judge: an LLM scores documents for educational value, and those scores train the FineWeb-Edu quality classifier that filters a web-scale corpus.
The difficulty is that this power is expensive. A surface heuristic is a few arithmetic operations per document; an LLM judgment is a full forward pass through billions of parameters, far more work each time. Multiply that by the roughly five billion surviving documents, and compute becomes the real obstacle.
The cost in euros
Of those two prices, the compute one can be made concrete. When you build a pipeline like this, the budget decides what you can actually run, as much as the quality you are aiming for does. The budget depends on two things: how much work there is, and what hardware it runs on.
CPU work and GPU work
The two kinds of processor are good at different things. A GPU packs thousands of small cores that all run the same arithmetic at once, which makes it fast at the dense matrix multiplication inside a neural network and slow at much else. A CPU has a few powerful, flexible cores built for sequential logic: branching, comparing, and shuffling data around. Most of a data pipeline is that second kind of work, converting file formats, hashing and comparing documents, reading and writing bytes, none of it matrix arithmetic. So it runs on CPUs, where an hour of compute is cheap.
Recall the Part 2 budget. Conversion, filtering, and deduplication together took about 273.952 CPU hours. At a representative cloud price of four to five cents per CPU hour, that whole pipeline costs on the order of 12.000 € to run once. For hundreds of terabytes of processed text, that is remarkably little.
Now imagine running the same work on GPUs, on the intuition that GPUs are faster. A GPU hour costs roughly seventy times a CPU hour, and the work gains nothing from one: hashing and string comparison cannot use the matrix hardware you would be paying for. The same pipeline would cost on the order of 800.000 €, and finish no sooner.
Pay for a GPU only where the work is the dense, parallel arithmetic it is built for. Everything else belongs on a CPU, where it runs just as fast for a fraction of the price.
The judge at web scale
That rule has one exception in this pipeline, and it is the LLM judge. Reading a document for meaning is the dense matrix arithmetic a GPU is built for, so here the expensive hardware earns its price. The bill now has two multipliers working together: a GPU hour is expensive, and there are billions of documents to score.
A heavy judge, a model in the tens of billions of parameters, scores on the order of ten thousand documents per GPU hour. At about 3 € per GPU hour that is roughly 0,0003 € per document, which sounds negligible until you count the documents. Filtering leaves on the order of five billion, so judging all of them once comes to about 1,5 million € and around half a million GPU hours, more than fifty years of nonstop work on a single card.
The shape is the one Part 2 made concrete: a cost that is tiny for one document grows enormous once it is paid billions of times over. Plotted against the number of documents, the effect is plain. Judging a small sample is inexpensive; judging the full corpus of survivors costs millions of euros:
So the obstacle is the economics. A heavy judge reads meaning more reliably than any counting rule, yet running one across the whole web costs millions of euros and years of GPU time. That turns the next step into a sharp question: can a pipeline keep the judgment of a language model while paying something close to the price of a surface heuristic?
Recap
Part 3 separated two questions that Part 1’s surface heuristics had treated as one. Form asks whether a document is well-formed; value asks whether it is worth learning from. Heuristics settle form cheaply and at any scale, but they measure only the shape of the text, so value stays invisible to them. A language model can read the text and judge its value, but doing that across billions of documents is slow and expensive.
That is the open problem this part leaves on the table: how to get the judgment quality of a language model at the running cost of a surface heuristic. Part 4 is the answer this material follows, and it solves the problem with a move in the spirit of Part 2: trade a large one-time compute cost for a cheap one that runs at scale.
Open-source frameworks package these stages so you do not build them from scratch: DataTrove (the pipeline behind FineWeb) and NVIDIA’s NeMo Curator both handle filtering, deduplication, and large-scale processing. For the LLM-judge idea specifically, propella-1 annotates documents across many quality properties at once rather than a single score, and releases the resulting annotations as a dataset.
Part 4 Cheap proxies for good text
Part 3 ended on a sharp question: can a pipeline keep the judgment of a language model while paying something close to the price of a surface heuristic? Answering it means facing a harder question the pipeline has so far avoided. Every step has sorted documents into good and bad without ever saying what good means: Part 1 caught malformed text and removed duplicates, Part 3 scored surface form, yet none of them defined the quality they were chasing. This part confronts that gap first, then shows the practical answer a real pipeline uses.
- Explain why every quality signal is a cheap proxy for a judgment too expensive to run directly.
- Define what makes text bad from good (open, and settled only after training).
Bad is easy to spot, good is not
Defining bad text is straightforward. Bad text has signs you can count: it is too short, it is mostly symbols, it repeats one line, it is a duplicate of something you already kept. Part 1 caught all of these by counting characters, because a malformed document reveals itself in its shape.
Good text has no such signs. There is no measurement you can run on a single document that returns “this is worth learning from”, because worth is fitness for use (Part 0): it depends on the goal you are training for and on what the model still needs to learn, neither of which is written on the page. So nobody filtering data ever truly writes down what good is. They pick a proxy for it: a property they can measure on each document, chosen because it tends to track quality even though it is not quality itself. The proxy that FineWeb introduced is educational value: how much genuine, learnable information a document carries, scored on a small 0-to-5 scale by a classifier trained on LLM-written ratings (Penedo et al. 2024). It was chosen for three reasons, in rising order of importance: it is intuitive to people, it is cheap enough to score across the whole web, and, above all, models trained on the data it selects do better on the benchmarks. Educational value is a deliberate choice, open to debate, which is why even trained human annotators disagree on the borderline cases (Ali et al. 2025).
This difference shapes the rest of the work. You can define bad up front and act on it cheaply. For good, you choose a proxy, apply it, and only find out whether the choice was sound much later, by training a model on the result and measuring how it does.
Every quality signal is a cheap proxy
Seen this way, the tools in Parts 1 to 3 share one idea. A surface heuristic is a cheap proxy for “is this well-formed?”. An LLM judge is an expensive proxy for “would a careful reader call this good?”. Each replaces a judgment we cannot make directly with a measurement we can.
Every quality signal is a cheap proxy for an expensive judgment. The recipe is always the same: take the most trustworthy judgment you can afford, then distill it into a cheaper one, and repeat, each step trading a little accuracy for a large cut in cost, until the signal is cheap enough to run at the scale you need.
That recipe stacks the judges in tiers, three of them. It helps to start at the most trustworthy, most expensive tier and work down, since each tier exists only to make the one below it affordable.
The top tier is human judgment. The closest thing to a true measure of whether a document is worth learning from is a person reading it, the way an editor decides what belongs in a book. It is also the costliest: a qualified annotator spends minutes per document and is paid by the hour, so scoring runs into euros per document, against fractions of a cent for a model. Reading the roughly five billion surviving documents by hand is impossible, so people never judge the whole corpus. They score a small, curated seed instead, the same kind of high-trust curated data from Part 1.
The middle tier is a large language model, calibrated against that seed. A capable model can read a document and rate it much as a person would, and the seed is how you check that it does: you keep the model whose scores track the human ones. But a model big enough to judge well is still far too expensive to run on the whole web, around 1,5 million € for a single pass (Part 3). So it never sees every document either. It judges a larger sample, big enough to teach the next tier but far short of the full corpus.
The bottom tier is a small model that runs everywhere. The sample the large model judged becomes training data for a small, cheap model that reproduces its verdicts at a fraction of the cost, light enough to score every surviving document. Each tier, in short, trains or calibrates the one beneath it: people vouch for the large model, and the large model labels the data that teaches the small one.
This three-tier chain is what JQL (Judging Quality across Languages, pronounced Jackal) (Ali et al. 2025) builds; it came after OpenGPT-X and sharpens the same idea, and the details belong to the paper. The shape is always the same: pay one large cost once, on a sample, so a cheap step can run at scale.
Plot the tiers by cost and coverage and they fall on a single downward line: the cheaper a judgment is per document, the more documents it can afford to run on. The costliest tier, human review, reaches only a curated seed; the cheapest, surface filters, runs across the whole web:
Each tier down is cheaper than the one above, which invites one last question: could you take one more step and drop the model too? If some surface heuristic tracked the model’s verdicts closely enough, you could filter with plain character counts and skip the reading entirely. Nobody has found one that does, and Part 3 is the reason to doubt one exists: surface form does not determine value, so a rule that only counts characters has nothing to read meaning with. Whether a rich enough set of surface features could ever line up perfectly with content is genuinely open; what we know is that none does today. So in practice the chain bottoms out at a small model that reads what the words mean, and for now there is no cheaper tier beneath it.
The distilled model now puts a cheap value score on every surviving document, which turns the next step into a question: given a score for each document, where do you set the cutoff?
A wider family of proxies
Every proxy so far has done the same thing: score a whole document from its text, by its shape with a surface heuristic or by its meaning with an LLM judge, then keep it or drop it. Those are not the only instruments. Two more are worth knowing, because they show how broad the idea of a quality signal really is.
The first reads something other than the text at all. A document carries implicit signals from how the web treated it: how many pages link to it, where a search engine ranks it, how often people click or stay. These judge a page by its reception rather than its words, so they need neither a surface heuristic nor an LLM judge. OpenGPT-X did not use them, but they are a standing option.
The second does read the text, but asks a narrower question: how surprising is the text? To measure it, you can run a small language model over a document and measure its perplexity, how surprised the model is by the words: text that resembles the clean reference the model was trained on scores low, while gibberish, boilerplate, and spam score high.
Taken together, these signals are a reminder that quality can be guessed from almost anywhere, not only a literal reading of the words on the page.
What “good” still leaves open
JQL gives a strong, cheap proxy for good text and runs it across every language. It still leaves the definition of good itself open. Each step in this material has approximated good with something it could measure: Part 1 with well-formedness, and with whether a document added anything new or merely repeated what was already kept, Part 3 with those same surface heuristics, which it showed cannot reach value, and Part 4 with a model’s value score. Underneath all of them, the final test of whether any of it worked is how the trained model does on a benchmark, which is itself a proxy for the quality we actually care about. The approximations stack, and at the bottom there is no measurement that reads “this text is good”.
This is the honest shape of the problem. Defining good text well enough to measure it, and across every language at once, is open research, and the choice of proxy decides what kind of model the data will make.
Recap
Part 4 reframed the whole pipeline as a sequence of proxies. Bad text is definable and can be filtered cheaply; good text cannot be defined outright, so every quality signal is a cheap proxy for it, something measurable, and the only test of the choice comes downstream. Because a proxy good enough to read value is expensive to run, the move is to pay for it once on a sample and distill it into a cheap model that runs on the billions. Filtering on rank rather than raw score makes that proxy behave consistently across judges and languages. What remains open is the thing underneath: what makes text good at all.
Closing
This material set out to answer one question: where does the enormous amount of suitable text a language model needs come from, and what does suitable even mean. Four parts on, both halves have an answer. Where the text comes from is now concrete: curated and web sources, drawn along the two paths of Part 1 and reduced, cheaply and in the right order, to the clean fraction worth keeping. What suitable means turned out to be the harder finding: bad text can be defined up front by the tells you can count, but good text has none, so it can only be approximated by a proxy and confirmed after the fact.
Read back through the parts, here is what each was meant to leave you with:
| 1) The pipeline |
Turn curated and web text into one clean, shared record, and keep that record traceable. |
Save the metadata and intermediate outputs as you go. |
| 2 Working at scale |
At web scale every step is also a compute decision; wall-clock time, tied to the project’s compute window, is the budget that binds. |
Run the cheap, decisive cuts first and the expensive comparisons last. |
| 3) The quality question |
Cheap surface heuristics judge a document’s form but stay blind to its value; reading value means paying for a model. |
Never ask a surface filter whether text is worth learning from; it knows only the shape. |
| 4) Cheap proxies for good text |
Good text cannot be defined outright, so every quality signal is a cheap proxy you can confirm only after training. |
Treat a quality score as a hypothesis to test downstream, not a ground truth to trust up front. |
A language model is a product of its training data, so every decision taken before training, what to keep, what to call good, what to discard, is a decision about what the model can become. None of it shows in the finished weights, yet all of it is already there.
This material stops at preparing crawled and curated text. The biggest neighbor it leaves out is synthetic data, text generated by a model rather than collected, which sidesteps the crawl entirely but brings its own quality and diversity questions (see this survey of best practices). Tokenization and the mixing weights that set how much of each source a model sees are likewise their own topics.
A closing word on where to take this. The field moves quickly, and the specifics will date faster than the principles: the libraries turn over and the judges keep improving. What carries forward is the way of thinking, keep the data traceable, match each step to its budget, and treat every quality signal as a proxy you still have to check. Hold on to those, and the next tool will be easy to learn.
References
Ali, Mehdi, Manuel Brack, Max Lübbering, et al. 2025. “Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models.” Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 8870–909.
Brandizzi, Nicolo’, Hammam Abdelwahab, Anirban Bhowmick, et al. 2024. “Data Processing for the OpenGPT-x Model Family.” arXiv Preprint arXiv:2410.08800.
Gao, Leo, Stella Biderman, Sid Black, et al. 2020. “The Pile: An 800GB Dataset of Diverse Text for Language Modeling.” arXiv Preprint arXiv:2101.00027.
International Organization for Standardization. 2008.
ISO/IEC 25012:2008 Software Engineering — Software Product Quality Requirements and Evaluation (SQuaRE) — Data Quality Model. Standard. International Organization for Standardization.
https://www.iso.org/standard/35736.html.
Jansen, Tim, Yangling Tong, Victoria Zevallos, and Pedro Ortiz Suarez. 2022. “Perplexed by Quality: A Perplexity-Based Method for Adult and Harmful Content Detection in Multilingual Heterogeneous Web Data.” arXiv Preprint arXiv:2212.10440.
Lee, Katherine, Daphne Ippolito, Andrew Nystrom, et al. 2022.
“Deduplicating Training Data Makes Language Models Better.” Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8424–45.
https://doi.org/10.18653/v1/2022.acl-long.577.
Leveling, Johannes, Lennard Helmer, Benny Joerg Stein, et al. 2024. “Evaluation of Document Deduplication Algorithms for Large Text Corpora.” International Conference on Machine Learning, Optimization, and Data Science, 390–404.
Penedo, Guilherme, Hynek Kydlı́ček, Loubna Ben allal, et al. 2024.
“The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.” Advances in Neural Information Processing Systems 37: 30811–49.
https://proceedings.neurips.cc/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_Track.html.
Wang, Richard Y., and Diane M. Strong. 1996.
“Beyond Accuracy: What Data Quality Means to Data Consumers.” Journal of Management Information Systems 12 (4): 5–33.
https://doi.org/10.1080/07421222.1996.11518099.
