Training data collection has become one of the largest categories of web data work, and it has a workload profile unlike anything else in the field. Where a price monitor fetches a hundred thousand small pages and extracts three fields, a training corpus pipeline fetches tens of millions of pages, keeps almost all of the raw content, deduplicates aggressively afterwards, and runs continuously for months.
That profile changes nearly every decision. Bandwidth becomes the dominant cost rather than a line item. Coverage breadth matters more than extraction precision. Deduplication moves from a nice-to-have to a central engineering concern. And the legal and licensing questions, which most scraping work can treat lightly, become genuinely consequential.
This guide covers how to plan, build, and operate collection pipelines for model training and fine-tuning — from source selection and volume estimation through quality filtering, deduplication, provenance tracking, and the compliance posture that makes the whole thing defensible.
Why This Workload Is Different
Six characteristics that distinguish training data collection from ordinary scraping.
Volume is measured in terabytes. Not gigabytes. A corpus of ten million pages at 150KB each is 1.5TB before you've touched images, and most serious efforts are considerably larger.
You keep almost everything. Ordinary scraping extracts a few fields and discards the page. Training data collection keeps the content, because the content is the product. Storage requirements scale with collection rather than with extracted records.
Deduplication happens after collection, not before. You can't know a page is a near-duplicate until you have it. This inverts the usual efficiency logic — you deliberately collect redundantly and filter later.
Breadth beats depth. A hundred sources with a thousand pages each is usually more valuable than one source with a hundred thousand. Diversity is a quality property, not just a coverage one.
Runs are continuous and long. Months rather than hours. Infrastructure decisions compound, and anything that requires manual intervention will require it a hundred times.
Provenance matters more than in any other scraping context. You need to know where every document came from, when, under what terms, and whether you're permitted to use it. Downstream, someone will ask.
The practical consequence: the bottleneck is almost never extraction. It's bandwidth economics, deduplication at scale, quality filtering, and knowing what you've got.
Pretraining, Fine-Tuning, and RAG: Different Data Problems
Before planning any collection, be clear about which problem you're solving, because the data requirements differ by orders of magnitude.
Pretraining from scratch. Requires enormous, broad, diverse corpora — hundreds of billions of tokens at minimum for anything competitive. Quality matters but breadth matters more, and the collection effort is a serious infrastructure project. For nearly all organisations this is the wrong path; the cost is measured in millions and the result rarely beats an existing open model.
Continued pretraining. Taking an existing base model and continuing training on domain-specific data. Needs substantially less than pretraining from scratch but still large volumes — billions of tokens in a specialised domain. Viable for organisations with genuinely distinctive corpora.
Supervised fine-tuning. Adapting a model to a task or style using curated examples. This is where most practical value lives, and the data requirement is dramatically smaller — often thousands to tens of thousands of high-quality examples. Quality and consistency matter enormously; volume barely matters at all. A carefully curated set of five thousand examples routinely outperforms a sloppy set of five hundred thousand.
Preference and alignment data. Comparison pairs or ranked outputs. Almost always generated or annotated rather than scraped, and quality control is the entire game.
Retrieval-augmented generation. Not training at all — you're building a searchable knowledge base the model queries at inference time. Collection requirements are about coverage, freshness, and chunk quality rather than volume. Frequently the right answer when someone thinks they need fine-tuning, and it's far cheaper.
The practical guidance: most projects that begin with "we need to scrape the web to train a model" should be RAG or fine-tuning projects. Establishing which one you're actually doing before building collection infrastructure saves enormous amounts of wasted effort, because the pipelines look almost nothing alike.
Text Extraction and Content Processing
Getting clean text out of web pages is more consequential for corpus quality than almost anything else, and it's routinely under-invested.
Boilerplate is the enemy. Navigation menus, footers, sidebars, cookie banners, share buttons, related-content widgets, and advertising frequently account for the majority of a page's markup and none of its value. A corpus full of repeated navigation text is actively harmful — it teaches the model that boilerplate is what text looks like.
Content extraction approaches:
- Readability-style heuristics identify the main content block using density and structural signals. Fast, general, and imperfect.
- Site-specific extraction rules produce much cleaner output for sources you're collecting heavily. Worth writing for your top sources.
- Structured data first. Where a page includes JSON-LD, microdata, or an accessible API, use it. It's cleaner than anything you can extract from rendered HTML.
- Markdown conversion preserves structural information — headings, lists, code blocks, emphasis — that plain text extraction discards. For technical content this structure carries real signal.
Things worth preserving: heading hierarchy, list structure, code blocks and their language, table structure, and paragraph boundaries. Flattening everything to a wall of text throws away information the model could use.
Things worth removing: navigation, advertising, cookie notices, comment sections in most cases, share widgets, and repeated site furniture.
Encoding and normalisation. Handle character encoding properly, normalise Unicode, fix mojibake, standardise whitespace, and decide on a policy for HTML entities. These sound trivial and produce a surprising fraction of corpus quality problems.
Measure extraction quality per source. Ratio of extracted text to raw bytes, and manual review of samples. A source where extraction is producing 90% boilerplate should be either fixed with a custom rule or dropped.
Source Selection
The quality of a corpus is determined mostly at this stage, before any code runs.
Openly licensed sources first. Public domain texts, permissively licensed repositories, government publications, open access research, Creative Commons content, and datasets released explicitly for research. These carry the least risk and often the highest quality-per-byte. Exhaust these before scraping anything.
Existing corpora. Several large web-derived datasets already exist and are published for reuse. Starting from one and supplementing is usually cheaper and faster than building from nothing, and it gives you a well-understood baseline.
Structured and semi-structured web sources. Documentation, reference sites, forums with clear threading, Q&A sites, and technical wikis produce cleaner text with less boilerplate than general web pages.
General web crawl. The broadest source and the noisiest. High volume, low signal-to-noise, and requiring the most aggressive filtering. Necessary for scale, insufficient for quality on its own.
Domain-specific collections. For fine-tuning, a narrow high-quality corpus almost always beats a broad one. Targeted collection from authoritative sources in your domain is worth far more per byte than general crawl.
Multimodal sources. Image-text pairs, video transcripts, audio with captions. Bandwidth requirements multiply dramatically, and licensing questions get harder.
The selection principles that matter:
- Check licensing before collecting, not after. Retrofitting compliance to a corpus you've already built is painful and sometimes impossible.
- Prefer sources with clear terms over ambiguous ones, even at some cost in volume.
- Diversity across sources, languages, domains, and time periods is a quality property in itself.
- Document every source as you add it. The provenance record starts here.
- Weight by quality, not just availability. The easiest content to collect is frequently the least valuable.
Working With Existing Datasets
Since starting from published corpora is nearly always the right first move, a short guide to doing it well.
Know what's already covered. Several large web-derived corpora are published for reuse, along with many domain-specific collections — code repositories, academic text, legal documents, multilingual sets. Before collecting anything, establish what your target domain already has available.
Read the dataset card before the data. Composition, collection methodology, filtering applied, known limitations, and licensing. A dataset with no documentation is a dataset you can't defend using.
Check the licence carefully. Research-only, non-commercial, and share-alike terms all exist in this space and all constrain what you can do downstream. The licence on a derived dataset may differ from the licence on its sources.
Understand the filtering already applied. Heavily filtered datasets are convenient and may have removed things you wanted. Lightly filtered ones require you to do the work but preserve your options.
Check the collection date. Web corpora have a snapshot date, and anything after it is absent. For domains that move quickly, this matters a great deal.
Assess overlap before combining. Merging two published corpora frequently produces substantial duplication, since they often crawled the same web. Deduplicate across the combination, not within each.
Supplement rather than replace. The productive pattern is to start from a published general corpus and add domain-specific content it lacks. This is far cheaper than building general coverage yourself and produces a better result.
Document what you used. Which datasets, which versions, which licences. This becomes part of your own provenance record and you'll be asked about it.
Volume and Cost Planning
Underestimating here is the most common planning failure, and the errors are large.
Estimate consumption properly:
Two adjustments people forget:
- Success rate divides your budget. You pay bandwidth for failed requests too. At 70% success you're funding 43% more traffic than your document count suggests.
- Average size is usually underestimated. Measure it on a few hundred real documents from your actual sources. Raw HTML with boilerplate is much larger than the extracted text you're imagining.
Then add the multipliers:
- Redundant collection for deduplication means fetching more than you keep
- Redirect chains are billable traffic
- Retries multiply consumption on low-success sources
- Images and media, if multimodal, dwarf text by orders of magnitude
Choose the bandwidth model deliberately. This decision matters more here than in any other use case:
- Per-gigabyte residential billing is punishing at corpus scale. A pipeline moving several terabytes a month will generate an uncomfortable invoice.
- Unmetered datacenter bandwidth is dramatically cheaper and works on any source that doesn't check IP classification — which is a large share of the open web, including most of the highest-quality sources.
- Throughput-priced residential is the answer when sources require residential classification and volume is sustained and large.
Test classification requirements per source. Run five hundred requests through datacenter addresses against each significant source and record the status distribution. Route the permissive majority through cheap unmetered infrastructure and reserve expensive addresses for the minority that need them. This per-source segmentation is the single largest cost lever available and it's routinely skipped.
> Tip: Calculate cost per usable document after deduplication and filtering, not per fetched page. A pipeline with a 40% post-filter retention rate is paying 2.5x its apparent cost per document that reaches the corpus.
Infrastructure for Sustained Collection
Continuous multi-month operation has requirements that short campaigns don't.
Saturate the capacity you're paying for. On throughput-priced plans especially, idle capacity is wasted money, and most pipelines run far below their ceiling. The usual culprits:
- Default connection pool limits in your HTTP library. Nearly every library ships a conservative cap, and it's the most common invisible bottleneck.
- Insufficient concurrency. Push into the hundreds and measure where throughput plateaus.
- No keep-alive. Connection setup overhead at high request rates consumes real capacity.
- Synchronous bottlenecks. If parsing, deduplication, or storage writes run in the request path, they cap your throughput regardless of bandwidth.
Decouple everything. Fetch to raw storage. Process asynchronously from storage. Deduplicate as a separate stage. Filter as another. Each stage should be independently scalable and independently restartable.
Checkpoint aggressively. Multi-month crawls get interrupted by everything — deployments, network events, provider issues, your own bugs. Progress must be durable and resumption must be automatic.
Make requests idempotent. Retries and resumptions will refetch things. That should be harmless.
Build backpressure. A fetcher capable of saturating a large connection will overwhelm your storage or processing if you let it. Every stage needs the ability to signal upstream to slow down.
Spread load continuously rather than in bursts. Throughput plans reward steady utilisation, and targets tolerate steady load far better than spikes.
Per-source rate budgets. Aggregate saturation and per-site politeness aren't in conflict. Cap each source at a respectful rate and run many sources concurrently.
Multimodal Collection
If your corpus includes images, audio, or video, several things change substantially.
Bandwidth requirements multiply enormously. A page of text is 150KB; a single high-resolution image is comparable, and a set of images per page multiplies that. Video is a different category entirely. Any multimodal effort should assume unmetered bandwidth is mandatory rather than preferable.
Pairing quality is the whole game. Image-text pairs are only useful if the text actually describes the image. Alt text quality varies from excellent to useless to actively misleading. Caption extraction, surrounding-context heuristics, and filtering on pairing plausibility all matter more than raw volume.
Deduplication needs perceptual hashing. Byte-identical detection catches almost nothing, since the same image appears at many resolutions and compressions. Perceptual hashes catch visual near-duplicates and are essential rather than optional.
Storage costs become significant. Text corpora are cheap to store; media corpora are not. Decide on resolution and format policies early, and consider whether you need originals or processed derivatives.
Licensing is harder and stakes are higher. Image and video rights are more actively enforced than text, the provenance chain is often unclear, and reverse image search makes attribution straightforward for a rights holder. This is an area where legal input early is worth considerably more than usual.
Content safety filtering is mandatory rather than advisable. Scraped image collections require it, and the requirement is not negotiable for most applications.
Format and encoding normalisation consumes real engineering effort — codecs, colour spaces, sample rates, container formats.
Processing is compute-heavy. Unlike text, where extraction is cheap, media processing requires substantial compute for decoding, hashing, filtering, and normalisation. Budget for it as a separate line rather than assuming it's incidental.
Deduplication at Scale
The engineering problem that distinguishes corpus building from ordinary scraping, and the one most underestimated.
Web content is enormously redundant. Boilerplate repeats across every page of a site. Articles get syndicated across dozens of outlets. Templates produce near-identical pages. Without aggressive deduplication, a large fraction of a corpus is repetition, and duplicated training data has well-documented effects on model behaviour.
Exact deduplication catches byte-identical documents. Hash the content, store hashes, drop collisions. Cheap and necessary but insufficient — most redundancy isn't byte-identical.
Near-duplicate detection catches documents that are substantially the same with minor variations. The standard approaches are locality-sensitive hashing techniques — MinHash with LSH bucketing being the most widely used — which let you find similar documents without comparing everything to everything.
Substring and passage-level deduplication catches repeated passages within otherwise different documents. More expensive and often worth it, since boilerplate and syndicated blocks are pervasive.
URL-level deduplication catches the same content reachable at multiple addresses. Normalise aggressively: tracking parameters, trailing slashes, case, protocol, www prefix, session identifiers.
Practical considerations:
- Deduplicate incrementally, not as a single end-of-project batch. A months-long collection needs running deduplication or the final job becomes intractable.
- Deduplicate across the whole corpus, not within individual crawl batches, or you'll keep the same content once per batch.
- Keep the deduplication index separate from the corpus and design it to scale, because it grows with collection.
- Decide which copy to keep deliberately — earliest, highest quality, most authoritative source — and record the decision.
- Measure your duplication rate. It's a useful signal about source quality. A source producing 80% near-duplicates is contributing far less than its page count suggests.
Quality Filtering
Raw web content is mostly not worth training on. Filtering is where a corpus becomes valuable.
Boilerplate removal. Navigation, headers, footers, cookie notices, advertisements, related-article widgets. This is often the majority of a page's byte count and none of its value. Content extraction libraries handle the common cases; expect to tune per source.
Language identification and filtering. Keep what you intend to keep and label the rest. Mixed-language documents need handling decisions.
Length filters. Documents too short to be meaningful and pathologically long ones both cause problems.
Quality heuristics. Ratio of text to markup, punctuation density, proportion of stop words, average sentence length, repetition within a document. These are crude and effective.
Perplexity filtering. Scoring documents against a reference model and dropping outliers. More expensive, more effective, and standard practice at scale.
Toxicity and safety filtering. Depending on your application, removing or flagging harmful content. This is a genuine requirement rather than an optional extra for most applications.
Personal data detection. Identifying and handling names, addresses, contact details, and identifiers found in scraped content. This is both a quality and a compliance concern.
Machine-generated content detection. An increasingly serious problem. A growing share of the web is model output, and training on it has known degradation effects. Detection is imperfect and worth attempting anyway.
Boilerplate-heavy source detection. Some sources produce a high ratio of template to content. Measuring this per source lets you weight or drop them.
Record what you filtered and why. Filter decisions are part of provenance, and being unable to explain why a document isn't in the corpus is nearly as bad as being unable to explain why one is.
Measuring Corpus Quality
Volume is easy to measure and nearly useless as a quality indicator. The metrics that actually predict whether a corpus is good.
Post-filter retention rate. What fraction of collected documents survive deduplication and quality filtering. A low rate isn't a failure — it means your filters are working — but it directly determines your real cost per usable document, and it tells you whether your source selection is efficient.
Duplication rate per source. Sources producing high near-duplicate rates are contributing far less than their page counts suggest. This is the number that tells you which sources to drop.
Boilerplate ratio. Extracted text as a proportion of raw content, per source. A source consistently yielding 10% content and 90% template needs a custom extraction rule or removal.
Source diversity. How concentrated is the corpus? If 60% of documents come from three domains, you have a much narrower dataset than the document count implies, and it will show in model behaviour.
Language distribution. Both what you intended and what you got. These frequently differ.
Length distribution. Pathologically short and pathologically long documents both cause problems, and the shape of the distribution is informative about extraction quality.
Temporal distribution. When was the content created? A corpus skewed heavily to one period has a corresponding blind spot.
Token count, not document count. The unit that matters downstream. Document counts obscure enormous variation in actual content volume.
Manual sampling. Regularly read a random handful of documents from the corpus. Automated metrics catch what you thought to measure; reading catches what you didn't. This is the least scalable and most consistently valuable quality practice available.
Incremental and Ongoing Collection
Corpora go stale, and full recrawls at this scale are prohibitively wasteful.
Use freshness signals. Sitemap last-modified timestamps, HTTP Last-Modified and ETag headers, and conditional requests. A conditional request that returns 304 costs almost nothing and tells you the content hasn't changed.
Track per-URL change frequency. Some pages change daily, most never change again after publication. Learning the change rate per URL and scheduling accordingly reduces recollection volume enormously.
Prioritise by value, not by age. A stale document from a high-value source matters more than a fresh one from a marginal one.
Detect content changes, not just page changes. A page whose only difference is a rotating advertisement hasn't meaningfully changed. Content hashing after boilerplate removal catches this.
Separate discovery from recollection. Finding new URLs and refreshing known ones are different workloads with different schedules.
Budget your recollection explicitly. Decide what proportion of your ongoing bandwidth goes to new content versus refreshing existing content, rather than letting it happen by accident.
Handle disappearance. Content gets removed. Decide whether your corpus reflects the web as it was at collection time or as it is now, and make that a documented choice rather than an emergent property.
Provenance and Documentation
The discipline that separates a defensible corpus from a liability.
Record per document: source URL, collection timestamp, HTTP status, content type, detected language, the licence or terms under which it was obtained, the parser or extraction version applied, and which filters it passed.
Record per source: the domain, its terms of service position, its robots directives at the time of collection, the licence if any, contact details, and the date you assessed it.
Keep the raw response. Everything above can be recomputed from raw content plus request metadata. Nothing can be recovered without it, and re-collecting the past is impossible.
Version your pipeline. Which extraction, filtering, and deduplication logic produced any given corpus snapshot.
Maintain a dataset card. Composition, sources, collection methodology, filtering applied, known limitations, and licensing position. This is increasingly expected practice and it's far easier to write as you go than to reconstruct.
Support removal requests. You may need to remove specific content, whether for legal reasons or at a source's request. If your corpus can't be filtered by source or by URL after the fact, that becomes an expensive problem.
A Realistic Build Sequence
The order that avoids the most expensive rework on a corpus project.
Phase one — problem definition. Establish whether you need pretraining data, fine-tuning examples, or a retrieval corpus. This determines everything downstream and getting it wrong wastes months. Be honest about whether an existing open model plus RAG would solve the actual problem.
Phase two — source inventory and licensing assessment. List candidate sources, assess licensing and terms for each, and rank by quality-per-byte. Do this before writing collection code, because it will change what you build.
Phase three — exhaust existing datasets. Check what published corpora already cover your needs. Building from scratch what someone has already released and filtered is a straightforward waste.
Phase four — per-source classification testing. Five hundred requests through datacenter addresses against each significant source, recording status distributions. This determines your bandwidth economics and takes an afternoon.
Phase five — extraction quality on a small sample. Collect a few thousand documents, extract text, and read a random sample by hand. Fix extraction before scaling, because extraction quality determines corpus quality and it's much cheaper to fix at this stage.
Phase six — provenance and storage design. Raw storage, metadata schema, and the ability to filter and remove by source or URL. Build this before volume collection, not after.
Phase seven — incremental deduplication. Running rather than batch, across the whole corpus. Retrofitting this at terabyte scale is genuinely difficult.
Phase eight — volume collection with monitoring. Retention rates, duplication rates per source, extraction ratios, and throughput utilisation. Now you can scale, and you can see what's happening while you do.
Phase nine — filtering and quality measurement. With enough volume to measure meaningfully, tune the quality filters and drop the sources that aren't earning their bandwidth.
Phases five and six are the ones people skip under time pressure, and they're the two that determine whether the resulting corpus is usable or merely large.
Legal and Licensing Considerations
The area where training data collection differs most sharply from other scraping, and where the ground is genuinely unsettled.
- Copyright applies to web content by default. Publicly accessible is not the same as public domain, and the legal position on training use is actively contested in multiple jurisdictions with litigation ongoing.
- Licensing terms vary and matter. Creative Commons variants differ substantially in what they permit, particularly around commercial use and derivatives.
- Terms of service are contracts. Many explicitly prohibit automated collection or use for model training. Breaching them carries commercial and reputational risk even where no criminal statute applies.
- Data protection law applies to personal data in your corpus. GDPR and comparable regimes cover personal information regardless of whether it was publicly accessible, and include rights that are difficult to honour in a training set.
- Respect published crawl directives, including any that specifically address AI training use. Several standards and conventions have emerged here and more are appearing.
- Jurisdictions differ significantly. Text and data mining exceptions exist in some places, with conditions, and don't exist in others.
- The situation is changing. Regulation, litigation, and industry norms in this area are all moving, and a position that was defensible two years ago may not be now.
Not legal advice, and this is one of the areas where that disclaimer carries the most weight. If you're building a corpus for anything commercial, involve a lawyer early rather than after collection.
Common Mistakes
Building a pretraining pipeline for a fine-tuning problem. The most expensive error available. Establish which problem you're solving before building anything — most projects need thousands of curated examples, not billions of scraped tokens.
Routing everything through premium residential proxies. Most high-quality sources — government publications, open repositories, documentation, academic content — have no bot detection. Testing per source and routing the majority through unmetered datacenter bandwidth is frequently the largest cost saving available.
Estimating volume instead of measuring it. Average document size is routinely underestimated by a factor of two or three, and the error propagates straight into a wrong infrastructure decision.
Deduplicating at the end. A months-long collection produces a final deduplication job that may not be tractable. Deduplicate incrementally, across the whole corpus.
Under-investing in extraction. A corpus full of navigation menus and cookie notices is actively harmful to train on, and extraction quality is where that's determined.
Not recording provenance from document one. Reconstructing it later ranges from painful to impossible, and it's the thing you'll be asked about.
Discarding raw responses. Every filtering and extraction decision becomes permanent and unrevisable.
Ignoring licensing until after collection. Retrofitting compliance to a built corpus is one of the worst positions to be in, and sometimes there's no fix short of rebuilding.
Optimising for volume. Document count is a vanity metric. Post-filter token count from diverse sources is the number that matters.
No removal capability. If you can't filter your corpus by source or URL after the fact, a removal request becomes a crisis.
Ignoring machine-generated content. A growing share of the web is model output, and training on it has documented degradation effects.
Skipping manual review. Reading a random sample of your own corpus regularly catches problems no metric will surface.
Frequently Asked Questions
How much data do I actually need?
Depends entirely on whether you're pretraining or fine-tuning. Fine-tuning can produce excellent results with thousands of high-quality examples. Pretraining from scratch is a fundamentally different order of magnitude, and for most organisations fine-tuning an existing model is the right answer.
Should I build a corpus or use an existing one?
Start with existing published datasets. They're free, well understood, and already filtered. Build only to supplement with domain-specific content those datasets lack.
Datacenter or residential proxies?
Test per source. A large share of high-quality sources — documentation, government publications, open repositories, academic content — have no meaningful bot detection, and unmetered datacenter bandwidth is dramatically cheaper at corpus scale.
How much does deduplication remove?
Varies enormously by source mix, and it's routinely a large fraction. Measure it, because it directly determines your real cost per usable document.
Is training on scraped web data legal?
Contested and jurisdiction-dependent, with active litigation. Get legal advice specific to your situation and your jurisdiction rather than relying on what everyone else appears to be doing.
How do I handle personal data in scraped content?
Detect it, decide on a policy — removal, redaction, or retention with a lawful basis — document the decision, and maintain the ability to remove content on request.
What about training on AI-generated content?
A real and growing problem with documented degradation effects. Detection is imperfect. Prefer sources with a clear human provenance and content predating widespread generation where the distinction matters to you. It's also worth measuring: a source whose content has shifted sharply in style over the past two years may be worth investigating before you keep collecting from it.
How do I know when I have enough data?
For fine-tuning, when adding more curated examples stops improving evaluation performance — which happens sooner than most people expect. For retrieval corpora, when coverage of the queries you care about is adequate. Measure against an evaluation set rather than against a volume target.
Should I collect continuously or in campaigns?
Continuously for anything where freshness matters, in campaigns for static domains. Continuous collection also makes better use of throughput-priced bandwidth, since steady utilisation is what those plans reward.
How do I keep the corpus current?
Incremental collection using last-modified signals and change detection, rather than periodic full recrawls. Full recrawls at corpus scale are enormously wasteful.
Getting the Collection Infrastructure Right
At corpus scale the bandwidth model is the decision that dominates everything else. ProxyScrape's unlimited bandwidth datacenter plans remove the meter entirely for the large share of high-quality sources that don't check IP classification, which is where most corpus volume should be coming from anyway. For sources that do require residential addresses, their unlimited residential plans price by sustained throughput rather than per gigabyte, which is the model that fits continuous multi-terabyte ingestion, while metered residential suits smaller targeted collections. Their AI and machine learning data documentation covers the setup side, and their ethical sourcing policy is worth having on file given how much scrutiny training data provenance now attracts.
→ Compare bandwidth models against your projected corpus volume
Training data collection rewards a discipline that ordinary scraping doesn't require: knowing exactly what you have, where it came from, and why it's in the corpus. The teams that build good datasets aren't the ones that collected the most — they're the ones that documented provenance from the first document, deduplicated continuously rather than at the end, filtered aggressively enough to hurt, and tested per-source classification requirements instead of routing a terabyte of open-licensed government publications through premium residential addresses.