Data diversification: training on non-English sources

The dominance of English in pre-training corpora has long been a pragmatic given in foundation model development. Because the publicly accessible internet is largely documented in English, training pipelines primarily focused on sources such as Common Crawl, Wikipedia, and GitHub. This led to architectures where other languages often hitched a ride on the statistical patterns of English as a byproduct. Now that the limits of available high-quality English web data are coming into view, technical focus is inevitably shifting toward data diversification. Integrating substantial volumes of non-English sources is no longer just a matter of language support, but a necessity to enhance the overall representational power and robustness of models.

When examining the fundamentals of machine learning, one-sided data inevitably leads to structural blind spots. To understand how these patterns are built during the initial compute phase, the guide on how an AI learns during training phases explains how weights are optimized based on word prediction. In this article, we analyze what happens when pre-training corpora are deliberately enriched with non-English sources, the computational obstacles that emerge during tokenization, how cross-lingual transfer mechanisms operate, and the methodological pitfalls associated with collecting data outside the English domain.

The imbalance in pre-training data and the 'English tax'

The historical predominance of English in training sets directly impacts the internal representations of neural networks. In a typical pre-training corpus, 80 to 90 percent of the volume historically consisted of English tokens. Languages with hundreds of millions of speakers, such as Hindi, Spanish, Arabic, or Bahasa Indonesia, often accounted for merely fractions of a percent of the total token budget. For mid-sized languages such as Dutch, Swedish, or Polish, this share was even smaller.

This disparity creates what is known in the academic literature as the 'English tax'. Because the tokenizer's vocabulary is optimized for the most frequent subwords in the training corpus, non-English texts are fragmented into significantly shorter subword units or individual bytes. A concept consisting of one or two tokens in English often requires three or four tokens in Dutch, and sometimes as many as six to ten tokens in non-Latin scripts like Devanagari or Arabic. This results in higher latency, higher inference costs per semantic unit, and an effectively smaller context window for non-English tasks.

The article on the position of Dutch in linguistic diversity describes how this token inefficiency hinders model adoption in local infrastructures. Without deliberate data diversification, models remain disproportionately expensive and slow in non-English contexts, even when the underlying transformer capacity is sufficient.

Tokenizer Architecture and Vocabulary Allocation

The first technical step in data diversification involves redesigning the tokenizer, typically based on Byte-Pair Encoding (BPE) or WordPiece. The vocabulary size ($V$) represents a strict trade-off between computational budget and compression efficiency. A larger vocabulary lowers the token-to-word ratio across all languages, but scales the embedding layer and the final softmax layer linearly in memory and parameters.

When building the tokenizer vocabulary, one can choose between homogeneous frequency counts across the raw corpus or stratified allocation per language family. If a tokenizer is trained on a corpus consisting of 85 percent English, English claims virtually all available merges. As a result, non-English words are broken down into morphologically meaningless fragments.

To correct this imbalance, modern training setups employ temperature-scaled sampling for the tokenizer training process:

# Conceptuele berekening van gesamplede taalverdeling voor tokenizer training
import numpy as np

def calculate_sampling_weights(proportions, temperature=0.3):
    """
    Schaalt taalproporties om ondervertegenwoordigde talen meer gewicht te geven.
    Een lagere temperatuur vlakt de verdeling af richting uniformiteit.
    """
    probs = np.array(proportions)
    scaled_probs = probs ** (1.0 / temperature)
    return scaled_probs / np.sum(scaled_probs)

# Voorbeeld: Engels (0.80), Spaans (0.10), Nederlands (0.02), Arabisch (0.08)
raw_distribution = [0.80, 0.10, 0.02, 0.08]
token_weights = calculate_sampling_weights(raw_distribution, temperature=0.5)

Through this scaling, non-English languages claim sufficient subword slots in the lookup table, leading to a substantial reduction in inference costs for those languages without noticeably degrading English performance.

Cross-Lingual Transfer and Abstract Reasoning

A persistent misconception is that training on non-English sources is exclusively useful for speaking those specific languages. However, empirical research into representation layers reveals that cross-lingual transfer goes deeper: abstract reasoning, code comprehension, and logic directly benefit from input data diversification.

Early transformer layers primarily focus on surface-level features, morphology, and syntax. In the middle and deeper layers, activations converge into semantic representations that are largely language-independent (a conceptual interlingua). When a model is exposed to mathematical principles, causal reasoning, or formal logic exclusively in English, the internal reasoning space becomes entangled with English syntactic patterns.

By presenting reasoning patterns across structurally diverse languages—such as languages with Subject-Object-Verb (SOV) order (Japanese, Turkish), agglutinative morphology (Finnish, Hungarian), or root-based morphology (Arabic, Hebrew)—the model is forced to decouple conceptual steps from syntactic order. This results in a more robust abstract working memory within the network's residual stream.

Quality Filters and the Danger of Synthetic Translation Loops

Collecting non-English data runs into an immediate quality trade-off. Because the volume of high-quality, human-written non-English text on the open web is vastly smaller than the English volume, there is a temptation to use large amounts of translated data. This introduces serious risks to model stability.

When crawlers index the multilingual web, they increasingly encounter machine translation spam: websites automatically translated using older translation models to generate ad revenue. If these texts enter the training corpus unfiltered, the model learns unnatural sentence structures ('translationese'), along with persistent semantic errors and mismatched idioms.

In the analysis on public datasets for language models it is emphasized how crucial curated and verified sources are for preventing contamination of training corpora. Identifying synthetically translated web pages requires specialized filters that evaluate n-gram perplexity, function word distribution, and stochastic markers typical of machine translations.

Cultural alignment and avoiding Western bias

Data is not neutral; language reflects legal frameworks, societal norms, geographic realities, and cultural assumptions. A model trained 95 percent on Anglo-Saxon sources exhibits systematic bias in its default assumptions. When answering open-ended questions about law, etiquette, medical protocols, or ethical dilemmas, such a model automatically extrapolates from American-English norms unless explicitly instructed otherwise.

When asking a question about contract law, a monolingually trained model implicitly references Common Law principles, whereas civil law governs in continental Europe. In the medical domain, treatment guidelines, antibiotic stewardship, and triage protocols differ significantly between the Netherlands and the United States. By proportionally weighting national sources—such as public parliamentary records, local case law, and academic publications—during pre-training, the model internalizes the proper institutional context.

This is closely tied to the debate surrounding intellectual property and web scraping. The article on publishers and AI scrapers in the Netherlands shows that local rights holders and media companies are increasingly blocking access to automated scrapers. This creates a paradox: precisely when AI developers need high-quality, local data, traditional sources close their doors to protect their exploitation rights.

Pre-training mix optimization and curation strategies

What does a balanced pre-training corpus look like in practice? Simply merging all available non-English data leads to degradation across general benchmarks if the source data is low quality. For this reason, modern training clusters apply sophisticated sampling ratios adjusted across different training phases (curriculum learning).

During pre-training, 'upsampling' is often utilized for high-quality non-English sources. Because a language like Dutch simply has fewer unique tokens on the web than English, reliable Dutch sources are sometimes repeated for 2 to 4 epochs during the training run. As long as the total number of repetitions remains limited and the sources are of high editorial quality, this rarely leads to memorization or overfitting, while significantly improving grammatical and lexical representation.

Syntactic diversity as a driver for generalization

The computational benefits of non-English training become evident when examining syntactic structures. English is an analytic language with a relatively rigid word order (SVO) and minimal morphological inflection. Many other languages utilize entirely different linguistic mechanisms:

Compounds in Dutch and German (such as quality assessment system) force the model to internally decompose composite concepts. Agglutinative languages such as Turkish and Hungarian append grammatical functions as suffixes to a root stem, allowing a single word to carry the semantic weight of an entire English subordinate clause. Slavic languages employ an extensive case system where word order is more flexible and indicates pragmatic emphasis rather than strict grammatical relations.

When a transformer architecture learns to predict patterns across these fundamentally distinct structures, the self-attention matrices are forced to construct more flexible, long-range dependencies. Attention heads no longer specialize in trivial patterns such as "look at the word immediately to the left," but instead learn to dynamically extract syntactic functions (such as subject, direct object, or modality), regardless of their position within the sentence.

Methodological limitations and risks

Despite the clear theoretical and practical advantages, aggressive data diversification introduces specific risks that require careful monitoring:

First, scaling up non-English data too rapidly introduces the risk of capacity dilution. If a model has insufficient parameters (for example, a model with 1 to 3 billion parameters) and is trained on a hundred different languages, interference occurs between representations. Performance on complex reasoning tasks in the primary language can degrade because the model capacity becomes spread across too many competing language systems. This phenomenon is known as the curse of multilinguality.

Second, automated quality filtering outside of English is technically more complex. Many popular heuristics for web filtering—such as stopword counting, n-gram repetition frequencies, or language model perplexity scores—are calibrated for English. When applied without adaptation to morphologically rich languages, these filters inadvertently discard legitimate, complex sentences while letting repetitive translated data through.

Thirdly, resources for low-resource languages remain scarce. There is a risk that developers will resort to synthetically generated data in those languages, causing the model to learn from errors produced by earlier generative systems. Without rigorous validation by native annotators, this leads to the degradation of rarer languages.

Conclusion and strategic outlook

Data diversification has evolved from a secondary localization preference into a core pillar of advanced model development. By structurally integrating non-English sources into pre-training corpora, developers not only improve the accessibility and affordability of AI for non-English-speaking regions, but also enhance the abstract reasoning capabilities and structural flexibility of the underlying networks.

The challenge for the coming years lies in the responsible curation of these datasets. Simply increasing data ingestion volume through web crawls is no longer sufficient; the focus is shifting toward fine-grained quality classification, morphologically aware tokenizer optimization, and compliance with local copyright frameworks. Only through a balanced combination of linguistic diversity, curated sources, and carefully calibrated training ratios can a next generation of language models emerge that is truly globally deployable.