# Dutch language evaluation: comparing models on Dutch tasks [Skip to content](#lm-inhoud)Network/NL[EN](/en/)[Hubhub.llmnet.nlCompare models by task, language, cost, and license.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompting techniques, patterns, and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs robustly in software: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlAdopting AI in an organization, from pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, contextualized for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, tailored to your tasks.](https://benchmark.llmnet.nl/en/)[Jobsvacatures.llmnet.nlAI roles, salaries, and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain Dutch, from beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRunning AI privately on your own Mac, PC, NAS, or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research, and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for those building themselves.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/)[](https://x.com/intent/post?url=https%3A%2F%2Fnieuws.llmnet.nl%2Fnederlandstalige-evaluatie-hoe-je-modellen-op-nl-taken-vergelijkt&text=Nederlandstalige%20evaluatie%3A%20modellen%20op%20NL-taken%20vergelijken)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fnieuws.llmnet.nl%2Fnederlandstalige-evaluatie-hoe-je-modellen-op-nl-taken-vergelijkt)[](https://www.reddit.com/submit?url=https%3A%2F%2Fnieuws.llmnet.nl%2Fnederlandstalige-evaluatie-hoe-je-modellen-op-nl-taken-vergelijkt&title=Nederlandstalige%20evaluatie%3A%20modellen%20op%20NL-taken%20vergelijken)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fnieuws.llmnet.nl%2Fnederlandstalige-evaluatie-hoe-je-modellen-op-nl-taken-vergelijkt&text=Nederlandstalige%20evaluatie%3A%20modellen%20op%20NL-taken%20vergelijken)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fnieuws.llmnet.nl%2Fnederlandstalige-evaluatie-hoe-je-modellen-op-nl-taken-vergelijkt)[](https://www.reddit.com/submit?url=https%3A%2F%2Fnieuws.llmnet.nl%2Fnederlandstalige-evaluatie-hoe-je-modellen-op-nl-taken-vergelijkt&title=Nederlandstalige%20evaluatie%3A%20modellen%20op%20NL-taken%20vergelijken)[](#) # Dutch-language evaluation: how to compare models on Dutch tasks By Ivo Donker — compiled with AI assistance (Claude & Gemini) In the previous overview article on [linguistic diversity in multilingual models](https://nieuws.llmnet.nl/en/taaldiversiteit-in-llms-nederlands) we analyzed the theoretical challenges of smaller language areas within large language models. While we emphasized tokenization efficiency and cultural bias at the time, the market is now shifting toward concrete performance measurements. Anyone looking to select a model for a Dutch-language application can no longer rely on general English benchmarks, but requires specific test sets instead. Read the earlier theoretical foundation via the [pillar core article on linguistic diversity in Dutch](https://nieuws.llmnet.nl/en/taaldiversiteit-in-llms-nederlands) to understand how architectural choices fundamentally influence performance in Dutch. In this article, we make that theoretical Dutch perspective explicitly measurable. We cover the specific testing frameworks, datasets, and methodologies available today to evaluate both encoder models and large generative language models on Dutch tasks. To understand how the general data landscape for Dutch is structured, we recommend consulting the overview on [open data initiatives for AI](https://nieuws.llmnet.nl/en/open-data-initiatieven-voor-ai) . This article tracks developments up to the reference date; this information was verified on 2026-08-07 and remains representative as long as the underlying tests and repositories are maintained by the respective research institutions. ### What has changed since last time - Shift from theory to measurement: While the previous post focused on the theoretical lag of multilingual tokenizers, we are now focusing entirely on standardized evaluation sets. - Insight into availability: The dynamics in the landscape have led to the disappearance of some platforms (such as the DUMB leaderboard), highlighting the need for local evaluation runs. - Key role of standardized harnesses: The integration of Dutch datasets into frameworks such as the LM Evaluation Harness allows developers to run directly reproducible tests. - Focus on small models: The rise of specialized small models requires specific retrieval and comprehension measurements; see the overview of [small language models and their practical usability](https://nieuws.llmnet.nl/en/kleine-taalmodellen). ## The evolution of Dutch-language benchmarks The evaluation of Dutch natural language processing (NLP) has undergone a major transformation. Initially, researchers relied on separate academic datasets for specific tasks such as Named Entity Recognition (NER) or sentiment analysis. With the rise of BERT-based models, a need arose for an integrated benchmark. This led to the development of DUMB (Dutch Model Benchmark), presented in the paper by de Vries, Wieling, and Nissim at the EMNLP 2023 conference. DUMB brought together nine Dutch tasks, four of which had not previously been available in a standardized form for Dutch. DUMB was primarily designed to compare encoder models (such as BERTje and RobBERT) with one another. A key premise in the DUMB methodology was the use of Relative Error Reduction (RER) compared to a strong baseline, rather than looking purely at accuracy. At the time, the study by de Vries et al. (2023) showed that specific Dutch monolingual models surprisingly underperformed on various tasks compared to broader multilingual or larger models. Within that framework, the highest scores were achieved by DeBERTaV3-large, XLM-R-large, and mDeBERTaV3-base. However, the state of this infrastructure also illustrates the vulnerability of academic projects. As stated in the CLARIN K-Centre documentation (verified on 2026-08-07), the original online leaderboard-hosted environment (dumbench.nl) is no longer operational. Researchers wishing to evaluate encoder models based on the DUMB pillars must now run the evaluation scripts and datasets independently via the official sources in the [DUMB EMNLP 2023 paper](https://aclanthology.org/2023.emnlp-main.447) and the accompanying repository at [arXiv:2305.13026](https://arxiv.org/abs/2305.13026). ## Evaluating generative models with EuroEval and Eurolingua For modern generative language models (such as large instruction and chat models), traditional encoder benchmarks often fall short. Generative tasks require zero-shot and few-shot evaluation methods that center around output generation. In the European landscape, two initiatives have become leading for standardized comparisons. The first key initiative is EuroEval. This is a benchmarking framework established for more than 30 European languages, featuring specific sections for Dutch. Within this framework, the leaderboards "EuroEval Monolingual Dutch" and "EuroEval multilingual Germanic" serve as the successors to earlier initiatives such as [scandeval.com/dutch-nlg](https://scandeval.com/dutch-nlg). EuroEval focuses on measuring reading comprehension, knowledge, and generative capabilities within a strictly controlled test environment. Consult the current overview directly on the [EuroEval Monolingual Dutch leaderboard](https://euroeval.com/leaderboards/Monolingual/dutch/) (verified on 2026-08-07). The second overarching framework is the European LLM Leaderboard, maintained by the Eurolingua initiative. This project provides multilingual evaluation results for 21 European languages, including Dutch. It uses a customized fork of the well-known LM-evaluation-harness (EleutherAI). Through this standardized harness integration, models can be evaluated in a transparent and reproducible manner. The results and methodological documentation can be viewed via the [European LLM Leaderboard on Hugging Face](https://huggingface.co/spaces/Eurolingua/european-llm-leaderboard) (verified on 2026-08-07). ## Testing information retrieval with BEIR-NL When models are deployed in a Retrieval-Augmented Generation (RAG) architecture, generation quality is only half the equation. The quality of the embeddings and search performance on Dutch documents is at least as critical. For this purpose, BEIR-NL was developed: a zero-shot information retrieval benchmark specifically for the Dutch language area. The BEIR-NL benchmark was officially presented in a paper during the BUCC workshop (ACL Anthology, 2025). It fills an important gap: prior to the introduction of BEIR-NL, there were virtually no publicly accessible, diverse test sets to measure how effectively Dutch retriever models extract relevant information from large text corpora without specific fine-tuning. For developers and researchers building their own search systems or RAG pipelines, incorporating the dataset and methodology of BEIR-NL into the test protocol is highly recommended. Details regarding the setup can be found in the [BEIR-NL paper in the ACL Anthology](https://aclanthology.org/2025.bucc-1.5/). The datasets themselves can be downloaded and processed directly via the [CLIPS BEIR-NL dataset collection on Hugging Face](https://huggingface.co/collections/clips/beir-nl). To learn how to integrate such a retrieval test into your own quality assurance workflow, please refer to the [step-by-step guide for setting up your own benchmark on benchmark.llmnet.nl](https://benchmark.llmnet.nl/en/eigen-benchmark-opzetten-stappenplan). ## Reading Comprehension and Instruction Following: SQuAD-NL v2.0 Alongside retrieval and classification, reading comprehension remains a crucial metric for generative models. One of the most widely used datasets for this purpose is SQuAD-NL v2.0, a Dutch-translated and adapted version of the well-known Stanford Question Answering Dataset. This dataset is hosted by the GroNLP research group at the University of Groningen. SQuAD-NL v2.0 includes questions where the answer can be found directly within a provided text passage, as well as questions that are unanswerable based on the given context. The latter component is essential for testing whether a model is capable of preventing hallucinations when information is missing. The dataset is fully integrated into the LM Evaluation Harness, making it easy to benchmark models in an automated fashion. In an in-depth analysis on evaluating Dutch LLMs (published on 05-01-2024), researcher Pieter Delobelle demonstrated 0-shot and few-shot results on SQuAD-NL v2.0. That analysis compared the performance of, among others, general 7-billion-parameter instruction models and the dedicated Dutch model GEITje-chat. For an extensive look at the results and the prompts used, please refer to [Pieter Delobelle's blog on evaluating Dutch LLMs](https://pieter.ai/blog/2024/evaluating-dutch-llms). The raw dataset is accessible via the [GroNLP SQuAD-NL v2.0 repository on Hugging Face](https://huggingface.co/datasets/GroNLP/squad-nl-v2.0). To dive deeper into the evaluation of generated answers, you can read the guide on [evaluating Dutch-language model output on benchmark.llmnet.nl](https://benchmark.llmnet.nl/en/benchmarks-voor-nederlandstalige-modellen) . ## Overview of Dutch Model Types and Open-Source Resources The range of Dutch-language models is diverse, spanning from small curated encoders to adapted generative models. According to the inventory compiled by the CLARIN K-Centre for Language Modelling (verified on 2026-08-07), the landscape can be divided into several categories. The overview on the [CLARIN K-Centre Language Modelling wiki](https://kdutch.ivdnt.org/wiki/Language_modeling) provides a clear picture of the historical and current status of these models. To make comparisons transparent, models should be categorized based on their architecture and objective. The table below presents the key representatives based on verified data from the CLARIN K-Centre: Category | Model Name / Initiative | Type / Base | Recommended Evaluation Method | Encoders (Monolingual) | BERTje, RobBERT, RobBERTje | BERT / RoBERTa (Encoder) | DUMB framework, task-specific fine-tuning | Encoders (Domain-Specific) | MedRoBERTa.nl, belabBERT | Medical / Belgetal (Encoder) | Domain-specific classification & NER test sets | Generative (Adaptations) | ChocoLlama, Fietje 2 (Phi-2), Tweety-7b-dutch | Llama-2/3, Phi-2 (Decoder) | EuroEval, LM Evaluation Harness, SQuAD-NL | Generative (Instruction) | Reynaerde 7B Chat, Schaapje-2B, GEITje 7B | Mistral / Llama derivatives | EuroEval, SQuAD-NL, manual qualitative review | An important caveat regarding the availability of open models is repository continuity. For instance, the CLARIN overview explicitly states that some popular models, such as GEITje 7B, could no longer be actively downloaded from their original locations at the time of verification (2026-08-07). This underscores the importance of storing model weights locally and meticulously documenting the exact model version for every evaluation. ## Methodological guidelines for a fair comparison Comparing models on Dutch-language tasks requires a rigorous methodological approach. Simply adopting a score from an arbitrary leaderboard often leads to incorrect conclusions due to differences in testing conditions. Anyone running evaluations or analyzing evaluation reports should take four critical factors into account. First, the exact prompt phrasing and the number of examples (few-shot setup) have a major impact. A model that scores poorly in a 0-shot setting may perform significantly better in a 3-shot setup with clear Dutch instructions. Second, the chosen metric must align with the objective. The Relative Error Reduction (RER) used in DUMB provides a very different picture of progress compared to an absolute F1 score or accuracy percentage. Third, the evaluation date and version number play a decisive role. Models and underlying evaluation harnesses are continuously updated. When reporting test results, the exact date of the test run, the software commit used for the harness, and the exact parameter settings (such as temperature and top_p) must always be explicitly stated. ## Conclusion The evaluation of Dutch-language models has matured. While a few years ago one had to rely on ad-hoc translations of English benchmarks, a comprehensive toolkit is now available. With DUMB for encoder models, EuroEval and the European LLM Leaderboard for generative models, BEIR-NL for information retrieval, and SQuAD-NL v2.0 for reading comprehension, it is possible to obtain a representative picture of performance within the Dutch language area. When setting up a comparison, transparency and reproducibility are paramount. Do not rely on outdated or unverified rankings, but perform tests using standardized harnesses on controlled datasets. This ensures the right model can be chosen with confidence for the intended specific use case. © 2026 llmnet.nl — Knowledge network for AI and LLMs in the Dutch language area.