Web crawling for AI: robots.txt under pressure at publishers

The Robots Exclusion Protocol, laid down in robots.txt, formed the tacit agreement between webmasters and search engines for more than thirty years: search robots dutifully read the rules, indexed the permitted pages, and in return delivered valuable visitor traffic. The rise of large-scale language models and autonomous data collectors has completely disrupted this fragile balance. Publishers see their content disappear en masse into training corpora and real-time retrieval systems, without proportional traffic or compensation in return.

In this article we analyze why a small text file from 1994 is no longer a match for the aggressive scraping practices of model builders. Those wanting to understand the broader context of editorial resistance can consult the background analysis on the battle for Dutch content between publishers and AI scrapers . We look at the technical frictions, the legal tension between voluntary protocols and binding law, and the rise of active defense mechanisms.

The Historical Context and the Vulnerability of a Gentlemen's Agreement

When Martijn Koster designed the robots.txt mechanism in 1994, the internet was an academic, small-scale environment. The protocol was intended to prevent server overload caused by early web crawlers accidentally continuing to query recursive folder structures indefinitely. It was explicitly not a security mechanism, but a guideline for polite behavior. A crawler can Disallow: / technically simply ignore it, since the file merely contains instructions that are voluntarily interpreted by the client.

For three decades, this gentlemen's agreement worked because the interests of both sides ran parallel. Search engines wanted to show relevant results; publishers wanted to be findable by readers. This reciprocity disappeared the moment AI developers began treating web data as pure raw material for matrix calculations. When an AI agent scrapes a web page to formulate a generative answer, the user stays within the AI provider's interface, and the original author sees no visitor in return.

In addition, the nature of web crawlers has become fundamentally differentiated. Where traditional search engines ran a single recognizable crawler for search indexing, today's AI companies often use multiple bots with varying User-Agents. There are separate bots for offline model training, bots for live search integrations (such as SearchGPT or Copilot), and generic bots operating under vague aliases. Publishers who want to protect their intellectual property against model training while remaining findable in conventional search results get stuck in an opaque jungle of bot identities.

The Decoupling of Search Indexing and Model Training

The greatest friction for web publishers arises from the forced dilemma between visibility and exploitation. AI providers that also operate search engines do not always strictly separate their crawlers. A publisher that blocks a generic AI crawler risks, in some architectures, also lower visibility in regular search overviews or news aggregators. Although major players now declare specific User-Agents for training, the overview has become extremely complex for administrators.

As the table illustrates, an airtight robots.txt covers only part of the market. The official crawlers of well-known labs generally comply with the explicit disallow rules to discourage legal claims. The shadow market of commercial data aggregators, who scrape data and then resell it packaged to AI developers, ignores the file categorically, however. These bots disguise their User-Agent as a regular browser and rotate continuously between residential IP addresses.

Legal Frameworks: From Non-Binding Code to Binding Law

Ignoring or, conversely, enforcing crawl instructions relates directly to legislation. Within the European Union, the legal basis for automated data mining is laid down in the Directive on Copyright in the Digital Single Market (DSM Directive, Directive 2019/790). Article 4 of this directive governs text and data mining (TDM) for commercial purposes. It states that scraping is permitted unless the rightsholder has explicitly reserved this in an 'appropriate manner,' such as by machine-readable means.

In practice, European regulators consider a declaration in robots.txt a valid machine-readable opt-out for TDM. Those wanting to dig deeper into the legal foundations will find in the dossier on the frameworks and pitfalls surrounding AI and copyright an extensive analysis of these exception provisions. The question, however, is what happens if a party ignores the TDM opt-out. Because robots.txt itself is not a legally binding encryption mechanism, a publisher ends up in a lengthy civil procedure to prove copyright infringement.

In addition, the European AI Regulation imposes additional transparency requirements on providers of general-purpose AI models. They must publish a detailed summary of the data used to train their models. To verify exactly how these oversight obligations land with companies, the guide on the EU AI Act and its consequences for creators and users can be consulted. Publishers use these transparency rules to try to determine whether their content was illegally harvested after they had put a formal crawl ban in place.

Practical Example: Configuring a Defensive robots.txt

To give an idea of how publishers set up their server configuration to keep the most active AI bots out, let's look at a typical implementation. An administrator must explicitly name every bot, since a general User-agent: * block would also exclude regular search engines.

# Blokkeer specifieke AI-trainingsbots
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# Sta traditionele zoekmachines wel toe voor indexering
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Specifieke machineleesbare TDM-reservering conform EU DSM Art. 4
# (Toegevoegd via HTTP-headers of meta-tags op paginaniveau)

The weakness of this method is immediately apparent: administrators must continuously track which new crawlers emerge. As soon as a new AI startup launches a crawler under a different name, the configuration lags behind the facts. This leads to a permanent cat-and-mouse game between server administrators and data collectors.

Active Technical Countermeasures: WAFs and Rate Limiting

Because passive text files prove inadequate, medium and large media companies are shifting to active network security. Here, traffic is analyzed via Web Application Firewalls (WAFs), such as those from Cloudflare, Fastly, or AWS CloudFront. These systems don't primarily look at the declared User-Agent, but analyze visitor behavior at the network level.

Scrapers often give themselves away through deviant patterns: they request pages at an inhumanly high frequency, follow hyperlinks linearly through the page source rather than interactively through the DOM, and show anomalies in their TLS fingerprint (JA3/JA4). When the browser handshake doesn't match the software the client claims to be, the WAF can immediately block the request or serve a cryptographic JavaScript challenge.

However, this active defense comes with drawbacks. Running advanced bot detection increases latency for regular visitors and entails significant operational costs. Moreover, legitimate niche search engines, feed readers, and archiving services can be accidentally blocked (false positives), further fragmenting the openness of the web.

The Privacy Dimension and Scraping of Personal Data

Besides intellectual property law, data protection law also plays a crucial role. Publishers' web pages contain thousands of names, quotes, and biographical details of individuals. When crawlers harvest this data without a legal basis for model training, friction arises with the General Data Protection Regulation (GDPR).

Model builders often rely on 'legitimate interest' (Article 6(1)(f) GDPR) to justify collecting training data. European privacy regulators, however, question whether unlimited scraping of public data is even proportionate, and whether data subjects can effectively exercise their rights of access and erasure (Article 17 GDPR) against a trained neural network. For organizations that train or deploy models themselves, the guide on GDPR compliance and privacy choices in AI models offers a detailed decision framework for data processing.

Commercial Licensing Deals as an Alternative Way Out

Because technical blockades are not watertight and legal proceedings take years, well-capitalized media groups increasingly opt for a pragmatic route: exclusive data licensing deals. International players such as News Corp, Axel Springer, and Reddit, but also Dutch media groups, are signing multi-year contracts with AI developers worth tens to hundreds of millions of euros.

In exchange for these fees, model builders receive structured access via official APIs or direct data dumps. This resolves the litigation risk for AI companies and guarantees clean, well-labeled data with low latency. For publishers, it partly offsets the loss of advertising revenue. This development, however, creates a sharp divide in the media landscape:

Small publishers and individual creators are left out in the cold by these bilateral agreements. They lack the volume to get a seat at the table with major AI labs, as well as the budgets to set up enterprise firewalls. For them, the dysfunctional robots.txt often remains the only, toothless line of defense.

Toward New Standards and Protocols

The realization that the Robots Exclusion Protocol is failing in the AI era has led to initiatives for more modern standards. Various working groups within internet standards organizations are working on protocols that determine not only or whether a page may be read, but also under what conditions.

One example of this is protocolizing content licenses via HTTP response headers or structured metadata in the HTML head (such as tdm-reservation). Experiments are also underway with cryptographic signatures and paid micropayments via the web: a bot may scrape an article provided that a fraction of a cent is immediately settled via an automated protocol. However, the adoption rate of these kinds of standards is still in its infancy; as long as large tech companies can scrape for free via anonymous proxies, their incentive to voluntarily pay via open standards remains minimal.

Conclusion: The Restructuring of the Open Web

The crisis surrounding robots.txt marks the end of an era in which the public web was open and accessible to everyone by default. What was once intended as a simple traffic regulator for server capacity has become the front line in an economic battle over intellectual property and training data.

Publishers are being forced to retreat behind paywalls, advanced bot filters, and closed API structures. Those who leave their content openly on the web effectively accept that it will be assimilated by commercial models. The coming years will show whether tightened European legislation and collective rights organizations can build a dam, or whether the open web will definitively fragment into closed data silos.