Skip to main content

Kaer NewsSoftware

Four AI firms back one robots.txt opt-out for scraping

OpenAI, Google, Microsoft and Anthropic pledged on 31 October 2026 to honour a single robots.txt signal letting publishers block AI training scraping, with Anthropic already live and Microsoft starting 1 February 2027.

3 min read
Four AI firms back one robots.txt opt-out for scrapingPhoto: The Verge

Key points

OpenAI, Google, Microsoft and Anthropic pledged to honour a single robots.txt signal letting publishers block AI training scraping, with Anthropic already live and Microsoft starting 1 February 2027.

OpenAI, Google, Microsoft and Anthropic pledged on 31 October 2026 to give publishers a single way to block their content from being scraped to train artificial intelligence models, according to a report dated that day. Today a site wanting to opt out must write separate rules for each company's crawler. The new shared signal would let one instruction in a website's robots.txt file be recognised by every adopting AI company.

The pledge matters because publishers face a surge in bots scraping their articles to feed AI models, with some, like The New York Times, taking the matter to court. In December 2023 the Times sued OpenAI and Microsoft for allegedly using millions of its articles without authorisation or payment to train ChatGPT, and the two companies invoke the fair use exception to copyright law. A single opt-out standard would replace per-company rules.

What the four companies pledged

The shared signal, called a policy token, was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF), the open standards organisation for the internet. It lives inside a website's robots.txt file, which has long signalled which web crawlers may or may not scrape a site for search indexing, archiving or AI training.

Publishers set up a new user agent in robots.txt specifically for AI training: User-Agent: __usage_ai-training, Disallow: /, with an Allow line such as /press-kit/ for pages they still want scraped. Google will interpret it as a stop sign specifically for the Google-Extended user agent, which crawls sites for material used to train Gemini models. The section does not block crawling altogether or hide content from Google search.

How the single signal works

Microsoft said the setting applies only to future training of models and does not prevent bots from crawling otherwise, and that content scraping stays allowed by default until a publisher changes the setting, according to a Microsoft blog post cited on 31 October 2026. The directive cannot undo scraping already done: if companies crawled a page previously, they already hold its content. The rule only prevents future scraping of sites or pages.

Microsoft will offer granular controls letting publishers opt out of AI training or AI grounding, the process of pointing an AI model at a verified source to reduce hallucinations. Microsoft notes sites will still appear in Bing search results regardless of their preferences. Google previously implemented similar controls letting sites choose whether content appears in AI search features or is used for AI training, without affecting search visibility.

Microsoft and Google said in separate blog posts that opting out of AI training could reduce the chances a publisher's content surfaces in AI apps and experiences. Publishers are weighing that trade-off against existing defences: the Really Simple Licensing (RSL) standard lets publishers charge crawlers that visit their pages, and Cloudflare has blocked AI bots by default on new customer sites since July 2025.

What the directive does not undo

Meta communications head Andy Stone signalled support for the new setting but did not say whether Meta will honour it. "We believe creating a common signal that conveys such preferences to multiple providers at once is an important step," Stone wrote on X, adding that Meta looks forward to working with other AI companies on the standards and their implementation across the industry.

Anthropic has already started supporting the directive. Microsoft said it will start supporting it on 1 February 2027, Google said it will support it in early 2027, and OpenAI pledged to implement the new signal in the coming months. The solution will not fully take effect for months, according to the report published on 31 October 2026.

Frequently asked questions

What did OpenAI, Google, Microsoft and Anthropic agree to?

They pledged to honour a single robots.txt signal, proposed by Anthropic's Kevin Kelley under the Internet Engineering Task Force, that lets publishers block AI training scraping across all adopting companies at once instead of setting rules for each crawler.

When will the new scraping opt-out take effect?

Anthropic has already started supporting it. Microsoft begins on 1 February 2027, Google in early 2027, and OpenAI said it will implement the signal in the coming months.

Does opting out remove a site from search or undo past scraping?

No. The directive blocks only future AI training scraping; it does not prevent crawling or hide content from Google search or Bing results. It cannot undo scraping companies have already done.

More from this edition