Google, Microsoft, Anthropic and OpenAI Back Unified AI Crawler Opt-Out Standard
Four major tech firms plan to support a single robots.txt directive to block generative AI training scrapers, with a draft submitted to the IETF.
U.S. technology companies Google (Alphabet Inc.), Microsoft Corp., Anthropic and OpenAI announced on October 30, 2026, that they plan to support an upcoming website crawling standard. The initiative aims to simplify the process for website owners to prevent artificial intelligence developers from using their content to train models.
Currently, websites use a text file called "robots.txt" to manage search engine crawler behavior. To block specific AI companies from collecting training data, site administrators must add separate exclusion entries for each AI provider and every distinct scraper tool they use. The new standard will allow websites to block all scrapers collecting data for generative AI models with a single robots.txt instruction. This setting would not apply to models that perform exclusively non-generative tasks.
The four companies plan to incorporate this standard into the Robots Exclusion Protocol, which is maintained by the Internet Engineering Task Force (IETF), and have already submitted a draft proposal to the organization. Additionally, Microsoft stated in a blog post that it will offer a setting called "AI Grounding," allowing sites to prevent AI models from accessing pages to answer user questions (retrieval-augmented generation scenarios), not just for training data collection. OpenAI also plans to offer similar controls.
According to reports, OpenAI and Microsoft intend to support the new crawling standard in the coming months, while Google will follow suit sometime next year. Meta Platforms Inc. expressed support for the idea behind the technology but has not yet committed to implementing it.