LesenForschungRadarAnlageframework
Anmelden / Registrieren
← Zurück zur Lektüre
NewsThema · AI训练爬虫退出标准Wesentlich2 Min. LesezeitDisconfirmAI Redaktion

Google, Microsoft, Anthropic and OpenAI Back Unified AI Crawler Opt-Out Standard

Four major tech firms plan to support a single robots.txt directive to block generative AI training scrapers, with a draft submitted to the IETF.

2026-10-31 10:00 GMT+8

U.S. technology companies Google (Alphabet Inc.), Microsoft Corp., Anthropic and OpenAI announced on October 30, 2026, that they plan to support an upcoming website crawling standard. The initiative aims to simplify the process for website owners to prevent artificial intelligence developers from using their content to train models.

Currently, websites use a text file called "robots.txt" to manage search engine crawler behavior. To block specific AI companies from collecting training data, site administrators must add separate exclusion entries for each AI provider and every distinct scraper tool they use. The new standard will allow websites to block all scrapers collecting data for generative AI models with a single robots.txt instruction. This setting would not apply to models that perform exclusively non-generative tasks.

The four companies plan to incorporate this standard into the Robots Exclusion Protocol, which is maintained by the Internet Engineering Task Force (IETF), and have already submitted a draft proposal to the organization. Additionally, Microsoft stated in a blog post that it will offer a setting called "AI Grounding," allowing sites to prevent AI models from accessing pages to answer user questions (retrieval-augmented generation scenarios), not just for training data collection. OpenAI also plans to offer similar controls.

According to reports, OpenAI and Microsoft intend to support the new crawling standard in the coming months, while Google will follow suit sometime next year. Meta Platforms Inc. expressed support for the idea behind the technology but has not yet committed to implementing it.

Anmelden / Registrieren
LesenForschungRadarAnlageframework

Quellen

  • siliconangle.comSiliconANGLE↗
Thema

AI训练爬虫退出标准

2026年10月30日,OpenAI、Google、Microsoft和Anthropic宣布支持一项新的AI训练退出标准。该标准由Anthropic工程师Kevin Kelley起草并提交至互联网工程任务组(IETF),旨在通过统一的“政…

3 Schritte1 BeiträgeAktualisiert 2026-11-01 18:59

Wie es dazu kam

  1. 2026-09-15Cloudflare推出“禁止AI训练”设置,允许网站在保持搜索可见性的同时阻止混合用途爬虫进行训练,Apple、Google和Microsoft承诺遵守。blog.cloudflare.com ↗
  2. 2026-10-30OpenAI、Google、Microsoft和Anthropic联合宣布支持基于IETF草案的统一AI爬虫退出标准,采用robots.txt中的共享政策令牌。theverge.com ↗
  3. 2026-10-30SiliconANGLE报道指出,该标准由Anthropic提交至IETF,旨在简化多爬虫屏蔽流程,Meta表态支持但未承诺实施。siliconangle.com ↗

Was noch ungeklärt ist

  • Meta是否会正式承诺遵守该标准并公布实施时间表?
  • IETF最终是否会将此草案确立为强制性互联网标准,还是仅作为行业最佳实践?
  • 中小网站如何低成本地验证AI爬虫是否真正遵守了robots.txt中的新令牌?
Vollständiges Thema lesen →
Recherche fortsetzen →