Google, Anthropic and others to support new AI crawling opt-out standard

OpenAI, Google, Microsoft, and Anthropic have agreed to support a unified machine-readable signal within robots.txt. This common token allows website publishers to restrict the use of their content for AI training across all four companies with a single line of text. By utilizing the existing standard file used for automated crawlers, site operators can effectively block these artificial intelligence developers from accessing their data for model training once the rollout is completed by each company.