Google joins OpenAI and Anthropic in promising new scraping controls

Google, Microsoft, OpenAI, and Anthropic have pledged to support a unified AI training opt-out signal within robots.txt. Based on an IETF draft, this shared rule allows website owners to prevent their content from being used in generative AI training with a single command. Previously, site administrators had to create separate rules for each company's individual crawler. This new common signal streamlines scraping controls, ensuring that a single instruction is honored by all participating AI companies.