OpenAI to Support Shared Robots.txt Signal for AI Training Preferences

OpenAI has announced its commitment to honor a common signal in robots.txt, allowing websites to express their preferences regarding the use of their content for foundational generative AI training. This initiative aims to simplify the process for publishers and website operators to communicate their preferences to multiple AI providers simultaneously.

Previously, website operators had to create individual rules for each AI provider, which was a cumbersome task. The introduction of a shared signal will reduce this workload significantly.

The implementation of this signal is expected in the coming months, with further updates on the specifics to be shared later. Importantly, this new signal will not replace existing controls provided through robots.txt. OpenAI’s current crawler controls, which give web publishers flexibility and control over their pages, will remain in effect.

This initiative is part of the ongoing efforts by the Internet Engineering Task Force’s (IETF) AI Preferences working group. OpenAI is collaborating with other participants in this group to strengthen its involvement in IETF standards discussions. The company plans to continue adapting its practices as these discussions evolve, focusing on appropriate controls and fairness.

OpenAI views this development as a practical step toward fostering an open web where information can be published and discovered more easily. The company encourages other AI providers to honor this new signal.

Source: OpenAI News