OpenAI Launches Watermarks for Text Content in Compliance with EU Law

OpenAI has implemented a watermarking feature for text content to comply with the AI Act requirements in the European Union. In the coming weeks, the company plans to apply a digital watermark to the text output of ChatGPT and Codex models intended for use in the EU. However, unlike the similar technology by Anthropic, OpenAI does not plan to apply this technique to content generated outside of Europe, although API clients can choose to have the marking applied to any generated information.

OpenAI's watermarking technology does not always succeed in recognizing AI-generated text and can be easily fooled. According to the AI Act requirements, generative AI providers must make model outputs machine-readable. Anthropic was the first to announce its approach to complying with the AI Act by applying Google's SynthID-Text algorithm globally. Microsoft and Meta are also developing similar technologies for images, as concerns about content origin and the use of generative AI for misinformation are relevant not only in Europe. OpenAI uses a watermarking method called textGrain, which "adds an invisible statistical signal to the model's word choice." This detector analyzes the signal to determine if the text contains OpenAI's watermark. Models predict words one by one based on the probabilistic distribution of possible words. Periodically choosing another word, ideally– a synonym, deviates the text from the statistical pattern formed by the unbiased model.

Theoretically, the detector should mark such a change in word usage. The expected error is one percent, but in short texts of 200 words, the detector can only capture 80 percent of watermarks, compared to 95 percent in texts of 400 words. Results are also worse for functional texts, such as mathematical or code fragments, as word substitutions are more likely to introduce errors. The textGrain document does not address the situation where statistically altered word choice might change the meaning of the content. This remains possible if the algorithm alters any significant fact or figure. Anthropic's approach, which also replaces certain words in the output, albeit using a different algorithm, carries similar risks. In any case, OpenAI's watermark can be easily neutralized by replacing words. In tests on 400-token passages, replacing 10 percent of words with synonyms reduced watermark detection from 92 percent to 66 percent. Replacing 25 percent of words resulted in only 17 percent of the watermark being detected. Thus, the results indicate that the changes made to comply with regulations may not be as reliable as planned.