In what appears to be a rendition of the classic “you’re asking for too much” pre-execution cliche, OpenAI has proven to be – shock – not so open after all. The AI chatbot company appears to have started sending threatening emails to users who ask overly probing questions of the company’s latest models, codenamed “Strawberry.”
I get a creepy letter if I even mention the words “Argumentation Trace” in a prompt, lol13 September 2024
Some have reported (via Ars Technica) that using certain phrases or questions when speaking to o1-preview or o1-mini results in an email warning stating: “Please stop this activity and make sure you use ChatGPT in accordance with our Terms of Service and Acceptable Use Policy. Further violations of this policy may result in loss of access to GPT-4o with Reasoning.”
For example, X user thebes claims that he receives this warning when he uses the words “reasoning trace” in a prompt. Riley Goodside, prompt engineer for Scale AI, received a warning for violating the in-chat guidelines for telling the model not to tell them about his “argumentation track,” which is pretty concrete evidence that certain potentially suspect probing phrases are forbidden regardless of context.
So it seems that OpenAI doesn’t want to talk openly about the “reasoning” of its latest model. These models, in case you didn’t know, try to think through problems linearly. Users can see a filtered form of this reasoning, but OpenAI keeps the finer details hidden.
According to OpenAI, the decision to hide such “thought chains” was made “after weighing several factors, including user experience, competitive advantages, and the ability to continue thought chain monitoring.”
All of this reminds us that while OpenAI’s parent company is technically a nonprofit, the reality is much murkier. The company actually has a hybrid, kind of nonprofit and kind of commercial structure – recall that Elon Musk’s lawsuit against OpenAI claimed that the company deviated from its original founding agreement when it began pursuing profits. It’s not surprising that a somewhat for-profit company would want to maintain a competitive advantage by hiding its trade secrets, which in this case are “thought chains.”
It’s also a reminder to users that their chats are not completely private and free, which is sometimes easy to forget. I’ve worked on training these types of AI models before and can attest that many people “on the inside” so to speak can view user conversations when needed and relevant, whether for training purposes or other reasons.
And while it would be nice if these models had additional context awareness around the supposedly suspect phrases like “argumentation trace,” I think from OpenAI’s perspective it’s better to be safe than sorry.
