Industry News
Cristiano, the inventor of RLHF and a director at OpenAI: It’s time for American AI laboratories to pause their activities for now.
The person who said this sits on the board of directors of OpenAI.
Paul Christiano is the inventor of RLHF, and ChatGPT uses this technology for training. He told the Dwarkesh Podcast on Monday that with the current level of security in U.S. AI labs, it's "not a good thing" to continue improving pre-training and reinforcement learning.
Asked whether the laboratory should pause, he replied: "I think it should." Regarding algorithm improvements that may be stolen quickly, he said more directly: "I think they should stop now."
He worries about one specific scenario: China steals a superintelligence midway through the race. At that point, even if the laboratory finds that its model is out of control, it cannot slow down.
Host Dwarkesh Patel mentioned in passing an example: A security startup entered Anthropic’s entire Slack within 12 hours.
Patel then asked: "But you are a director of OpenAI."
Cristiano responded: “It’s very important in this conversation that I’m not speaking as a member of the OpenAI board of directors.”
He also gave his own probabilistic judgment: If there are no factors that slow down progress, the possibility of superintelligence within a year is about one-third; and the risk of AI takeover during this period, he believes, is "more like 10%, not 1%."
The most weighty thing about his words was the reason he gave. He didn’t say that the model was too dangerous so he had to stop it, but that the laboratory couldn’t keep its results: the protection couldn’t keep up, and the faster it ran, the more valuable things could be stolen.
RLHF made the chat model useful, and now its inventor has come forward and said it's time to lock the door before running forward.