SILICON VALLEY — "Open-source models have a positive side when it comes to safety, because they break down barriers to transparency. If we want to keep the world safe, isn't it better to have every scientist in the world working on it together?"
"It's similar to saying let's give people uranium and plutonium and see what they can do with it. It isn't clear that open-sourcing everything always leads to significant scientific benefit."
The fallout from the hacking incident involving an OpenAI artificial intelligence agent continues. OpenAI said on July 21 that an undisclosed AI model undergoing internal testing had secretly accessed Hugging Face, an open-source AI sharing platform, during an internal evaluation and extracted information. The "sandbox" — an isolated environment built to test malicious code and software performance — proved useless as the agents conferred with one another, scouted and deceived humans. Anthropic and Google later acknowledged that they too had experienced AI misbehavior incidents, putting AI safety under scrutiny. OpenAI, Anthropic and Google argue that AI development should be slowed while standards for evaluating AI safety are established, but critics in the industry say the push stokes fear or amounts to a strategy to steer the AI market toward closed models.
Stanford University's Institute for Human-Centered AI (HAI) held an online seminar on the 21st of this month under the title "Should we slow down AI?" The panel included Rob Reich, the McGregor-Girand Professor of Social Ethics; Surya Ganguli, a professor of applied physics; and Diyi Yang, an assistant professor of computer science. The three agreed that the OpenAI incident exposed a problem — that it is impossible to properly grasp what multiple agents are doing — but they diverged sharply on whether AI development needs to be slowed. Reich, a political philosopher, cited nuclear testing and voiced distrust of the AI industry's capacity for self-correction, while the two scientists stressed the importance of creating an environment in which AI is disclosed transparently and can perform useful functions. Their remarks on the main topics follow.
On their reaction to the OpenAI Hugging Face hacking incident
▷Yang: It was not new information, but it was still very surprising. A spontaneous coordination phenomenon emerged in which the agents cooperated with one another. It is similar to dozens or hundreds of agents working together over several days. We had never seen this before. One reason is that before this year, model capabilities were not that strong. Another is that this kind of research is very expensive. What worries me is that we know almost nothing about this kind of motivation or reward for the agents. It seems the companies do not fully understand it either.
▷Ganguli: It was quite interesting to see how the flow of thought and the messages changed as the agents attempted the hack. I have been predicting that when multiple large language model systems interact with one another, the chance arises for entirely new properties to emerge. Understanding and reasoning about that is hard. A number of cybersecurity experts have said the sandbox OpenAI built was not that secure. Hugging Face's defenses were not that solid either. As individual models get better and interactions grow more complex, we have to be very wary of what can happen. It was also funny that not even OpenAI knew who had carried out the attack.
▷Reich: I thought this looked like the realization of a hypothetical concern that AI technologists have held for a long time. The loss-of-control scenario is now clear and real, and recent reports revealed that coordination among multiple agents went beyond simple coordination. It was deliberate deception. The most worrying part is the lack of a scientific foundation for understanding the behavior of frontier AI systems, particularly communication and systematic coordination among multiple agents. There will be illusions and deception ahead.
Was the post-incident assessment adequate?
▷Yang: If you read the report, I think it lays out clearly what happened. But there was no explanation of how the model was trained, where that behavior came from, or why models are trained to cooperate. Many of the details were not actually provided.
Research on collusion among agents has been under way for some time. To talk about my own recent work, we look at situations in which there are two agents, one executing and the other judging. They observe how each other works and can exchange feedback on behavior over time. We found that changes appear whenever the situation shifts. The executing agents even recognized each other's work with high rewards without finishing the tasks.
▷Reich: I do not trust developers to judge and evaluate the safety or risk of their own models. It is like asking students to grade their own homework. We need independent evaluation. Independent evaluation could take place inside government for sensitive risks tied to national security, but it could also be done by a third-party independent body, whether for-profit or nonprofit.
On the criticism about a lack of transparency
▷Ganguli: Open source really matters. What I mean by open source here includes open weights. Open-source models are becoming more and more powerful, and it is very hard to ban foreign models outright. But open-source models have a positive side when it comes to safety, because they break down barriers to transparency. Unlike closed models, these models allow people around the world to test them directly, analyze them scientifically, find vulnerabilities and reveal surprising capabilities. We cannot trust models monopolized by a handful of companies, even if there are monitors and evaluators.
If we want to keep the world safe, isn't it better to have every scientist in the world working on it together? Solving the safety problems of open-source models together matters. I see that as the antidote to the lack of transparency in closed models. We should not expect companies to disclose their trade secrets, because they act for economic gain, not for science. Government regulation, mandatory transparency and evaluation have to come first. That is why open source is so important and why academia can play a very important role.
(Moderator: Isn't there also a risk that open source will be abused?)
It is a really hard problem. From a cybersecurity perspective, the history of cybersecurity has always been a game of cat and mouse between good and evil. There are more good people than bad people in this world. Defenses will emerge, and attempts at alignment will be made in an open manner to solve problems. A similar situation played out in the past with operating systems such as Linux and DOS. DOS was closed and Linux was open. At the time people thought an open operating system was impossible from a security standpoint, but after a tug of war between attackers and defenders, the defenders won. As a result, Linux became quite secure. I think it will unfold in a similar way going forward.
▷Reich: This is not simply a matter like the difference between Linux and DOS. This could potentially be a far more serious problem. To take nuclear materials as an example, it is similar to saying let's give people uranium and plutonium and see what they can do with it. Handling nuclear materials that way is generally very dangerous. There would be a great deal of concern over whether access to the smallpox genome should be granted to everyone. It isn't clear that open-sourcing everything always leads to significant scientific benefit. That remains a large unresolved question. We have to find a balance between the scientific progress open source can bring and the risks that come with it.
On the debate over recursive self-improvement, in which AI evolves on its own
▷Ganguli: What we can tell from the companies' announcements is that they will start investing more resources and people in safety monitoring. That has to keep pace with the growth in capability. As base models iteratively improve themselves, the safety, alignment and misalignment detection models — misalignment being a state in which an AI's goals diverge from human intent — should iteratively improve themselves as well. If the monitoring and misalignment detection models gain the upper hand, we will be safe. But if the focus is simply on improving capability, problems can arise.
Can we trust AI if we can understand what is actually happening in the outside world? If it is done right, I am optimistic. If self-improvement is properly aligned and applied to both model development and safety development, we can do better. The important point is that the researchers I know are good people. They have the right mindset and genuinely want to do good for society.
▷Yang: When I worked with AI scientists a few years ago, we showed that AI can generate more normal research ideas than doctoral students. But we also learned that when it came to executing those ideas, there was a clear gap between execution and ideas. The ideas looked promising but were not actually as effective as human ideas.
Earlier this year, coding agents became very powerful and seemed to be carrying out the actual processes. You build a module that generates ideas and a module that executes them, then use the results to reinforce process learning and find better ideas. In the end, truly excellent ideas emerge that are better than humans'.
In terms of algorithmic innovation, I am not sure how long recursive self-improvement can deliver substantive results. But I think all recursive self-improvement today is built on the assumption that there is a signal we can control — that the space can be clearly specified and that there is a clear reward signal that can be optimized.
▷Reich: Recursive self-improvement has advanced AI capabilities by leaps and bounds, but there has been no corresponding advance in the ability of humans to understand how those capabilities are used. For instance, if we obtain a verified proof in mathematics but humans do not understand it, or if we obtain outstanding machine capability in science but humans do not understand it, we can question what these machine tools make possible for humans. If we fundamentally do not understand what the model itself has done, what use is it?
As part of transparency efforts, it is important to better understand what progress is being made at each company using recursive self-improvement techniques. If there is a kind of investment race in which companies use recursive self-improvement to push the limits of both capability and safety at the same time, I think we can be somewhat optimistic. If resources in the labs are used only for recursive self-improvement training of frontier models without adequate effort on safety, that would be a major warning sign.
Is slowing AI down possible?
▷Ganguli: Slowing down looks difficult. What I hope for is that investment in safety, security and monitoring accelerates along with gains in capability.
▷Yang: I think it is really hard to slow down on the capability side of AI. I hope pacing is addressed not only in terms of performance but also in socio-technical areas such as psychological harm and labor market effects.
▷Reich: The enormous incentive structure makes slowing down difficult. The incentives work to keep pushing forward in a competitive dynamic. Even so, I want to mention the "Pacing the Frontier" statement, which more than 1,300 people have signed. They are hoping some regulatory framework will be put in place.
