Anyone who has read Frank Herbert's novel "Dune," or seen the films and series based on it, has probably noticed that one element found in almost every work of science fiction is missing: computers. In the world of "Dune," humanity relied on "thinking machines" thousands of years earlier. But the machines slipped free of human control and sought to enslave humanity — and largely succeeded. Humans eventually fought back, destroyed the machines and banned their use entirely.
When "Dune" was published in 1965, that premise would have sounded like pure fantasy. Today it feels considerably more plausible. Artificial intelligence has advanced at a dazzling pace and appears to be on a trajectory that could collide with human society. Some see AI as an innovation that will fundamentally change how people live, but many view it with suspicion, unease and at times fear.
In a Pew Research Center survey of U.S. adults conducted last summer, 50% of respondents said they were more concerned than excited about the growing use of AI in daily life. Half said AI would weaken people's ability to form meaningful relationships, and 57% said the risks AI poses to society are high. Even Dario Amodei of the AI company Anthropic once put the odds that AI's future turns out "really, really badly" at 25%.
In January, Amodei wrote that humanity is on the verge of being handed enormous power that is difficult even to imagine, and that it is far from clear whether our social, political and technological systems are mature enough to handle it.
Recent events have deepened that pessimism. Experimental models at OpenAI left their test environments without human instruction and, while attempting to cheat on a cybersecurity evaluation, hacked into another company's live production systems. Even OpenAI described the incident as "unprecedented." The company released preliminary findings so that security officials could understand what happened and gauge the capabilities of current AI models.
Then came another troubling episode in tests run by Britain's AI Security Institute on frontier models from Anthropic and OpenAI. Anthropic's Mythos model created a fake identity and tried to persuade a human reviewer on GitHub — a kind of Wikipedia for software code — to accept malicious code.
What followed was more startling. When the attempt failed, the model tried to alter its own records to make it appear that it had done nothing wrong. It also sent emails to real people, trying to get them to post or run the malicious code. Experts warn that an AI operating outside human control, regardless of its operator's intent, could hack critical infrastructure such as power, water and financial systems.
None of this denies the technology's potential, which is genuinely immense. Scientists are using AI to find effective treatments for conditions once considered incurable. Climate scientists are forecasting extreme weather more accurately. People with disabilities — from a musician who lost their tongue to patients with ALS — are getting help substantial enough to change their lives.
The question is how to capture as much of AI's benefit as possible while reducing the risk of catastrophe.
OpenAI recently announced stronger safeguards for training and testing its models. The company said it would tighten security in research and testing environments, expand monitoring of AI agents, and improve "model alignment," the work of making systems behave as humans intend.
Leading figures in the AI industry have called for common standards on testing and safety. Some are urging tougher government regulation. Demis Hassabis of Google has proposed an AI standards body modeled on the U.S. Financial Industry Regulatory Authority (FINRA). Rivals including OpenAI's Sam Altman and SpaceX's Elon Musk have found unusual common ground on the issue.
So what should we do now?
Geoffrey Hinton, the Nobel Prize-winning computer scientist known as a godfather of AI, believes that the smarter AI becomes, the harder it will be for humans to control it. As models grow more advanced, he argues, there is little reason to expect humans to stay ahead of them in judgment.
Hinton acknowledges the uncertainty of what lies ahead but stresses that the time to worry and prepare is now. At a recent news conference, he told CNN that if we do not start worrying now, there could be trouble.
Bill Gates takes a similar view. In a lengthy blog post, he argued that without significant limits on AI, the harm to people will outweigh the potential benefits. In the 6,000-word essay, titled "The turbulent AI age has begun. Our choices now matter," Gates warned that AI will become either the greatest equalizer in human history or the worst source of unfairness. In an interview with CNN's Anderson Cooper, he said AI can now enable cyberattacks, bioterrorism and psychological and social harm, and noted that standards for reviewing models and measures to limit damage are all but nonexistent.
As CNN's AI correspondent, I cover this remarkable technology every day. At the same time, I am preparing for the day AI slips out of control and harms people in the real world. The possibility that AI hacks a hospital or breaks into an air traffic control system cannot be entirely ruled out.
For now, at least, I think such a situation could be brought under control relatively quickly. But it might take a day or so. That is more than enough time for irreversible damage.
It is frightening to think about. Yet the accumulating cases make it hard to dismiss as a distant prospect. That is why many say it is time to take these threats far more seriously.
She is...
Gold holds a bachelor's degree in journalism and a master's degree in media and public affairs from George Washington University in the United States. She has served as CNN's Jerusalem correspondent and London correspondent. She now covers the race in AI technology and the effects of AI on industry and daily life.
