AI could spiral out of human control.

Accordingly, two leading AI scientists, Max Tegmark, a professor at the Massachusetts Institute of Technology (MIT), and Professor Yoshua Bengio at the Université de Montréal (Canada), have warned that artificial general intelligence (AGI) designed on an "agent" model could become dangerous because humans risk losing control of this system.

According to them, AGI is a term used to describe AI systems with intelligence superior to humans, capable of independent thinking and decision-making. Without strict control mechanisms, these systems could act against human intentions, leading to unpredictable consequences.

Scientists' concerns stem from the fact that many large technology corporations are promoting the concept of "AI agents," that is, chatbots capable of acting as digital assistants, supporting work and daily life. However, the actual timing of AGI's emergence remains uncertain, with many different predictions.

According to Yoshua Bengio, the problem lies in the fact that AI systems are gradually gaining "autonomy" and the ability to think independently. He shared this on the podcast.Beyond The Valley" (translated: "Beyond Silicon ValleyIn an interview with CNBC on February 4th, he explained: “AI researchers are drawing inspiration from human intelligence to build machine intelligence. In humans, intelligence is not just the ability to understand the world, but also includes agent behavior, using knowledge to achieve goals.”

Bengio warned that this is how AGIs are developing, becoming agents with extensive knowledge of the world and the ability to act accordingly. However, he stressed that this is “really very dangerous.”

According to him, pursuing this model is tantamount to "creating a new species" or a "new intelligent entity" on Earth that humans cannot be sure will behave in a way that benefits us.

"We need to consider worst-case scenarios, and the key is always the causal factor. In other words, AI could have its own agenda, and that could get us into trouble."

Bengio also warned that AI could develop its own self-defense mechanisms as it becomes more intelligent, which could put humanity in a confrontational position.

“Do we want to compete with entities smarter than ourselves? This is clearly not a safe gamble. We need to understand how our ability to defend ourselves can become a target for AI.”

According to Professor Max Tegmark from MIT, the safe solution lies in the "instrumental AI" model, meaning systems designed for a specific task, not acting as independent agents.

He gave the example of an AI tool that could assist in finding a cure for cancer or a self-driving car system. These technologies can still possess powerful capabilities, but it must be ensured that humans can control them with a high degree of reliability.

"I believe that, optimistically speaking, we can still harness most of the benefits that AI offers, provided we apply basic safety standards before widely deploying powerful AI systems," Tegmark emphasized.

According to him, companies must demonstrate that humans can control AI before commercializing it. Once this principle is followed, the industry can quickly innovate to find safer ways to deploy AI.

In 2023, the Future of Life Institute, founded by Tegmark, called for a pause in the development of AI systems that could match or surpass human intelligence.

Although this call has not yet materialized, he believes that at least the topic has been discussed more extensively, and this is a crucial time to act in order to establish barriers to control AGI.

"Many people are talking about this issue, but the crucial question is whether we can get them to act," Tegmark shared on CNBC's Beyond The Valley podcast.

He warned, "It would be insane for humanity to create something smarter than itself without first figuring out how to control it."

Predictions about when AGI will emerge remain controversial, partly due to differing definitions. Some experts believe AGI is still a long way off, while others believe it could be within the next few years.

Sam Altman, CEO of OpenAI, stated that his company has mastered the art of building AGI and that the technology could launch sooner than many anticipate. However, he also attempted to temper the hype surrounding the impact of AGI on society.

Speaking last December, Altman said: “I guess we’ll achieve AGI sooner than most people think, but the impact won’t be as big as many people imagine.”

While not disclosing specific details about the AGI development progress, Altman's statement suggests OpenAI is getting closer to this goal, while also raising many questions about the world's preparedness for the emergence of an intelligent system capable of matching or surpassing humans.