Artificial Super Intelligence: Between Opportunity and Existential Threat

The recent AI summit in Paris dominated the news with its stellar attendance (JD Vance, whose high octane visit of Europe kicked off in France, gave a memorable speech) as well as for its lofty goals.

In that French President Emmanuel Macron framed it as an “action summit”, international safety experts were left unimpressed, however.

BBeyond spoke post-summit to Cyrus Hodes, a member of the Global Partnership on AI (GPAI), where he co-leads the Safety and Assurance of Generative AI (SAFE) project.

Listening to him, it was clear that two parallel events took place that week: one that packed titans of the tech industry, investors, politicians and celebrities, and another that packed scientists and proponents of AI safety.

I asked Hodes to identify the main existential threats that unbridled development of AI poses to humanity.

Existential Threats

<<The first is Misalignment. This is where the goals of system and those of the user do not align. Values need to be embedded in AI systems.

The second is Loss of Control. This is where the system develops its own volition and goals. This can present defense and infrastructure threats, hence the need for global cooperation>>

“We need global standards and regulations and ethically-aligned design. At the moment, there is only self-regulation, however no policy makers, apart from OECD, are talking about this”

“An AI safety conference is essential; we don’t need another AI trade show”

What Hodes and a number of prominent scientists are sounding the alarm over is AGI, or Artificial General Intelligence.

<<AGI is capable of accomplishing the same tasks as a human. It will have the capacity of implementing self-improvements which can lead to ASI, or Artificial Super Intelligence, which would be impossible to control. We are not prepared for what that means: the loss of agency, the loss of jobs.  This concerns the very conservation of our society and of our species”

N.B. This image was created using DALL·E 3 on Open AI

The parallel event on AI focused on safety exclusively. Attendees included frontier labs, investors, policy makers, researchers, scientists. One prominent voice advocating for AI safety, was Stuart Russell, a Berkeley professor of computer science and an advisor at the Future of life Institute (https://futureoflife.org/).

“The gathering focused on mapping global risk, something that the G7 have been calling for since 2023. The Mapping is being peer reviewed; its utility is as a tool for policy makers”.

The global scientific community is keen on having this type of gathering, which will now take place twice a year, given the urgency.”

It will seek to engage not just policy makers, but multi-stakeholders, including investors. We need to educate investors that AI safety will make a lot of money too – we understand that this is very important.”

Get there first!

“Making money” is, of course, one of two drivers of the AI industry; the other is geo-political power. The two go hand in hand and the race is very real.

Sam Altman, CEO of OpenAI, has President Trump’s ear – as has Elon Musk, who recently made an offer for Open AI.

The new Gold Rush is the mad rush to building AGI and to dominate that space.

The CEO of Google DeepMind, Demis Hassabis, asserted that the tech industry is “perhaps five years away” from achieving AGI and that safety conversations needed to continue. “Society needs to get ready for that and … the implications that will have.”

He added that AGI carried “inherent risk”, particularly in the field of autonomous “agents”, which carry out tasks without human intervention.

But then, he struck the same “optimism v. trepidation” note as JD Vance, “I’m a big believer in human ingenuity. I think if we put the best brains on it, and with enough time and enough care … then I think we’ll get it right.”

He who controls the AI space will control the world.

Investors and policy makers understand this  – what they do with the knowledge will determine the fate of the rest of us.