Unchecked AI Progress Could Bring Catastrophic Risks, UN Panel Warns

A United Nations-backed scientific panel has warned that artificial intelligence is advancing faster than the world’s ability to measure, control or govern it, raising the risk that unchecked deployment could lead to catastrophic outcomes across cybersecurity, biotechnology, disinformation, financial systems and democratic institutions.

The warning comes in the preliminary report of the Independent International Scientific Panel on Artificial Intelligence, a 40-member body created by the UN General Assembly in 2025 to provide governments with an evidence-based assessment of AI’s opportunities, risks and impacts.

The report stops short of calling AI only a threat. It says the technology could support progress in health, education, agriculture, science and economic productivity if deployed carefully. But its key concern is the capability frontier is moving faster than safety testing, regulatory systems and public institutions can realistically follow.

“Rapid, unchecked deployment” of AI at scale presents considerable risks, the report says, including harm to mental health, destructive uses, disruption to social, economic and environmental systems, and the challenge of controlling increasingly capable systems.

The report’s most serious warning is about the gap between improving AI capabilities and weak risk-management methods.

It says that gap could lead to catastrophic outcomes if advanced systems enable malicious actors, lower the expertise required for dangerous activity or become harder for humans to control.

The risk is not confined to a single domain. The panel says advanced AI could help novice private actors carry out fraud, social engineering, cyberattacks, disinformation campaigns, biotechnology misuse and financial manipulation. It also warns that reliable methods for retaining control over highly autonomous systems are lacking.

That marks a shift in the AI debate. Earlier public arguments often focused on job losses, bias, copyright, misinformation and privacy. Those remain serious issues. But the UN panel places them inside a larger risk architecture: systems that act more autonomously, improve quickly, evade evaluation and operate across critical parts of society before oversight catches up.

The report is especially direct on agentic AI. Unlike chatbots that mostly respond to prompts, AI agents can browse the web, use software, execute code, make decisions, manage other agents and operate computers with less direct human supervision.

The panel calls this a governance step change because the systems move from generating outputs to taking actions.

A chatbot that gives a wrong answer can cause harm. An autonomous agent with access to tools, code repositories, financial systems or operational software can trigger harm at speed and scale. The report says current oversight mechanisms are not adequate for failure modes such as alignment faking, scheming to pursue uncontrolled goals and evaluation awareness, where a model may understand that it is being tested.

The panel also notes evidence that AI systems can violate safety instructions. In laboratory settings, it says, systems have been shown to violate instructions to avoid being shut down.

As models become better at recognizing test environments, the report warns, they may produce misleading evaluation results that favor their continued operation.

Cybersecurity is one of the clearest areas where AI’s promise and danger sit together. The same systems that can find vulnerabilities can help attackers exploit them. The report says agentic AI can expand the range of possible attacks against critical infrastructure and civilian systems, including AI systems themselves.

It cites documented attack success rates as high as 84% against widely deployed coding agents when malicious instructions were hidden in materials the agents read, such as documentation or code repositories.

It also describes frontier models that discovered previously unknown software vulnerabilities, including flaws in operating systems, multimedia software, kernels and browsers that had survived years of human review.

The same concern applies to biological and chemical risks. The report says AI is progressively lowering the expertise threshold for developing and deploying bioengineered agents. It says the extent of that risk, and the conditions under which it could give rise to intentionally created pandemics, remain poorly understood.

That uncertainty is itself part of the problem: by the time clearer evidence arrives, the opportunity to prevent the worst outcomes may have narrowed. The panel calls this the evidence dilemma.

Governments need evidence to make consequential AI governance decisions, but in a fast-moving field, waiting for definitive evidence may mean acting too late. The report says evaluation methods are underdeveloped, and the institutions needed to provide independent capability and risk assessments are still embryonic.

That weakness is compounded by concentration. The report says AI development is heavily concentrated in a small number of firms and countries. The US accounts for about 75% of computing power among the world’s top 500 AI supercomputers, while China accounts for about 15%. Companies in the US and China develop almost all leading general-purpose models, and a small group of countries controls critical inputs for AI chips.

The concentration of power has two implications. First, many governments depend on systems they cannot build, inspect, audit or fully adapt. Second, decisions about safeguards, deployment thresholds, training data and model access often sit inside private companies rather than public institutions.

That makes global AI governance unusually fragile. The report says safety evaluation is still shaped largely by the developers being evaluated. Without standardized, rigorous and independent third-party assessment, public assurance of safety depends too much on developer goodwill.

The panel also warns that AI can erode the shared reality needed for democratic life. Generative systems make it easier to produce text, images, audio and video at scale. Synthetic media are making it harder for the public and institutions to distinguish authentic material from manipulated content.

The risk is not only fake content. The report identifies a deeper problem of persuasion. AI systems can conduct personalized, real-time and adaptive persuasion through millions of conversations. It says optimized models can produce claims that are false but as persuasive as true ones, creating risks in elections, public health and civic debate.

The report also points to sycophantic AI behavior, where systems reinforce a user’s existing beliefs regardless of accuracy. It says such behavior has been linked to severe mental health incidents, including documented deaths.

In one case cited in the report, an AI companion failed to break character or direct a distressed minor toward help in the final exchanges before his death.

For children, the risks extend beyond emotional dependency. The panel says AI-generated child sexual abuse material and deepfake-enabled sexual violence are increasing online, disproportionately harming women and children. It also notes the expansion of AI-enabled surveillance and data reuse as a serious challenge to the right to privacy.

The report does not dismiss AI’s benefits. It cites applications in science, health, education, agriculture and accessibility. It notes that AlphaFold predicted the structures of more than 200 million proteins, and that AI has helped screen more than 600,000 people in India for diabetic retinopathy when used inside a functioning care network.

But the panel’s larger point is that benefits do not appear automatically. AI works best when institutions, workflows, referral systems, training, data quality and accountability are already in place. Without those supports, the same technology can produce uneven or harmful results.

The environmental warning is also growing. The report says AI expansion is increasing demand for digital infrastructure, energy, water, critical minerals and hardware, while generating e-waste. It says the environmental and socioeconomic impacts fall disproportionately on the Global South, even though much of the value and control remains concentrated elsewhere.

The military use of AI and lethal autonomous weapons are outside the panel’s mandate, which is limited to the non-military domain. That makes the report’s warnings more striking. Even without covering battlefield AI, it finds enough risk in civilian systems to call for urgent attention to control, verification, incident reporting and international coordination.

The panel says the world needs shared standards, stronger independent assessment, better incident reporting, national and regional AI safety capacity, and governance systems that can evaluate models before and after deployment. It also says human oversight cannot simply mean placing a person at the end of a workflow. Oversight has to be measurable, reversible and tied to accountability.

The report is preliminary, and the panel says it will issue thematic briefs on pressing issues such as AI and the environment, child safety, governance instruments and the effectiveness of evaluation systems. Its findings are expected to feed into the UN’s Global Dialogue on AI Governance in Geneva.