AI is penetrating every corner of the military domain at an unprecedented pace. From cyber defense to weapon systems, from intelligence analysis to command and decisionmaking, AI’s presence is ubiquitous. However, the widespread adoption of this technology also brings with it unprecedented risks. In 2024, the leaders of China and the United States reached a consensus on ensuring that nuclear weapons remain under human control at all times. In May of this year, the two countries’ heads of state further proposed building a constructive strategic stability relationship. The governance of military AI should be an integral part of this effort. Yet these political declarations still fall short of preventing AI from spiraling out of control, being misused, or triggering escalations in conflict. China and the United States should explore how to translate their principled consensus into actionable institutional arrangements, while also identifying the practical challenges currently facing them. In future dialogues on military AI, broader issues—including the regulation of lethal autonomous weapons systems—should also be incorporated.

How can the principles be put into practice?

Both China and the United States have repeatedly emphasized the importance of ensuring the principle of “human control,” as reflected in their existing joint statements as well as in their respective policy positions. Melanie Sisson’s article further argues that the authority to launch cyberattacks against nuclear command systems and critical infrastructure must remain firmly in human hands. However, to prevent these declarations from becoming mere formalities, China and the United States must establish practical cooperation mechanisms in three key areas.

First, they should develop a list of red lines for military AI—clearly prohibiting AI from autonomously launching attacks on NC3 systems or autonomously deciding to use nuclear weapons. At the same time, they should also explore a list of critical infrastructure sectors (such as finance, health care, energy, etc.) and clearly define the boundaries.

Second, they should reach a consensus on what constitutes “meaningful human control.” In the 1970s, United States Air Force Colonel John Boyd proposed the renowned OODA loop—in which whoever can “observe, orient, decide, and act” more quickly in a competition or conflict will seize the initiative.  Later, discussions centered on “human-in-the-loop” and “human control” are based on that theoretical foundation. Yet even before Boyd, Chinese Premier Zhou Enlai had already issued instructions demanding “absolute certainty” (万无一失) in China’s nuclear testing efforts. In the view of Chinese experts and practitioners, this instruction continues to hold sway today and embodies the principle of “human control.” Given the two sides’ fundamentally different institutional frameworks and strategic cultures, even when addressing the same issue, they may employ different terminology and adopt divergent approaches. China and the United States need to ensure mutual acceptance of each other’s definitions of “meaningful human control” and related safeguard mechanisms through regular dialogue or by establishing a terminology working group.

Third, they should establish a China-U.S. military hotline for AI. Building on existing China-U.S. military communication mechanisms—such as the defense minister hotline and the Maritime Military Security Consultation Mechanism—adds a dedicated channel for reporting emergency incidents triggered by AI errors or AI cyberattacks. The goal is to reduce the risk of cascading crises caused by AI malfunctions that could mislead humans or even override human judgment. The 2023 balloon incident should be a wake-up call for both China and the United States. Without the safeguard of a hotline mechanism, AI-driven drones and unmanned vessels are highly likely to trigger more accidental escalations in the future.

Greater real-world challenges

Meanwhile, geopolitics and rapid technological iteration—both of which neither side can afford to ignore—are intensifying the security dilemma. Even if China and the United States were to establish a cooperative mechanism on “human control,” they would still face at least three major challenges in practice. The first is the speed dilemma. With the help of AI agents, cyberattacks are unfolding at an unprecedented pace. AI-powered defense systems could automatically trigger countermeasures. Imagine a scenario where an AI detects abnormal traffic targeting the NC3 system, prompting the defensive AI to automatically disconnect the network or launch countermeasures. As a result, the other side might mistakenly interpret this as a cyber intrusion, ultimately pushing both sides’ AI systems into a mode of “automatic confrontation.” AI-enabled cyberattacks can be carried out within milliseconds. In such ultra-high-speed confrontations, the “final human decision” is likely to exceed physiological limits.

Second, there’s the challenge of attribution. It is extremely difficult to trace the origin of cyberattacks. Critical infrastructure can be targeted by a state actor or by nonstate actors such as hacker groups, terrorists, or even AI agents themselves. If critical infrastructure falls victim to an AI-powered cyberattack, how can we determine whether the attack was authorized by the government or whether a nonstate actor has abused AI—or even deliberately framed a state actor to provoke conflict? Without a reliable attribution mechanism, human accountability cannot be effectively established, and retaliatory actions are all too likely to trigger unintended escalations.

Third is the security dilemma. Compared to traditional military technologies, the opacity of AI technologies makes it impossible to accurately assess the actual level of development of an adversary’s AI capabilities, the extent to which these technologies can enhance military operational effectiveness, or even to distinguish whether a particular military AI technology is offensive or defensive in nature. As a systemic undertaking, the effectiveness of AI’s military applications is further influenced by factors such as the software and hardware integration with various subsystems, human-machine collaboration, military institutional mechanisms, and strategic culture—factors that require a relatively lengthy and complex evaluation process. As a result, both sides often imagine their adversaries’ AI deployments based on the worst-case scenario and adopt countermeasures that are equally confrontational. Consequently, threats may be seriously exaggerated, and the spiral of hostility continues to escalate.

These challenges will be difficult to eliminate completely in the near term and can only be managed through continuous strengthening of dialogue and the establishment of safeguards. However, cyberattacks represent merely the tip of the iceberg when it comes to military AI risks between China and the United States; many other critical issues also deserve to be incorporated into bilateral military dialogues. Among these, lethal autonomous weapons systems (LAWS) urgently require effective control. In the Russia-Ukraine theater and the Middle East, the widespread use of autonomous weapons systems has already led to severe civilian casualties and even heightened the risk of nuclear escalation. Once LAWS fall into the hands of nonstate actors, the security risks they pose are immense.

Given the difficulty in achieving progress on LAWS negotiations within the framework of the United Nations Convention on Certain Conventional Weapons, China and the United States should seek a minimum common ground on “prohibiting fully autonomous lethal attacks” and “ensuring human oversight,” and engage in dialogue and cooperation around measures to prevent the proliferation of LAWS and their misuse by nonstate actors. Throughout the history of U.S.-China relations, there have been numerous accidental incidents involving unmanned vehicles or equipment. Most of these systems are deployed in hotspot regions, and once combined with AI and offensive military capabilities, they could very likely trigger unexpected escalations of crises. Both sides should also promptly explore crisis management and response plans for autonomous weapon systems, working together to uphold regional and global strategic stability.

The governance of military AI is not a negotiation that can afford to wait—it’s a race against time. As machines increasingly seize “the right to fire” and even “the right to engage in combat,” China and the United States, as two AI superpowers, bear the historic responsibility of preventing technology from spiraling out of control and safeguarding human civilization. We look forward to seeing the leaders of China and the United States build on the positive momentum of their ongoing AI dialogue during their September meeting, setting an example for global governance of military AI.

Original link：https://www.brookings.edu/articles/advancing-human-control-of-military-ai/