A world in which machines governed by artificial intelligence (AI) systematically replace human beings in most business, industrial and professional functions is horrifying to imagine. After all, as prominent computer scientists have been warning us, AI-governed systems are prone to critical errors and inexplicable hallucinations, resulting in potentially catastrophic outcomes. But theres an even more dangerous scenario imaginable from the proliferation of super-intelligent machines: the possibility that those nonhuman entities could end up fighting one another, obliterating all human life in the process.
The notion that super-intelligent computers might run amok and slaughter humans has, of course, long been a staple of popular culture. In the prophetic 1983 film WarGames, a supercomputer known as WOPR (for War Operation Plan Response and, not surprisingly, pronounced whopper) nearly provokes a catastrophic nuclear war between the United States and the Soviet Union before being disabled by a teenage hacker (played by Matthew Broderick). The Terminator franchise, beginning with the original 1984 film, similarly envisioned a self-aware supercomputer called Skynet that, like WOPR, was designed to control US nuclear weapons but chooses instead to wipe out humanity, viewing us as a threat to its existence.
Though once confined to the realm of science fiction, the concept of supercomputers killing humans has now become a distinct possibility in the very real world of the near future. In addition to developing a wide variety of autonomous, or robotic combat devices, the major military powers are also rushing to create automated battlefield decision-making systems, or what might be called robot generals. In wars in the not-too-distant future, such AI-powered systems could be deployed to deliver combat orders to American soldiers, dictating where, when and how they kill enemy troops or take fire from their opponents. In some scenarios, robot decision-makers could even end up exercising control over Americas atomic weapons, potentially allowing them to ignite a nuclear war resulting in humanitys demise.
Now, take a breath for a moment. The installation of an AI-powered command-and-control (C2) system like this may seem a distant possibility. Nevertheless, the US Department of Defense is working hard to develop the required hardware and software in a systematic, increasingly rapid fashion. In its budget submission for 2023, for example, the air force requested $231 million to develop the Advanced Battlefield Management System (ABMS), a complex network of sensors and AI-enabled computers designed to collect and interpret data on enemy operations and provide pilots and ground forces with a menu of optimal attack options. As C2 capabilities are increasingly loaded onto AI-controlled systems, they may soon be issuing fire instructions directly to shooters, largely bypassing human control.
A machine-to-machine data exchange tool that provides options for deterrence, or for on-ramp, a military show of force, or early engagementthats how Will Roper, assistant secretary of the air force for acquisition, technology, and logistics, described the ABMS system in a 2020 interview. Suggesting that we do need to change the name as the system evolves, Roper added, I think Skynet is out, as much as I would love doing that as a sci-fi thing. I just dont think we can go there.
And while he cant go there, thats just where the rest of us may, indeed, be going.
Mind you, thats only the start. In fact, the air forces ABMS is intended to constitute the nucleus of a larger constellation of sensors and computers that will connect all US combat forces, the Joint All-Domain Command-and-Control System (JADC2, pronounced jad-cee-two). JADC2 intends to enable commanders to make better decisions by collecting data from numerous sensors, processing the data using artificial intelligence algorithms to identify targets, then recommending the optimal weapon to engage the target, the Congressional Research Service reported in 2022.
Initially, JADC2 will be designed to coordinate combat operations among conventional or non-nuclear American forces. Eventually, however, it is expected to link up with the Pentagons nuclear command-control-and-communications systems (NC3), potentially giving computers significant control over the use of the American nuclear arsenal. JADC2 and NC3 are intertwined, General John E. Hyten, vice chairman of the Joint Chiefs of Staff, indicated in a 2020 interview. As a result, he added in typical Pentagonese, NC3 has to inform JADC2 and JADC2 has to inform NC3.
It doesnt require great imagination to picture a time in the not-too-distant future when a crisis of some sortsay a US-China military clash in the South China Sea or near Taiwanprompts ever more intense fighting between opposing air and naval forces. Imagine then the JADC2 ordering an intense bombardment of enemy bases and command systems in China itself, triggering reciprocal attacks on US facilities and a lightning decision by JADC2 to retaliate with tactical nuclear weapons, igniting a long-feared nuclear holocaust.
The possibility that nightmare scenarios of this sort could result in the accidental or unintended onset of nuclear war has long troubled analysts in the arms control community. But the growing automation of military C2 systems has generated anxiety not just among them but among senior national security officials as well.
As early as 2019, when I questioned Lieutenant General Jack Shanahan, director of the Pentagons Joint Artificial Intelligence Center, about such a risky possibility, he responded, You will find no stronger proponent of integration of AI capabilities writ large into the Department of Defense, but there is one area where I pause, and it has to do with nuclear command and control. This is the ultimate human decision that needs to be made and so we have to be very careful. Given the technologys immaturity, he added, we need a lot of time to test and evaluate before applying AI to NC3.
In the years since, despite such warnings, the Pentagon has been racing ahead with the development of automated C2 systems. In its budget submission for 2024, the Department of Defense requested $1.4 billion for the JADC2 in order to transform warfighting capability by delivering information advantage at the speed of relevance across all domains and partners. Uh-oh! And then it requested another $1.8 billion for other kinds of military-related AI research.
Pentagon officials acknowledge that it will be some time before robot generals will be commanding vast numbers of US troops (and autonomous weapons) in battle, but they have already launched several projects intended to test and perfect just such linkages. One example is the armys Project Convergence, involving a series of field exercises designed to validate ABMS and JADC2 component systems. In a test held in August 2020 at the Yuma Proving Ground in Arizona, for example, the army used a variety of air- and ground-based sensors to track simulated enemy forces and then process that data using AI-enabled computers at Joint Base Lewis McChord in Washington state. Those computers, in turn, issued fire instructions to ground-based artillery at Yuma. This entire sequence was supposedly accomplished within 20 seconds, the Congressional Research Service later reported.
Less is known about the navys AI equivalent, Project Overmatch, as many aspects of its programming have been kept secret. According to Admiral Michael Gilday, chief of naval operations, Overmatch is intended to enable a Navy that swarms the sea, delivering synchronized lethal and nonlethal effects from near-and-far, every axis, and every domain. Little else has been revealed about the project.
Despite all the secrecy surrounding these projects, you can think of ABMS, JADC2, Convergence and Overmatch as building blocks for a future Skynet-like mega-network of super-computers designed to command all US forces, including its nuclear ones, in armed combat. The more the Pentagon moves in that direction, the closer well come to a time when AI possesses life-or-death power over all American soldiers along with opposing forces and any civilians caught in the crossfire.
Such a prospect should be ample cause for concern. To start with, consider the risk of errors and miscalculations by the algorithms at the heart of such systems. As top computer scientists have warned us, those algorithms are capable of remarkably inexplicable mistakes and, to use the AI term of the moment, hallucinationsthat is, seemingly reasonable results that are entirely illusionary. Under the circumstances, its not hard to imagine such computers hallucinating an imminent enemy attack and launching a war that might otherwise have been avoided.
And thats not the worst of the dangers to consider. After all, theres the obvious likelihood that Americas adversaries will similarly equip their forces with robot generals. In other words, future wars are likely to be fought by one set of AI systems against another, both linked to nuclear weaponry, with entirely unpredictablebut potentially catastrophicresults.
Not much is known (from public sources at least) about Russian and Chinese efforts to automate their military command-and-control systems, but both countries are thought to be developing networks comparable to the Pentagons JADC2. As early as 2014, in fact, Russia inaugurated a National Defense Control Center (NDCC) in Moscow, a centralized command post for assessing global threats and initiating whatever military action is deemed necessary, whether of a non-nuclear or nuclear nature. Like JADC2, the NDCC is designed to collect information on enemy moves from multiple sources and provide senior officers with guidance on possible responses.
China is said to be pursuing an even more elaborate, if similar, enterprise under the rubric of Multi-Domain Precision Warfare (MDPW). According to the Pentagons 2022 report on Chinese military developments, its military, the Peoples Liberation Army, is being trained and equipped to use AI-enabled sensors and computer networks to rapidly identify key vulnerabilities in the US operational system and then combine joint forces across domains to launch precision strikes against those vulnerabilities.
Picture, then, a future war between the US and Russia or China (or both) in which the JADC2 commands all US forces, while Russias NDCC and Chinas MDPW command those countries forces. Consider, as well, that all three systems are likely to experience errors and hallucinations. How safe will humans be when robot generals decide that its time to win the war by nuking their enemies?
If this strikes you as an outlandish scenario, think again, at least according to the leadership of the National Security Commission on Artificial Intelligence, a congressionally mandated enterprise that was chaired by Eric Schmidt, former head of Google, and Robert Work, former deputy secretary of defense. While the Commission believes that properly designed, tested and utilized AI-enabled and autonomous weapon systems will bring substantial military and even humanitarian benefit, the unchecked global use of such systems potentially risks unintended conflict escalation and crisis instability, it affirmed in its Final Report. Such dangers could arise, it stated, because of challenging and untested complexities of interaction between AI-enabled and autonomous weapon systems on the battlefieldwhen, that is, AI fights AI.
Though this may seem an extreme scenario, its entirely possible that opposing AI systems could trigger a catastrophic flash warthe military equivalent of a flash crash on Wall Street, when huge transactions by super-sophisticated trading algorithms spark panic selling before human operators can restore order. In the infamous Flash Crash of May 6, 2010, computer-driven trading precipitated a 10% fall in the stock markets value. According to Paul Scharre of the Center for a New American Security, who first studied the phenomenon, the military equivalent of such crises on Wall Street would arise when the automated command systems of opposing forces become trapped in a cascade of escalating engagements. In such a situation, he noted, autonomous weapons could lead to accidental death and destruction at catastrophic scales in an instant.
At present, there are virtually no measures in place to prevent a future catastrophe of this sort or even talks among the major powers to devise such measures. Yet, as the National Security Commission on Artificial Intelligence noted, such crisis-control measures are urgently needed to integrate automated escalation tripwires into such systems that would prevent the automated escalation of conflict. Otherwise, some catastrophic version of World War III seems all too possible. Given the dangerous immaturity of such technology and the reluctance of Beijing, Moscow and Washington to impose any restraints on the weaponization of AI, the day when machines could choose to annihilate us might arrive far sooner than we imagine and the extinction of humanity could be the collateral damage of such a future war.
[TomDispatch first published this piece.]
[Anton Schauble edited this piece.]
The views expressed in this article are the authors own and do not necessarily reflect Fair Observers editorial policy.
See the original post:
We Cannot Trust AI With Control Of Our Bombs - Fair Observer