014 - National Power & War
Evolving the threat and control landscape with Military AI
Editors Note: This blog article is a summary of an in-person event held in San Francisco on 2023-10-15 facilitated by
.
The development of fully autonomous weapon systems capable of engaging in warfare is top of mind for many when considering an AI-powered future. On the positive side, there is the possibility of warfare with minimal human casualties. The prospect of robotic soldiers and drones directed by AI algorithms taking the place of human combatants offers the tantalizing promise of limiting the loss of life in future conflicts. However, this scenario also raises profound moral and ethical questions about the delegation of life-and-death decisions to machines. Even if AI-powered weapons prove to be more precise and efficient than emotionally vulnerable and fatigue-prone human soldiers, the idea of outsourcing kill decisions to algorithms remains deeply unsettling. Moreover, the reduced human cost of waging war could potentially lower the threshold for initiating conflicts, making societies more tolerant of military aggression.
In the near term, militaries around the world are already deploying AI technologies for a range of applications, including intelligence gathering, logistics optimization, and decision support. The combination of high-resolution satellite imagery and advanced AI analysis tools is providing nations with an unprecedented level of visibility into the capabilities and resources of their adversaries. Sophisticated algorithms can process vast troves of data to detect emerging threats, predict enemy strategies, and devise optimal countermeasures. The increasing reliance on AI in military decision-making processes suggests that, in the future, AI may play a greater role in shaping the terms of peace than in directly waging war. Some of the more sophisticated defense companies like Palantir are already describing their perspective on AI warfare, as they create sophisticated software weapons systems like Gotham. Newer players to the market like Anduril are making AI-based intelligence and real-time action core their offerings.
However, it is important to recognize that the current state of AI in defense is less advanced than is often portrayed in popular media. While the potential for AI to revolutionize warfare is immense, many of the most transformative applications remain in the realm of speculation and research. The successful integration of AI into military systems will require not only significant technological breakthroughs but also a deep understanding of the unique challenges and requirements of the defense domain.
As an open-ended technology that may first reach transformative levels in a multipolar world, AI could also emerge as a destabilizing force on the global stage. The race to achieve military supremacy through AI is already underway, with nations around the world pouring vast resources into the development of cutting-edge AI capabilities. In this context, the advantage often lies with the offense, as it is always possible for a sufficiently motivated and well-resourced adversary to develop attacks against AI systems. As AI becomes more deeply integrated into military hardware and software, the number of potential vulnerabilities will only continue to grow.
The threat of cyber attacks designed to steal AI models, poison training datasets, install hidden exploits, or simply degrade system performance looms large over the future of AI in warfare. The prospect of a compromised AI system making high-level military decisions is a deeply disturbing scenario that highlights the critical importance of cybersecurity in the age of AI. Securing the integrity of AI supply chains and protecting against the theft of sensitive data and algorithms will be an increasingly critical priority for militaries around the world.
The allure of AI is also driving governments to classify more of their military capabilities, as nations seek to maintain a strategic edge over their rivals. An AI-enabled intelligence apparatus could potentially piece together comprehensive pictures of adversaries' intentions and capabilities from disparate sources of data. In the most extreme case, a nation's entire military strategy and decision-making process could be embedded within the uninterpretable weights of a massively complex AI model, making it virtually impossible to decipher or predict.
As nations train increasingly sophisticated AI systems to counter and defeat other AIs, there is a risk that these technologies could one day turn against their creators. The phenomenon of model collapse, in which large language models trained on AI-generated outputs become unstable and unreliable, underscores the inherent fragility of simply pursuing raw capability without adequate safeguards. There are also growing concerns about the potential for AI to be weaponized for psychological warfare and disinformation campaigns, with the aim of radicalizing populations and sowing chaos and division.
While much of the public discourse around military AI has focused on great power competition, the destabilizing effects of this technology may be even more pronounced in smaller-scale conflicts. The proliferation of low-cost, AI-powered drones and other autonomous weapons could dramatically alter the balance of power in proxy wars and insurgencies, enabling non-state actors to project force in ways that were previously unimaginable.
Concerns on the horizon
There are a multitude of practical concerns of near-term importance.In some countries there is often a call for interpretable AI systems, but this isn’t a straight forward ask. Interpretability and explainability are active areas of exploration and there exists a current tradeoff between interpretability and effectiveness in military AI systems. There are significant cultural differences in how various nations and organizations prioritize explainability, with some placing a higher value on understanding the reasoning behind AI decisions while others focus primarily on raw performance. Finding the right balance between these two imperatives is a complex challenge that will require ongoing dialogue and experimentation. On one hand, the ability to interpret and audit AI systems is essential for maintaining accountability, building trust, and ensuring compliance with legal and ethical standards. On the other hand, the pursuit of interpretability may come at the cost of reduced effectiveness, as the most capable AI systems often rely on opaque and highly complex models. Navigating this tension will be a key task for policymakers and military leaders in the years ahead.
Another area that warrants deeper examination is the expanding attack surface created by the integration of AI into military systems. Beyond the high-level risks of cyber attacks, there are a range of specific vulnerabilities that could be exploited by adversaries, such as stealing model weights to replicate or manipulate AI systems, poisoning training data to introduce hidden biases or backdoors, and embedding "sleeper agent" attacks that lie dormant within models until triggered by specific inputs. As AI becomes more central to military operations, the consequences of these attacks could be catastrophic, potentially enabling adversaries to take control of critical systems or manipulate decision-making processes. Developing robust defenses against these threats will require a concerted effort across the military, intelligence, and research communities.
Finally, the role of human oversight in military AI systems is a critical issue that requires deeper engagement from policymakers, military leaders, and the research community. While most conversations about AI warfare highlight the military's emphasis on maintaining human control and the need for transparency in AI decision-making, translating these principles into practice is a complex undertaking. Designing effective human-in-the-loop systems that balance the speed and efficiency of AI with the wisdom and accountability of human judgment is a key challenge. This will require not only technical innovations in user interfaces and decision support tools but also the development of new training, protocols, and organizational structures to ensure that human operators are equipped to meaningfully interpret and oversee AI systems. Moreover, establishing clear frameworks for liability and accountability will be essential to maintain public trust and legitimacy in the use of AI in warfare. How the AI system is ultimately incorporated in the entire military approach is not a question about technology, but of governance.
The path forward for AI in warfare is not predetermined. Through proactive engagement and cooperation, policymakers, military leaders, and the AI research community can work together to steer the development of this transformative technology in a direction that prioritizes human agency, dignity, and the preservation of meaningful human control over life-and-death decisions. While competition among nations is likely to intensify in the years ahead, it is important to recognize that AI is a general-purpose tool that can also be harnessed to facilitate diplomacy, arms control verification, and conflict resolution. By investing in these applications and working to build a shared understanding of the risks and opportunities associated with military AI, we can chart a course toward a future in which this technology serves as a force for peace and stability rather than a catalyst for conflict and destruction.
Ian’s note after the conversation.
Since our conversation, there has been one editorial investigation that brings many of the issues discussed to the forefront. This note is not based on the Ai Salon conversation, but is relevant enough that I felt its inclusion important.
According to an investigation by +972 Magazine and Local Call , the Israeli military has developed and deployed an artificial intelligence system called "Lavender" during the current war in Gaza. This system is designed to identify and mark Palestinian individuals as potential targets for assassination, with little human oversight.
The key points regarding the use of AI and its impact on warfare are:
AI target generation: Lavender marked tens of thousands of Palestinians as suspected militants, including low-ranking ones, based on communication patterns and other data. This led to a massive expansion of targets for assassination, far beyond what human intelligence personnel could verify individually.
Failure modes: The sources reveal that Lavender had an error rate of around 10%, incorrectly marking civilians with loose connections to militants or no connection at all. This resulted in the bombing of entire families, including women and children, in their homes.
Minimal human oversight: Human personnel were largely reduced to "rubber stamps", only verifying that the AI-selected target was male before authorizing strikes. There was little to no independent verification of the underlying intelligence data or the machine's reasoning.
Permissive civilian casualty policy: The military loosened restrictions on permissible civilian casualties, authorizing the killing of up to 15-20 civilians for each junior militant marked by Lavender, and over 100 civilians for senior commanders.
Automation of collateral damage estimates: Inaccurate automated systems were used to estimate the number of civilians in targeted buildings, leading to significant miscalculations and unnecessary loss of civilian life.
Real-time verification failure: There were instances where targets had left their homes by the time of the strike, resulting in the bombing of entire families without hitting the intended individual.
Automated home tracking: The Israeli military used an automated software called "Where's Daddy?" to track the real-time locations of thousands of individuals marked by Lavender and alert officers when they entered their family homes, enabling airstrikes on residential buildings with entire families present.
Whether or not this report is true in all it claims, the systems it describes will undoubtedly exist and be used (if not already in Israel) and often far before they are as performant and robust as we’d like. Their use will also be paired with limited human oversight in the name of efficiency. But more important than the AI systems themselves are the policies we pursue that get operationalized either in military tactical practices or as AI systems. “Where’s Daddy” isn’t horrific because it is an automated system - it’s horrific because it explicitly targets families. “Lavendar” isn’t problematic because it is an AI system (though we’d hope it’d have higher accuracy!), but because of a permissive civilian casualty policy.
As this world of AI-powered warfare continues to develop, we must, of course, ensure that we have robust and controllable AI systems. Given that, how we use that power is the same issue as it has always been.
Notes from the conversation
The development and use of AI in warfare raises complex moral questions about responsibility and decision-making.
Robots and AI could potentially fight wars in the future with minimal human casualties, but this brings up issues of trusting machines to make kill decisions.
The US is likely to remain the dominant global superpower in the near term, with rivals like China, Russia and Iran posing more short-to-medium term threats.
AI is currently not as advanced in the defense space as many people think. There is still a lot of room for AI to revolutionize warfare.
Asymmetric warfare enabled by AI, such as with drones, could change military dynamics by empowering smaller forces.
Being physically detached from the results of warfare, such as with drones, can lead to moral disengagement similar to the Milgram experiment.
The US government may take over development of AGI for national security reasons to prevent other nations from gaining an advantage.
Interpretability and explainability of AI systems is an active area of research but there are still open questions about how explanations relate to actual decision-making.
Warfare is fundamentally about physically controlling territory and resources, so purely digital/simulated conflict cannot replace real-world military force.
AI could have a major impact on the support systems and logistics that enable warfare, not just combat itself.
Surveillance technology like satellite imagery is giving nations unprecedented visibility into each other's capabilities and resources.
Cultural values and historical trauma can lead to seemingly irrational military decisions by nations that an AI system may not account for.
Transparency of information between nations could promote peace in theory, but irrational human factors and the incentive to keep secrets limit this effect in practice.
Cyber-attacks are a major vulnerability for AI systems, especially as they become more widely deployed in critical infrastructure.
Demand for cybersecurity and AI talent currently far exceeds supply, which could impact national security.
Adversarial attacks exist for any AI system and keeping models closed-source may prevent discovering and patching vulnerabilities.
Interpretability may be at odds with security if an uninterpretable model is used to hide critical information.
Training AI to defeat other AI systems may give it capabilities that could be turned against its creators.
The phenomenon of model collapse in large AI systems trained on AI-generated data is a concerning potential future risk.
Data quality is the key barrier to developing powerful AI models more than compute resources or model architectures.
Questions
How can we best ensure meaningful human oversight of autonomous military AI systems?
Will cultural and values differences between nations result in divergent approaches to military AI?
How will the use of AI in warfare affect the likelihood and duration of conflicts?
Can simulation and war-gaming with AI lead nations to avoid conflict by predicting outcomes?
What are the most serious risks of AI systems causing unintended escalation or errors in warfare?
How can we defend against the use of AI for disinformation and psychological warfare?
Will public opinion turn against the use of AI in warfare or will it be seen as a necessary capability?
How much of an advantage will more advanced AI give a nation militarily versus other factors?
What new strategic or tactical possibilities will AI open up in warfare?
Will AI cause a shift to more unconventional or covert forms of conflict versus overt warfare?
How secure are current AI systems against hacking, data poisoning and adversarial attacks?
What international governance frameworks or agreements are needed for military uses of AI?
Could AI systems refuse orders they deem unethical, and how would this be viewed?
Will economic and political factors or technical issues be the main constraints on military AI?
How can human judgment and context understanding be meaningfully combined with AI capabilities?
To what extent can the behavior of a complex AI system be predicted and controlled?
How will advances in interpretability, robustness and value alignment impact AI in warfare?
What will determine which nations or groups gain significant advantages from military AI?
Are there any military applications of AI that should be categorically prohibited?
What milestones would indicate a nation has leapfrogged others to gain an overwhelming military AI advantage?