War Machines feat Stop Killer Robots and PAX

00:00.06
Gus
Welcome to The Tech Pill, a podcast that looks at how technology is reshaping our lives every day and exploring the different ways that governments and companies use tech to increase their power. My name is Gus Hosein and I'm the Executive Director at Privacy International.

00:14.17
Caitlin
And I'm Caitlin and I'm PI's Campaigns Coordinator. Hi. And this week we're joined by:

Professor Peter Asaro, Chair and Co-Founder of International Committee for Robot Arms Control and Chair of the Stop Killer Robots Campaign, which works to ensure human control and the use of force and calls for new international law on autonomy and weapon systems.

And by Frank Slijper, Arms Trade Project Lead at PAX, an organisation working on building inclusive peace, protecting civilians from the violence of war and ending armed conflict

to discuss military use of AI.

00:43.54
Caitlin
You may have seen the recent controversy in the news about Anthropic's battle with the US Department of Defense, now called the Department of War. And and that's why we've brought Peter and Frank here to talk about it. So shall we start with Peter? Do you mind explaining what on earth is going on with the DOW and Anthropic?

01:06.26
Peter
Technically still the Department of Defense, but if they want to identify as the Department of War, that's fine. um So it's been a sort of a strange saga, and it seems to be a mix of debates about the future of technology, but also kind of a battle of egos and and a bit of a media publicity.

01:28.26
Peter
slash intimidation sort of maneuver from the Pentagon. And as far as I've been able to piece together from journalistic reports, it appears that for some time, Anthropic has been the only large language model chatbot used by the Department of Defense on their classified data systems.

01:49.30
Peter
And that's been for a year and a half or two years, I think. um And that is utilized as a feature within the Palantir Maven system, which is a kind of all-source data gathering analytics type of system that the Pentagon uses for all sorts of intelligence planning and operational planning.

02:15.93
Peter
And it was reported somewhere in that um they had used AI in the Maduro raid in Venezuela. And it seems that an Anthropic employee asked a Palantir employee how it was used.

02:32.76
Peter
And this sort of set off a series of events. And it got to the Undersecretary of Defense, Emil Michael, who's Undersecretary for Research and Engineering and is basically in charge of all of the Pentagon contracts for technology.

02:50.01
Peter
And he felt that this questioning about how it was used was an attempt by Anthropic to somehow control how it was being used or tell the Department of Defense how to use it,

03:05.37
Peter
and took offense at that at some level. And this opened some discussions and part of contract and part of Anthropic's business practice is that they have these two ethical principles that their systems will not be used for autonomous weapons, which is an issue we care very much about.

03:28.76
Peter
And also that it will not be used for domestic mass surveillance of US citizens, which is in some ways a kind of arbitrary distinction, but under US law is a significant one.

03:41.82
Peter
And it's also important to note that the founders of Anthropic were formerly the founders of OpenAI, or one of them was a founder of OpenAI, the other one was their chief policy officer at OpenAI, which was founded, one, to be an open sort of software model for for AI, but also based on ethical principles to defend humanity from a possible super intelligence that would be unlocked through AI, and left the company to start Anthropic because the company was moving away from those ethical principles a few years ago.

04:18.58
Peter
So these are very important principles to the, to the founders and CEO - Amodei and...So they were very concerned about how the Pentagon was using this. The Pentagon was then became insistent that we can use it for any lawful purpose, and we want you to sign a new agreement that you you know will not hold us to your ethical principles or something like that. And and they refused to back down from their ethical principles.

04:49.85
Peter
And that sort of resulted in a very kind of public um sort of battle between the two and a sort of battle of wills. And uh then it was more threats that if they wouldn't do this not only would they lose the contract, which is you know a significant but not ah a major you know business really... So the way these contracts are structured they have a sort of a minimum and a maximum and so this is a two to two hundred million dollar contract so I think they were actually only paid two million dollars but

05:24.47
Peter
the way contracts work if they wouldn't have to negotiate a new contract or seek additional appropriations unless it goes over 200 million. But if you look at Anthropic their annual revenue is like 18 billion, so a two million dollar contract isn't gonna you know break the company in any sense um but it's you know it's significant business. So the Pentagon kind of raised the stakes by then saying, well, we're going to label you a supply chain risk.

05:57.82
Peter
And this is a fairly new kind of designation that emerged in recent times, mostly I think over the introduction of Chinese chips and other electrical components into things like drones and satellites and concerns that they could have Trojan horses sort of built into the hardware and things like that.

06:19.86
Peter
And it's only ever been used against foreign country companies and actually only against foreign companies that are significantly controlled by their governments and other adversarial governments. So it's only been used really against Chinese and Russian companies previously.

06:36.09
Peter
So the idea of using it against a US company is is quite novel. And then it's also ah potentially has much more larger implications, which is that no Pentagon contractor can use any software provided by Anthropic.

06:55.48
Peter
And I think it falsely claimed, because it's not even in the statute that they're trying to invoke, but I think Hegseth has said that it would result in all government agencies having to stop using Anthropic, which is not the way the thing works. That's not how... *laughs*.

Gus
Oh, okay. I didn't know that. Okay. And that's a much larger number.

07:18.17
Peter
Yeah. And that would be a lot more business. And then also you have to sort of think, well, in terms of your kind of strategic competition, Anthropic is received as a sort of better and more reliable large language model, and also why it was able to certify to be used on classified systems earlier. There are then now contracts with OpenAI, which were signed at the same time that they were sort of trying to negotiate this deal.

07:49.21
Peter
But there was a previously signed contract with XAI, which is Elon Musk's AI company, to also be operational on the classified systems. But that's going to take each of those companies six months to a year or two to achieve all of the kind of security protocols that they need to be operational there. So, even though they're claiming that it's a national security risk that Anthropic's system is being used, they're also planning, depending on the timing, to continue using it for the next year, regardless, or six months, I think, to a year to phase it out.

08:30.01
Peter
So it must not be that much of a national security risk. And that's pretty much what the legal case that now Anthropic has sort of sued the government over this for and they've also received then a sort of amicus brief from Microsoft supporting that lawsuit for a number of different reasons but basically saying that you know how can there be a security risk if you're continuing to use it and actually relying on it for a lot of the operations in Iran, which I think we'll get into later this podcast

09:06.71
Gus
Yeah, if I could just riff for a second, what I find fascinating about how this story has unfolded, just from a slightly nerdy perspective, is that usually when a story has a single angle to it, it's easy to follow. But this story had two angles, which was the the the the mass surveillance angle and the autonomous weapons angle.

09:28.22
Gus
And what's fascinating about the coverage and the discussions is that they have covered... like it's become two huge discussion points. Should the Department of Defense be conducting mass surveillance? What is lawful mass surveillance? And finally, some scrutiny going on there that opened up questions as to what data was it? And then all of a sudden, because of leaked memos, we now understand that the Department of Defense slash Department of War is by buying vast amounts of data from data brokers. And so it's not like they're conducting communications surveillance, they're just buying this data from third parties. And that was part of the the disagreement. And then on the other side around autonomous weapons, the coverage has been so...

10:15.96
Gus
rich. And like I've seen the term kill chain in my daily news reading more in the last two months than I have in the last five years. And it's extraordinary that these, both these issues, have managed to stay front and center to this really fascinating case.

10:35.99
Caitlin
You talked about like moral red lines and but is Anthropic's ah kind of objection ethical or is their objection around the capacity of what they believe that Claude and their large language module is able to do? Because I've seen different coverage saying either we don't think it's capable of doing this versus we think it's unethical that it do this. And those are obviously two different a conversations. One is potentially arguably overcome-able and what is not.

11:09.46
Peter
Yeah, I mean, I think it would probably vary depending on which specific person you're talking to within the company even. And that's one of the actual problems that I have with ethical principles as a means of governing technology is they're subject to different kinds of interpretation.

11:27.06
Peter
Legal structures are much stronger in that respect. And while they are subject to interpretation, you also have a system of courts and law to adjudicate those sorts of disagreements. And that's not really the case with ethical principles, but I think they definitely believe these ethical principles and believe that they are ethical. It seems that the criteria by which they measure are somewhat different or the basis for the principle, right? So what what I've heard, and I think this is that what you're getting to, is that they believe at this point in time,

12:02.81
Peter
the AI is not sufficiently capable to be you know responsible for making autonomous targeting decisions. I've argued that ethically they are incapable of it because of the nature of their being machines and software, however sophisticated and sort of technically capable they may be, they're not legal or moral agents. And so they are never going to be responsible, legal, moral agents, even though they can effectively make decisions. Those decisions are not ...have no sort of basis and responsibility.

12:38.30
Peter
Um, so they think though, potentially that these systems will be capable at some point in the future, but they're just not reliable enough right now. So I think, you know, probably some people or most of the people believe that.

12:54.84
Peter
Whether or not they would be capable or what the criteria would be to evaluate whether they know we we should let them do it because they're really good at it or something, we ultimately have been arguing for human control and meaningful human control over all of these things and that that should be codified in law. And that keeps it also to being a kind of a human enterprise, right? Like once you pass off this kind of responsibility to machines, like machines can decide to go to war or escalate conflicts and make all kinds of decisions that, you know, what is the criteria for "that's a good decision"? It's an effective military strike, but is that achieving your political goal?

13:37.75
Peter
Is that achieving, you know, a de-escalation or ah a resolution to a conflict at some point in the future? Probably not, right? And these systems aren't capable of understanding any of that. They don't really have any sort of desire or goals of their own, which is actually kind of, I think, the underlying fear from a lot of the people who work in this space, OpenAI and Anthropic and other AI companies, is that, you know, they're building some kind of godlike super-intelligence that we need to sort of actually defend ourselves against at a certain level.

14:12.47
Peter
Right? And so giving them the sort of the buttons to all the weapons and control over all the weapons systems is also sort of scary moral proposition for them,because they're - you know - they fear that. I'm not particularly afraid of that in the near future, if maybe it's possible in the long term.

14:32.44
Peter
And right now, they're they're still worried that these systems aren't good enough to even reliably control a single strike, much less you know take over the world or something. So there's a lot of sort of hype around you know the fear, but there's also genuine fears that we've had for quite a long time at the campaign.

14:51.77
Caitlin
I mean, going beyond Anthropic and this particular kind of discussion, Frank, you've been looking a lot, I think, at um all of the kind of broader ah companies and systems that have been ah making their inroads into military contracts and military weapons. Like, um, how what what have what have you been seeing, I guess, in the broader space?

15:19.70
Frank
I think what um like but but you can see is that a lot of the tech companies, and I would say especially the legacy tech companies, the IBMs, Intel, Hewlett Packard, AMD, they've been working with the Pentagon for decades, um but mostly on a much smaller scale, for mostly rather specialist purposes.

15:51.64
Frank
For example, supercomputers doing the work of simulating nuclear tests. And, um, so those companies have been working on these side of sort of contracts for a long time.

16:09.46
Frank
And as we know, with especially Project Maven and all the the discussions and controversies that burst out in 2018 around that. That was sort of the start with the tech companies, and then mostly the newer tech companies, the Googles, the Microsoft, and and the development of their ethical principles.

16:43.99
Frank
And that sort of generated much more reluctance to get involved in military contracts. And it was a bit quiet for some time.

16:57.88
Frank
But at around the same time of of Project Maven, there was also a a big contract pending um for for offering cloud services to the Pentagon.

17:13.14
Frank
And that contract was first um awarded, but then all sorts of of legal battles and then it was repurposed and in late 2000, what was it, 2022, it was awarded to Microsoft, Oracle, Amazon and and Google.

Gus
Like they all got it?

17:38.84
Speaker
They... they... It was a combined contract, $9 billion dollars, and and I think that was one of the most significant contracts until then for the tech companies.

17:52.92
Franks
And it's not just offering cloud services. It was dedicated... It's called the project Joint Warfighting Cloud Capability. So the link with with the war is is clear there.

18:07.51
Frank
And I think that um Amazon never had any problems with the military ah use of their products. Oracle also not. I think for Microsoft and Google it's been different.

18:25.53
Frank
Also um because of internal resistance, there was much more backlash there, um in all discussions about responsible use of of AI.

18:40.63
Frank
But then um I think with the war in Ukraine, with the invasion of of Ukraine by Russia and increased military budgets and um the emergence of the large language models, um all of that has has contributed to, and last but not least, the big US budget - military budget - has contributed to the tech companies becoming much more open and and eager to fight for these military contracts.

19:21.43
Frank
And I think also at the same time, you see um partly public perceptions of these sort of things changing, like we live in a different world. It's also the sort of rhetoric that Microsoft and Anthropic use - like we need to use AI to to strengthen democracies and to to make sure that the bad countries will not become better in AI than we are.

19:54.01
Frank
And some, I think, there you've seen quite a significant shift over the past, let's say, four or five years.

Gus
That timeline you paint is spot on. And just to to riff off your point about how these companies are now talking about defense of of democracies. Palantir doesn't talk about defense of democracies. They talk about the defense of the West. And so very it's a very different mindset. and it's all you can... And what I found fascinating about the timeline is...

20:25.05
Gus
While these contracts were negotiated originally in 2018 and 2021, it was in 2022 we had the large language model moment. And the reason we paid attention to that was because we saw that the UK government equivalent of the National Security Agency, being GCHQ, they had negotiated a contract with Amazon in 2021.

20:50.01
Gus
And at the time in our imaginations, that was just, oh, for storage of vast amounts of data and a little bit of processing. But by 2022 and 2023, all of a sudden Amazon's offerings changed just a little bit. And they probably offer large language model processing.

21:08.09
Gus
And cloud compute, which is a completely different offering than the contract would have been in 2021, when arguably there might have been some oversight into that decision. And so I think similarly, oh, we'll use Claude in our government processing of data in the Department of Defense. Then all of a sudden it's like, oh, but we can use Claude for more things now. And it's that constant shifting in what the capacities are of these systems. And yet these contracts remain the same. It's just fascinating.

21:39.26
Frank
Yeah, and and maybe also going back a bit to to what what you previously discussed. Yes, Anthropic doesn't want their their technologies to be used for internal surveillance and for lethal autonomous weapons.

21:55.96
Frank
But the very simple fact that their large language models are being used for targeting Venezuela, and for the whole war in Iran now, ah that apparently is less of a concern. And and if you relate that to to that discussion of the West that should be able to lead in the AI revolution,

22:19.29
Frank
I think that also nuances a bit the the public image maybe that Anthropic now has gotten because of its resistance, which I think is very good against the Pentagon.

22:31.54
Frank
But at the same time, they have not resisted, as far as I know, the use of their technology in these illegal wars.

22:42.94
Caitlin
Which actually brings me to, and and I don't know if it's, shout if it's not fair to ask this because it literally happened today, but I saw that um someone else submitted a new amicus to the supply chain case which said essentially, and it was on neither particular side, but said essentially that Anthropic red lines are whatever, um but there is existing international law, war crimes are an existing thing, and whether or not it's used in autonomous weapons, um Anthropic can't get round...

23:17.78
Caitlin
...and the Department of Defense can't get around the use of these AI technologies in potential war crimes and the existing criminal kind of international criminal situation. That might be a horrific description, by the way, of what the amicus actually said. um But...

23:34.10
Gus
No, I read it. It said pretty much that.

Caitlin
Okay, and...

Peter
Who was it from?

Gus
ah it was from Tech Justice League and Access Now and one other organization.

23:49.82
Caitlin
But I suppose my question from that is, um like, how are LLMs currently being used and what is the substantive difference between how they're currently being used for targeting decisions versus autonomous decision making or autonomous weaponry? Like, what is that gap that Anthropic doesn't want to be bridged?

24:10.81
Peter
Yeah, I mean, again, it's it's difficult to know exactly where they're drawing lines. And this was even an issue when the 2018 Project Maven - they added an ethical principle about not using or not developing AI for weapons systems within Google.

24:32.57
Peter
And we all asked, well, what about Project Maven? Is that a weapon system? And it's, you know, a system that was designed to identify... ah basically to process UAV drone footage coming out of Iraq and Afghanistan, looking for things of interest, right? Which is essentially target profiling and development.

24:54.49
Peter
And that's, I think, kind of how and what we're seeing the the use of these chatbots for and LLMs within within these defense systems. And we've seen a lot of reporting from Israel and Gaza over the use of Lavender and Gospel and Where's Daddy. and these different Each system is a little bit different. It's designed for a specific kind of data set and a specific function, but essentially to identify members of Hamas, to identify their locations and patterns of movement, and then to use that information to develop targeting lists for where to find them. Where's daddy is to try to find out where they sleep at night and target them there.

25:37.37
Peter
And all of that sort of being integrated into the kill chain that we're hearing so much about to speed up this process, automate this process, allow them to use the mass surveillance data that they collected. So, while I appreciate that there's still this two you know sides of this issue, mass surveillance and autonomous weapons, they're clearly being integrated in these systems. And it's almost hard to draw a line around them, because you're using, you know, security camera footage, satellite footage, phone call data, potentially, you know, health, business, tax records, whatever, to try to identify who people are, what are their systems of integration, and so are they a member of an enemy organisation, and then use that information to also, the same body of information to also identify their location and drop a bomb on them.

26:30.10
Peter
Right? And that sort of going to a human at a certain point who says, is this an acceptable target or you you know for human approval? And even when I spoke to the UN the first time in 2014 on this issue, I called this meaningless human control. Where you just have a system that produces a target for you, you know the light comes on, and you have a choice to press the button or not press the button.

26:55.35
Peter
But you don't really have the opportunity to delve into the context and the situation, and whether that killing is justified, whether the evidence supports that action, what the implications of that are militarily, strategically, politically.

27:11.54
Peter
You just don't know. You're just like, well, the computer says this, and I'm going to approve that. And what we heard about Lavender was, you know, all the... the only check humans were making was, is the person who's been identified male or female because Hamas doesn't have female soldiers and Israel does. And so if it was female they would reject those but any male target was accepted. They knew also going in, just on test data, that they'd used before the conflict: like 10% error rate. So, you know, 10% of the targets - and that was prepared clean data for testing - so what the actual error rates might have been and then you get into questions of proportionality and collateral damage and when you're doing again, this number of strikes because you're automating the process in order to generate more and more targets.

28:03.32
Peter
And that means less and less ability for individual humans to evaluate each target and determine its validity. But it also means more and more bombing, which means even if you have small ratios of civilian casualties, by having more targets, you still have a lot more civilian casualties. And I think that's one of the real sort of worries about this is the intensification of warfare, both in terms of speed and in terms of destruction. All with this sort of plausible deniability of, well, we have selected these targets and these are... computer says, these are lawful targets. So you know are we're covered from a war crimes tribunal perspective, which I don't think they are but it's enough for them to effectuate that. And then it's very hard to to prosecute those as crimes anyway.

28:56.82
Gus
Yeah, if I could segue between the two of you on that, because um I remember a podcast done with, not by us, but by Lex Fridman with Marc Andreessen back in 2023. Marc Andreessen being Andreessen Horowitz. He's the creator of one of the first web browsers and now investment like investor extraordinaire and AI fanboy beyond belief. And in 2023, he's arguing that he would want to see drones, only automated drones doing

29:38.81
Gus
doing killing. He says humans make mistakes all the time. It's so much better to have an automated weapon rather than a human. And this is a guy who for the last three years has essentially been helping to build this industry that is now the new defense tech industry. And going back to something you were saying before, Frank, about how it is the entry of these new tech companies, like the big tech companies. And it it feels like what we saw with this moment with Anthropic and say with Microsoft back in September when it came to Israel and in the Ministry of Defense is that it was almost like a naïveté, like the these...

30:20.95
Gus
It's these these young children thinking all these toys can only ever be used for good. And of course the data is clean. And of course we'd only kill bad people. And of course they're only gonna use our our systems for the reasons in the contract. But I wonder if the older companies you were mentioning, like the IBMs and IBM famously that played a role in the Holocaust, um do do these older companies, are they a little bit more wise to all of these? And is that maybe why they don't have ethical principles because they're not trying to sell themselves as such.

30:53.05
Frank
I think it's also because they're they're offering different technology, different work, and it's mostly the new technologies that play a role here with the Palantir and and its database analysis, which of course also Oracle traditionally very much was, and I think is slowly adapting and trying to catch up, but basically having lost a lot of territory in the military domain to mainly Palantir.

31:27.10
Frank
And so it's, I think that's actually also an important aspect that a lot of the companies now really in the news and and in the business are basically companies that didn't exist 25 years ago.

31:44.12
Frank
And I mean, if you look at Anthropic, just just four five years old, OpenAI 10 years old, then Google Alphabet maybe being one of the older ones. What is it? 25, 30 years old, and Amazon as well. But having also shifted enormously from being firstly and ah foremost consumer companies offering a marketplace, offering a web browser and

32:16.57
Frank
having everything changed so much into that military work that for very long, I think also from a commercial perspective, wasn't so interesting for them.

32:29.24
Frank
And yeah, I think that that is a big shift. And also if you see how the the the interdependencies are between these companies with Microsoft, of course, a lot involved in Anthropic and OpenAI and NVIDIA, which out of, I mean, almost nowhere. I mean, it's a company 30 years old, but how it has become the the world's biggest company in terms of market capitalization,

33:00.09
Frank
a company that most of us had never heard of 10 years ago and is now central again to these large language model companies that work together and make very much use of each other and invest in each other.

33:20.09
Frank
And I think that's also a very new dynamic that does not apply so much to the the legacy tech companies.

Peter
I mean, it's also interesting that, you know, there was that movement kind of in the late teens, pre-COVID, to have AI ethics principles and AI ethics teams.

33:38.42
Peter
And all of those were dissolved over the last few years. And I think most recently, 2025, January, we saw you know Microsoft and Google get rid of their remaining AI ethics principles. And, whether that was an anticipation of the new administration or just the shifts around AI and its many, many applications, um it's both concerning that they you know they were concerned about it at one point and then they sort of abandoned it. And the the Pentagon did the same thing. So there were AI ethics principles for the military use of AI, there was AI safety principles for government use more generally.

34:16.41
Peter
Some of that was executive order, some of that was internal policy, but through executive order, Trump abolished all of that on the first few days of his administration.

Frank
Do you think that the AI ethics principles that were there were meaningful? In the sense that, like, um were they there so that they could be seen to be there, or were they there because they really were thinking about them and were concerned about the ethical principles involved?

34:46.01
Peter
I think there was a fair bit of ethics washing is what we're calling it now. Avoiding regulation, real regulation by adopting these sort of self-governed principles.

34:56.89
Peter
I think it was also, as we talked earlier, you know ,appeasing dissent from within the companies, from their programmers and engineers - who really didn't you know want to be involved in those sorts of things and trying to say, no, no, it's all going to be okay.

35:15.00
Peter
But then also to avoid any kind of government regulation of these new technologies. And I think after you know you sort of have the post-GPT3 moment, when the world sort of figures out, oh this is what AI is and how it's going to work. It wasn't a huge technical leap, right? It was just an interface leap that made it accessible to a huge swath of the population very, very rapidly, but that changed our perception of it and dramatically.

35:44.31
Peter
And I think after that, the implications for many kinds of applications, the the vast amount of money that was going to be made off of it has just been accelerating really since then. But, um, that's the moment they decide to sort of back away from a lot of the ethics as a sort of, well, that's going to hold it back somehow.

36:06.51
Peter
And what we need is this unconstrained development and, some sort of gesturing at least at a sort of arms race with China or with other competitors who are going to also be trying to build AI. And we have to kind of capture a market share and, in the domain of the military, you know, capture this sort of new technology that is going to give us this huge military advantage.

36:31.90
Peter
And I think both of those sorts of fears and arms races are more about hype and than reality. But there is certainly a lot of money being made in all of this.

36:45.33
Gus
It's and you hinted at it right there as well, Peter. But what also happened in that very short window of time, say, 2003, 2023 to now, is that the tech companies started laying off tech staff.

37:00.82
Gus
It was, you know, you can almost see these ethical principles as recruitment, um advertising to say, hey, we're the good guys, come work for us. And then they realized they had enough people. And then with the change of the world and the talk of the rise of the West or defense of the West, they thought that they didn't need to recruit people based on ethical behavior anymore and that this was a new era.

37:24.73
Peter
Yeah. And this was identified by the Defense Technology Unit. So, when Eric Schmidt was CEO of Google, he started talking to the Pentagon and you know they had all this trouble with Maven and things. And when he left, he became a sort of advisor and really sort of gave himself the personal mission of bridging the world of Silicon Valley and the Pentagon. And at that time, there was this huge shortage of tech labor and specifically in the AI space and there were these astronomical salaries that you could get for working in AI, which is still the case in some of these small companies.

38:03.93
Peter
And there's a kind of proliferation of AI-related jobs like prompt engineering and things like that. But as you say, like programming jobs are are drying up because now Claude can program as well as most entry-level programmers.

38:19.06
Peter
So there is that part of it. But yeah, there was a recruitment issue. And that was actually what they identified 2018 to 2020 era as the big impediment to developing military AI because the engineers didn't want to work on it.

38:33.56
Peter
And that's actually, I think, maybe behind the scenes that we're not hearing about um right now is: is Claude and Claude code development being used to develop autonomous weapons?

38:47.42
Peter
Because I would think that would violate their principles, but nobody's really discussing that, and of course the military itself doesn't do development, but they invest in these companies that are doing development.

38:58.78
Peter
And are those companies, I'm sure those companies are using various AI coding schemes. So, I think that's a whole nother angle to worry about.

Caitlin
Can I ask kind of, and this might be a bit of an entry-level question, but to both of you: if you're these very new companies that are getting involved in the military space... obviously the military space is one that is inherently extremely high risk whether you're a soldier whether you're a civilian whether you're you know within 100 miles of

39:32.09
Caitlin
a war zone, it's an incredibly high risk kind of place to be, and a situation to be in. And these companies are incredibly new. And it might be, I'm quite naive. But when it comes to the new companies versus the kind of old traditional military equipment companies, yeah are they're not like, just military risks? Or are they're not kind of...

39:59.83
Caitlin
If you were a soldier who's you know in one of these war zones, would you not feel a bit not great about you know a company that's five years old that's suddenly being deployed in a war zone? I just It's that aspect of it.

40:15.90
Caitlin
And maybe I'm wrong that there are kind of higher levels of... process and safeguarding and review and consideration for failure and all these other things in the traditional companies. But, um you know, Claude code is quite good, but it's not that good. And like most people interact with LLMs now and they know they're, you know, seemingly surface level quite good, but they're not that good.

40:47.13
Caitlin
I don't know. And maybe that's partly because they're more complicated technology than, you know, a gun, but there's something there that freaks me out.

40:59.26
Frank
I think, I mean, we we still don't know to what extent Claude or any other large language model has been responsible or involved in that bombing of that girls school in Iran.

41:15.42
Frank
But obviously, I think anybody, and especially soldiers, should be very wary of of the the newness of these technologies and the risk that they bring in inaccuracies, in all sorts of biases built in the systems.

41:39.10
Frank
So I think, and that's also what we saw with the enforced introduction of AI in the Pentagon with these posters of Hegseth saying: I want you to use AI.

41:54.17
Speaker
A little bit silly, I can as well. But also what you hear a lot of the people at the Pentagon, and probably as well in the field don't really trust the technology as it is now. So I think that is that is the risk indeed. And I think that's also what you hear a lot, for example, about Palantir.

42:18.46
Frank
There's a lot of bragging and that's bringing a lot of investments. But it's still, I think, not completely clear what the benefits are that a company as Palantir is really bringing. And especially if you look at the consequences on the ground and beyond Iran, looking at at Gaza and how all these Where's Daddy, Gospel, Lavender have contributed to the mass destruction and the genocide in Gaza.

42:58.39
Frank
I think that should definitely give reason to a serious pause to how we introduce these decision support systems in new... in our current warfare.

Gus
But what we need is is is that moment of investigation. Like um in my PhD, remember in the research phase, one of the articles - the book chapters - that really just opened my eye to this entire domain was the case study of the M16 being used in Vietnam War. And, you know, the creation of that gun was, it was a political artifact. It was forcing together old manufacturers and new bullets. And in the end, it just didn't work. And the only people who knew it didn't work were the soldiers who couldn't... who were having misfires all the time or it wasn't working in the field. And it was only after it all,

43:53.72
Gus
when Congress did its investigation, did all of this finally come out. And it's just like, why do we have to wait until then? But that was the 1970s and that was after the Vietnam War. are we going to have this kind of moment at some point to look at what happened in Gaza, to actually understand the role of the decision support systems and the data going into it? I somehow doubt it. And so, but that's not, I don't want to end us on a negative note, because one of the reasons I was most excited that you both joined was because I know that you're both pushing for actual tangible change.

44:30.17
Gus
So like Frank, you said you just were at negotiations last week. And Peter, you and I first met at the UN in New York where you're part of, or Stop Killer Robots, is pushing convention change. And I was wondering if you could, on the back of this conversation, if you could say what it is that you're seeing where you think there there's an opportunity for real change, for - if it's not going to be a reckoning, at least a change going forward.

45:00.95
Peter
Yeah, I would just say in terms of the campaign, and we've been working since 2013 to get an international treaty, but it's really ultimately about a norm. and the norm is meaningful human control of military force, lethal force.

45:16.76
Peter
And it's you know I think it's it's deep, but it's also broad and it's also clear. And it functions a lot like other legal principles that you have to you know teach the soldiers of basic training, right? Like, you don't execute prisoners of war and you don't kill civilians. It's just a clear rule, that if you're using force that there's meaningful human control. How you actually interpret that in specific cases can get very complicated potentially, but as long as we have a globally shared norm and we put that in the law, I think that's a really good step. I think it gets a lot harder to come up with a norm that is that clear when it comes to mass surveillance.

45:59.06
Peter
You know Data collection can be used for lots of different things. Some of those could be good things, right? Many of them could be good things, but they can also be used for you know authoritarian regimes to be very impressive. They can be used to target and kill people who dissent.

46:15.29
Peter
They can be used in a variety of ways that we can't even imagine, right? And then you start to look at other ways in which these sorts of technologies can, you know, come up with new chemical weapons designs or, you know, lower the barriers to entry and costs for fringe extremist groups to develop weapons and weapons of mass destruction. You look at ,you know, propaganda and the the crisis of epistemology and how do we control information or understand like what is you know valid information what agents and what you know computer systems have the ability to sway public opinion and transform you know votes and democracies and all of these sorts of threats that are posed by this one technology, right which is sort of so many different things.

47:04.41
Peter
But I think there are ways to draw lines in there. I think a lot of it derives from you know private data and securing our data from these kinds of systems and understanding how these systems work is part of that and the risks that are being posed to civilians is a big part of that. But we also, I think more importantly, have to organize as civil society and to do it globally. Because if we get it in just one country and or even the whole EU, right that's not enough to to change what's happening in Silicon Valley or what's happening in China.

47:38.55
Peter
And right now, those are you know the leading sort of exporters of this technology on a global level, but know that could also shift, and we really need a kind of global cooperation and understanding and civil society to be engaged on this.

47:55.31
Caitlin
I think that's a really nice place to leave it. So thank you both so much for your time and for joining us. It's been an absolute pleasure to talk to both of you and we really appreciate your time. And unfortunately, I imagine this will keep coming up. And so we will probably ask you back in no short order.

Peter
It's been a pleasure and I'm happy to come back.

48:14.42
Caitlin
Thanks for listening. You can sign up to be the first to learn more about our work at pvcy.org/podsignup. going to include some links to relevant articles and information in the description wherever you're listening or on our website at pvcy.org/techpill.

Don't forget to rate and subscribe to the podcast on whichever platform you use.

48:32.31
Caitlin
Music courtesy of Sepia.