TLDRUpFront: The scare story circulating that an Air Force simulation using AI drones resulted in a drone killing its own operator is false.
FullContextIntheBack: There was no simulation, either in a computational form or more common military BOGGSATTs*. It was a submitted “thought experiment” (source undisclosed) that hypothesized a series of events, which was then paraphrased in an anecdote by someone at a conference
What’s funny is the people who figured this out weren’t just dour-somber military or AGI types; one commentator posted: “I knew this story was (redacted) because imagine the military coming out and saying an expensive weapons system they’re working on sucks.”
I’ve also attached a great graphic that representationally conveys this discussion. AGI Existentialism and AGI Lethalities, in the chart combined as Imagined Harm “It will kill us all” (on the left), is a fear-mongering tactic to distract from real concerns over AI developments (on the right.) I don’t necessarily agree with the “bad actors” comment, however, as it implies intentionality. A lot of the AGI Doomerism is spread among well-intentioned and otherwise ethical people who have unfortunately become habituated over a lifetime of meme sharing to being key enablers of fear-mongering tactics. The chart also doesn’t present the benefits in relation to the harms. *Both* benefits and harm are “actually” happening and are all things. It’s a discussion of tradeoff consequences. But that discussion has to be informed.
*BOGSATTs = Bunch of Guys & Gals Sitting Around a Table Talking are military ‘simulation’ exercises that lie somewhere between Calvinball and what most people think of as “crunchy” board wargames with tons of mechanics. My experience is very few of these simulations are actually computational as we may think of them. This is okay – at times, the walkthroughs can be useful in understanding processes and procedures. But at worst, when trying to simulate uncertain encounters between truly adversarial opponents, charismatics have a stronger role than they would in a real conflict.