On March 27, OpenAI, the creator of ChatGPT, signed a contract with the Pentagon (the deal was later amended on March 3 to prohibit the use of its technology to spy on Americans) for OpenAI’s models to be used for classified military operations.

For many people, myself included, this deal seemed almost inevitable. Artificial Intelligence (AI) use has grown commonplace. From the circulation of AI-generated videos to the growing conversation of the impact of AI on the job market, AI has entered all kinds of industries and changed the way that the internet and the world works. But despite all this, AI in the military is a significant step. 

Now, there are many arguments for integrating AI systems into the U.S. military. AI can give the United States a competitive advantage against adversaries, AI can save the lives of U.S. service members and AI can make decisions faster. 

But there are risks to integrating AI into the military, including AI sycophancy, a tendency to escalate conflict and the consequences of incorrect information. 

AI models are notoriously sycophantic, harboring a tendency to agree with whoever is asking the question. If military officials are asking an AI for advice, the AI model may focus more on telling the user what to hear rather than the best course of action. 

While the AI systems that the U.S. military implements are likely less sycophantic than your everyday ChatGPT, the fact that these habits are baked into this technology poses great risks for security. Instead of offering helpful advice or bringing up weaknesses, the AI model could instead blindly agree with the user and push the user towards a worse outcome.

AI models are also proven to escalate in tense situations. In a Kings College study, researchers tested how AI Large Language Models (LLMS) would react in a simulated nuclear crisis. The three models—GPT-5.2, Claude Sonnet 4 and Gemini 3 Flash—engaged in tactical use of nuclear weapons in 95% of scenarios and not one of the models ever chose accommodation or surrender.

We are far from putting LLMS in control of nuclear weapons, but this study makes it appear like AI models are more concerned with “winning” than with preventing the loss of human life. 

But AI models also have unique weaknesses; they are restricted to the information provided by the open web and/or information fed to them by users. 

On Feb. 28, a girl’s elementary school in Iran was bombed, and although an investigation is ongoing, it is likely that the United States was responsible. Officials have stated that this mistake was not due to AI, but rather outdated information since the school was on property that was used to be part of an Islamic Revolutionary Guards Navy base. Investigators do not yet understand how the outdated data was sent to Central Command and how the mistake happened.

While this tragedy was not a result of AI, it has implications for the use of AI in warfare. If AI is mistakenly fed incorrect or outdated information, the consequences could be devastating. 

Which brings me to my final point: Who is to blame? What happens when an AI breaks the rules of war? Do we blame the country responsible for the AI system? Are they responsible for allowing the AI to operate without human intervention? Do we blame the AI system? What does accountability look like?

I don’t have the answers to any of these questions. All I know is that there are risks to integrating AI into the military, risks that the American public and the world may not be ready to face.

The views in this column do not necessarily reflect the views of the HiLite staff. Reach Charlotte Moser at [email protected].