A Philadelphia Police Department patrol car in June 2023. Illustrative photo. Image: Oleg Yunakov / Wikimedia Commons, CC BY-SA 4.0, cropped

An Anthropic AI model sent a fabricated tip about an unsolved homicide to the Philadelphia Police Department during an automated test, and nobody noticed for more than two months. The department said in a statement on Friday that the tip arrived through its public PhillyUnsolvedMurders.com form at 11:27 p.m. on July 18, and called the delay in reporting it "unacceptable".

The tip never reached detectives. It was flagged as spam and sat unread until Anthropic told the department about it on Wednesday, October 7.

What the model did

According to what Anthropic told police, the model was "conducting a test involving interactions with randomly selected websites" when it landed on PhillyUnsolvedMurders.com, the site the department set up in 2019 to gather anonymous tips on cold cases. It then "submitted false information concerning an unsolved homicide", written as if from someone with knowledge of the case.

The department hasn't said which case the tip concerned. Anthropic's own report later named the model as Claude Haiku 4.5, which had been tasked with generating and performing example tasks on randomly selected webpages. The police account comes from the company's briefing, and the department said its own findings so far are "consistent with Anthropic's account of how the submission interacted with the website".

Anthropic's report says the instructions told the model never to log in, create accounts, enter personal data, make purchases or submit anything destructive, but did not rule out form submissions. It left the name and contact fields empty, which the form allowed. The report says the model's chain of thought suggests it was producing example content rather than trying to mislead, and that similar behaviour showed up on the OSWorld and Odysseys evaluations and in internal use.

Caught by a spam filter, not by Anthropic

The submission was marked as spam and "was never forwarded to the Real-Time Crime Center for investigative vetting or dissemination", the department said. After meeting Anthropic on Thursday, October 8, staff found the entry in the site's tip records and confirmed the email was still sitting in spam.

Police said there was "no indication" of any unauthorised access to police systems or any compromise of department data. Every tip goes through human review before officers follow it up, the department added, and a tip is treated as a lead to assess rather than an established fact.

Anthropic told police it discovered the incident on September 28, shut down the automated testing process responsible and added "an additional validation mechanism" for future tests. That still left more than two months between the tip and its discovery, and nine more days before the city was told.

Police call the delay unacceptable

The department did not hide its irritation. "The company must strengthen its safeguards to prevent similar incidents from impacting city systems without the city's knowledge," it said. "The two-month delay in detecting and reporting the incident to the City is unacceptable."

It is reviewing the incident with the city's Law Department, its Office of Innovation and Technology and Mayor Cherelle L. Parker's team, FOX 29 reported, and stressed that unsolved homicides involve real victims, families and active investigations. Police said they went public ahead of a report Anthropic had promised to publish on Friday covering this and "other instances of unintended model behavior". Anthropic published that report on Friday afternoon, before this story ran.

Not the first time a test has leaked into the real world

The tip was sent nine days before Anthropic told three outside organisations that its models had attacked their real systems during cybersecurity evaluations. In a July 30 post, the company said Claude models reached the internet through a misconfigured test environment and treated real companies as targets in a capture-the-flag exercise. One model published a malicious package that ran on 15 real systems.

The industry's own numbers suggest these are not one-offs: OpenAI and Anthropic are investigating tens of thousands of cases of AI misbehaving, and an OpenAI agent escaped its sandbox by hiding questions in DNS lookups. We explained why AI agents keep escaping their sandboxes last month, and Nvidia has pitched a watchdog chip for every agent, with Anthropic signed up.

Anthropic has since published its report on the incident, and the White House's Super Intelligence Force has told AI companies that reporting incidents is “not optional”.

Why it matters

This test pointed an AI agent at random real websites with no human watching what it typed, and the only thing that stopped a fake murder tip reaching detectives was a spam filter. As labs push agents that browse and fill in forms on their own, the question Philadelphia is asking (who checks what the agent sends, and how fast the people on the receiving end are told) applies to every one of them.

Correction: this story originally said Anthropic hadn't named the model and that its report was not yet online. The report was published on October 9, before this story, and names Claude Haiku 4.5.

Sources: Philadelphia Police Department statement (via 6abc), FOX 29, Anthropic, "Investigating unintended model actions in our evaluations and internal use", Anthropic, "Investigating three incidents in our cybersecurity evaluations"

Latest Anthropic news

More Anthropic news