AI companies have lowered the bar for passing your chats to the police

On 26 September a 30-year-old woman in Bonita Springs, Florida, wrote in Claude that she was going to “shoot up” the Lee County Sheriff’s Office. The next day she wrote that she had got a new gun. According to the arrest report, as described by the local station WINK, Anthropic’s safety systems flagged the messages and, because of their severity, passed them to a human review team, which reported them to police. When deputies found her, she said she used Claude as a diary. She is charged with making a written threat of violence, a second-degree felony in Florida.

The Florida law forbids sending written threats to kill or to carry out a shooting “in any manner in which it may be viewed by another person”. In this case, the other person who viewed it was an Anthropic reviewer.

It was not the first time Anthropic had called the police. According to the San Francisco Standard, in August a user wrote in Claude that he had bought an AR-15 rifle and had chief executive Dario Amodei in his sights, and the company reported it to San Francisco police. The user later said he had been joking and had not been arrested when the story ran. An Anthropic spokesperson called it “our safeguards process working as intended”.

Eight months earlier, OpenAI decided not to report

On 10 February 2026, in the small town of Tumbler Ridge in British Columbia, an 18-year-old killed two people at home and then opened fire at the secondary school, killing five students aged 11 to 13 and one educator and wounding 27 people.

OpenAI had banned the shooter’s ChatGPT account in June 2025. In a letter to Canada’s AI minister on 26 February, it said an automated system had found the account and sent it for human review, and that “we did not identify credible and imminent planning that met our threshold to refer the matter to law enforcement”. Lawyers for victims’ families say 12 OpenAI employees urged the company to tell the police. Chief executive Sam Altman apologised in a local newspaper in April: “I am deeply sorry that we did not alert law enforcement to the account that was banned in June.” In September the province sued OpenAI and Altman in federal court in San Francisco, seeking damages and an order requiring the company to change how it detects and responds to conversations involving threats of violence.

Three versions of OpenAI’s rule in a year

Put OpenAI’s three public statements side by side and you can see which way the line has moved:

• August 2025, blog post “Helping people when they need it most”: if reviewers find an imminent threat of serious physical harm to others, “we may refer it to law enforcement”; self-harm cases are “currently not” referred, to respect the private nature of the conversations.
• 26 February 2026, letter to Canada’s AI minister: the criteria are “more flexible”, because a user may not discuss the target, means and timing of planned violence yet there may still be a risk of imminent violence; under the new protocol, “we would refer the account banned in June 2025 to law enforcement if it were discovered today”.
• 28 April 2026, “Our commitment to community safety”: “When conversations indicate an imminent and credible risk of harm to others, we notify law enforcement.”

From “may refer” to “we notify”, and from needing a concrete plan to not needing a stated target or time, the bar has clearly come down. The April document no longer repeats the line about not referring self-harm, and does not say whether that has changed; it mentions only that parents of teenagers may be alerted in rare cases and that a “trusted contact” feature is coming.

The rules in each company’s own words

Company	When it says it will hand over user information on its own
OpenAI	A conversation indicates an imminent and credible risk of harm to others (“Our commitment to community safety”, April)
Anthropic	A good-faith belief that disclosure is reasonably necessary to “prevent serious harm to any person or to property” (privacy policy); when police ask for data, it generally requires a subpoena or warrant, with an exception for emergencies, and usually notifies the user (government requests policy, 16 March version)
Doubao (ByteDance)	Personal information may be provided without consent where “directly related to criminal investigation, prosecution, trial and enforcement of judgments”, or in an emergency to protect life and property (privacy policy, section 3.4, in force from 17 September)

In China there is also a legal duty. Article 14 of the Interim Measures for Generative AI Services says that when a provider finds a user using the service for illegal activities, it shall warn, restrict, suspend or end the service, “keep relevant records, and report to the relevant authorities”. The two US companies write about violent threats against others, and about what they “may” or “will” do after their own assessment; the Chinese rule covers any illegal activity, and says “shall”.

In its letter to Canada, OpenAI promised transparency about its referrals to law enforcement. As of 25 October we could not find a count of referrals on its website.

Once flagged, someone reads it

The processes OpenAI and Anthropic describe look much alike: automated systems screen first, a person reads what gets flagged, and a person decides whether to ban the account or call the police. The Florida user saw her writing as a diary; in Anthropic’s process it was a flagged threat.

Taken together, the two cases point the same way. After the Canadian lawsuit and apology, the cost of failing to report is out in the open, and both US companies are moving towards reporting more. It means that an angry outburst or something written only for yourself can, once flagged, be read by a stranger and judged as a credible threat. The Florida case is still before the courts, and whether she is guilty is for them to decide.