Students have been called to the office — and even arrested — for AI surveillance false alarms

By SHARON LURYE – AP Education Writer

Alexa Manganiotis, 16, said she was startled by how quickly monitoring software works. West Palm Beach’s Dreyfoos School of the Arts, which she attends, last year piloted Lightspeed Alert, a surveillance program. Interviewing a teacher for her school newspaper, Alexa discovered two students once typed something threatening about that teacher on a school computer, then deleted it. Lightspeed picked it up, and “they were taken away like five minutes later,” Alexa said.

Teenagers face steeper consequences than adults for what they write online, Alexa said.

“If an adult makes a super racist joke that’s threatening on their computer, they can delete it, and they wouldn’t be arrested,” she said.

Amy Bennett, chief of staff for Lightspeed Systems, said that the software helps understaffed schools “be proactive rather than punitive” by identifying early warning signs of bullying, self-harm, violence or abuse.

The technology can also involve law enforcement in responses to mental health crises. In Florida’s Polk County Schools, a district of more than 100,000 students, the school safety program received nearly 500 Gaggle alerts over four years, officers said in public Board of Education meetings. This led to 72 involuntary hospitalization cases under the Baker Act, a state law that allows authorities to require mental health evaluations for people against their will if they pose a risk to themselves or others.

“A really high number of children who experience involuntary examination remember it as a really traumatic and damaging experience — not something that helps them with their mental health care,” said Sam Boyd, an attorney with the Southern Poverty Law Center. The Polk and West Palm Beach school districts did not provide comments.

An analysis shows a high rate of false alarms

Information that could allow schools to assess the software’s effectiveness, such as the rate of false alerts, is closely held by technology companies and unavailable publicly unless schools track the data themselves.

Gaggle alerted more than 1,200 incidents to the Lawrence, Kansas, school district in a recent 10-month period. But almost two-thirds of those alerts were deemed by school officials to be nonissues — including over 200 false alarms from student homework, according to an Associated Press analysis of data received via a public records request.

Students in one photography class were called to the principal’s office over concerns Gaggle had detected nudity. The photos had been automatically deleted from the students’ Google Drives, but students who had backups of the flagged images on their own devices showed it was a false alarm. District officials said they later adjusted the software’s settings to reduce false alerts.

Natasha Torkzaban, who graduated in 2024, said she was flagged for editing a friend’s college essay because it had the words “mental health.”

“I think ideally we wouldn’t stick a new and shiny solution of AI on a deep-rooted issue of teenage mental health and the suicide rates in America, but that’s where we’re at right now,” Torkzaban said. She was among a group of student journalists and artists at Lawrence High School who filed a lawsuit against the school system last week, alleging Gaggle subjected them to unconstitutional surveillance.

School officials have said they take concerns about Gaggle seriously, but also say the technology has detected dozens of imminent threats of suicide or violence.

“Sometimes you have to look at the trade for the greater good,” said Board of Education member Anne Costello in a July 2024 board meeting.

Two years after their ordeal, Mathis said her daughter is doing better, although she’s still “terrified” of running into one of the school officers who arrested her. One bright spot, she said, was the compassion of the teachers at her daughter’s alternative school. They took time every day to let the kids share their feelings and frustrations, without judgment.

“It’s like we just want kids to be these little soldiers, and they’re not,” said Mathis. “They’re just humans.”


This reporting reviewed school board meetings posted on YouTube, courtesy of DistrictView, a dataset created by researchers Tyler Simko, Mirya Holman and Rebecca Johnson.


The Associated Press’ education coverage receives financial support from multiple private foundations. AP is solely responsible for all content. Find AP’s standards for working with philanthropies, a list of supporters and funded coverage areas at AP.org.

Tags

PREV

PREVIOUS

Q&A with former U.S. Rep. Gabriele Giffords | Gun legislation efforts continue

Former U.S. Rep. Gabrielle Giffords, a co-founder of GIFFORDS, a nonprofit that promotes gun-safety laws, survived being shot in the head in a 2011 assassination attempt in Tucson, but has spent years recovering. She spoke with Colorado Politics about her work on Aug. 6, before joining U.S. Sen. Michael Bennet for a town hall organized by the Colorado […]

NEXT

NEXT UP

Trump promises new steps to tackle homelessness and crime in Washington, DC

President Donald Trump is promising new steps to tackle homelessness and crime in Washington. Trump wrote in a social media post that he would hold a White House news conference on Monday to discuss his plans to make the District of Columbia “safer and more beautiful than it ever was before.” “The Homeless have to move out, […]


Welcome Back.

Streak: 9 days i

Stories you've missed since your last login:

Stories you've saved for later:

Recommended stories based on your interests:

Edit my interests