New Jersey high school students accused of making AI-generated pornographic images of classmates

WESTFIELD, N.J. — Parents and students at Westfield High School in New Jersey say students used an app or website to make AI-generated pornographic images of their classmates.

Parents recently got an email from the principal, warning of the dangers of artificial intelligence and saying the complaints from students had sparked an investigation. Some parents turned to the Westfield Police Department.

Dorota Mani, who owns a Jersey City day care, was already worried about the impact artificial intelligence could have on kids.

“AI problem. I would call it ‘AI pandemic’ at this point,” she said.

She says she’s even more worried now that her daughter, a Westfield High School student, has become a victim.

“My daughter texted me, ‘Mom, naked pictures of me are being distributed.’ That’s it. Heading to the principal’s office,” Mani said.

Parents later got the following email from the principal:

“I am writing to make you aware of a situation that resulted in widespread misinformation and resulted in significant worry and concern amongst the student body of Westfield High School.  Earlier today, students brought to our attention that some of our students had used Artificial Intelligence to create pornographic images from original photos.  There was a great deal of concern about who had images created of them and if they were shared. At this time, we believe that any created images have been deleted and are not being circulated.  This is a very serious incident.   We are continuing to investigate and will inform individuals and families of students involved once the investigation is complete.  This will happen before the weekend.   We made counseling available for all affected students and encouraged them to return to class when they felt able to do so.  Additionally, our School Resource Officer and the Westfield PD have been made aware of our investigation.  If a parent/guardian thinks their child is a victim of a criminal act in relation to this incident please report the matter to Westfield Police.

“I wanted to make you aware of the situation, as, in addition to harming the students involved and disrupting the school day, it is critically important to talk with your children about their use of technology and what they are posting, saving and sharing on social media.   New technologies have made it possible to falsify images and students need to know the impact and damage those actions can cause to others. 

“We will continue to educate your children on the importance of responsible use of technology and hope you reinforce these messages at home.”

These kinds of altered pictures are known online as “deepfakes.”

“She started crying, and then she was walking in the hallways and seeing other girls of Westfield High School crying, some of them victims and some of them just friends of the victims that just felt for them,” Mani said.

Mani says she did file a police report.

Police told CBS New York they have no comment.

“It’s something that we will need to learn, our children will need to be aware of,” Mani said.

Congress is considering a bill introduced by Rep. Joe Morelle, of New York, that would make it illegal to share nonconsensual “deepfake” images online.

Mani and her daughter plan to advocate for laws to protect people from deepfakes.

“Her words — ‘We don’t want to wait for a tragedy to happen, we should do something proactively,'” Mani said.

A district spokesperson tells CBS New York the incident happened over the summer but was brought to the attention of school administrators on Oct. 20.

School Superintendent Dr. Raymond González released the following statement:

“All school districts are grappling with the challenges and impact of artificial intelligence and other technology available to students at any time and anywhere. The Westfield Public School District has safeguards in place to prevent this from happening on our network and school-issued devices. We continue to strengthen our efforts by educating our students and establishing clear guidelines to ensure that these new technologies are used responsibly in our schools and beyond.”  

The district also says they can’t comment on how many students are affected or any disciplinary actions.