OpenAI said on Friday it is tightening limits on weapons-related information that ChatGPT provides to teens, including questions that appear harmless. The company said the safeguards are aimed at preventing real-world violence.

Under the new measures, OpenAI will restrict practical information on weapons and security systems that could be misused. The limits apply to teens even when there is no sign of violent intent.

For all users, ChatGPT will be able to automatically end chats where someone appears to be fixating on violence. OpenAI may temporarily pause accounts after multiple such chats. The company also said it is limiting detail on past mass attacks, including the weapons used and how they were obtained.

OpenAI said it strengthened its age prediction system to move users it estimates are under 18 into the teen experience. It plans to notify parents if a teen's account is paused for repeatedly discussing violence.

The safeguards will begin rolling out in the next few weeks, and OpenAI said it will continue to refine them.

The report notes that AI companies face growing scrutiny over the risks chatbots pose to young users and their role in detecting threats of harm. OpenAI said earlier this month that teens spend under 15 minutes a day on average on ChatGPT. A Common Sense Media report earlier this month found that ChatGPT's teen version often alerted parents late or not at all when teen accounts discussed suicide, self-harm or eating disorders.