A Dutch court has banned the AI chatbot Grok from generating nonconsensual "undressed" images, citing serious risks of abuse and insufficient safeguards.
An Amsterdam court ruled on Thursday that the feature, which allows users to create manipulated images of individuals without their consent, must no longer be available in the country. Grok is integrated into Elon Musk's social media company X.
The case was brought by Offlimits, a Dutch online abuse expertise center, which argued that the tool enabled the creation and spread of explicit images, including potentially illegal content involving minors.
Although the company behind Grok said it had introduced measures to prevent misuse, the court found those steps ineffective. It noted that shortly before the hearing, such images could still be generated.
The ban applies to images of individuals residing in the Netherlands, as well as to the production and distribution of such content within the country.
Under the ruling, Grok faces a fine of €100,000 (about $115,300) per violation, with penalties capped at €10 million ($11.5 million). The court also said the chatbot cannot remain active on X in the Netherlands as long as the feature remains accessible.
The decision comes amid growing scrutiny in Europe over AI-generated explicit content. The European Parliament approved a broader ban on so-called "undressing apps," expected to take effect across the EU before the summer.
A number of other countries have also taken moves against Grok over the feature.
The EU has also opened an investigation into Grok over concerns it could be used to generate harmful content involving women and children.
