Two days after an open letter called for a moratorium on more powerful generative AI models being developed so regulators can catch up with the likes of ChatGPT, Italy’s data protection authority has just put out a timely reminder that some countries do have laws that already apply to cutting edge AI — by ordering OpenAI to stop processing people’s data locally with immediate effect.
The Italian DPA said it’s concerned ChatGPT’s maker is breaching the European Union’s General Data Protection Regulation (GDPR).
Specifically, the Guarante said it has issued the order to block ChatGPT over concerns OpenAI has unlawfully processed people’s data — and also over the lack of any system to prevent minors from accessing the tech.
The San Francisco-based company has 20 days to respond to the order — backed up by the threat of some meaty penalties if it fails to comply. (Reminder: Fines for breaches of the EU’s data protection regime can scale up to 4% of annual turnover or €20M, whichever is greater.)
It’s worth noting that since OpenAI does not have a legal entity established in the EU any data protection authority is empowered to intervene, under the GDPR, if it sees risks to local users. (So where Italy steps in, others may follow.)
Suite of GDPR issues
The GDPR applies whenever EU users’ personal data is processed. And it’s clear OpenAI’s large language model has been crunching this kind of information — since it can, for example, produce biographies of named individuals in the region on-demand (we know; we’ve tried it). Although OpenAI has declined to provide details of the training data used for the latest iteration of the technology, GPT-4. But it has disclosed that earlier models were trained on data scraped from the Internet, including forums such as Reddit. So if you’re been reasonably online, chances are the bot knows your name.
Add to that, ChatGPT has been shown producing totally false information about named individuals — apparently making up details its training data lacks. Which potentially raises further GDPR concerns — since the regulation provides Europeans with a suite of rights over their data, including the right to rectification of errors. And it’s not clear how/whether people can ask OpenAI to correct erroneous pronouncements about them generated by the bot, in just one example scenario.
The Guarante‘s statement also highlights a data breach the service suffered earlier this month — when OpenAI admitted a conversation history feature had been leaking users’ chats and said it may have exposed some users’ payment information.
Data breaches are another area the GDPR regulates — with a focus on ensuring entities that process personal data are adequately protecting the information. The pan-EU law also contains requirements to notify relevant supervisory authorities of significant breaches within tight time-periods.
Overarching all this is the big(ger) question of what legal basis OpenAI has relied upon for processing Europeans’ data in the first place? Aka, the lawfulness of this processing.
The GDPR allows for a number of possibilities — from consent to public interest — but the scale of processing to train these large language models complicates the legality question, as the Guarante notes (pointing to the “mass collection and storage of personal data”), with data minimization being another big focus in the regulation — which also contains principles that require transparency and fairness. Yet, at the least, the (now) for-profit company behind ChatGPT does not appear to have informed people whose data it’s repurposed to train its commercial AIs. Which could be a pretty sticky problem for it.
If OpenAI has processed Europeans’ data unlawfully, DPAs across the bloc could order the data to be deleted — although whether that would force it to retrain models trained on data unlawfully obtained is one open question as an existing law grapples with cutting edge tech.
On the flip side, Italy may have just banned all machine learning by, er, accident… 
“[T]he Privacy Guarantor notes the lack of information to users and all interested parties whose data is collected by OpenAI but above all the absence of a legal basis that justifies the mass collection and storage of personal data, for the purpose of ‘training’ the algorithms underlying the operation of the platform,” the DPA writes in its statement today [which we’ve translated from Italian using AI].
“As evidenced by the checks carried out, the information provided by ChatGPT does not always correspond to the real data, thus determining an inaccurate processing of personal data,” it added.
The authority added that it is concerned about the risk of minors’ data being processed by OpenAI — since the company is not actively preventing people under the age of 13 from signing up to use the chatbot, such as by applying age verification technology.
Risks to children’s data is an area where the regulator has been very active — recently ordering a similar ban on the virtual friendship AI chatbot, Replika, over child safety concerns. In recent years, it has also pursued TikTok over underage usage — forcing the company to purge over half a million accounts it could not confirm did not belong to kids.
So if OpenAI can’t definitively confirm the age of any users it’s signed up in Italy it could — at the very least — be forced to delete their accounts and start again with a more robust sign up process.
OpenAI was contacted for a response to the Guarante‘s order.
Lilian Edwards, an expert in data protection and Internet law at Newcastle University — who’s ahead of the curve in conducting research on the implications of “algorithms that remember” — told TechCrunch: “What’s fascinating is that it more or less copy-pasted Replika in the emphasis on access by children to inappropriate content. But the real time-bomb is denial of lawful basis — which should apply to ALL or at least many machine learning systems, not just generative AI.”
She pointed back to the pivotal ‘right to be forgotten’ case, involving Google search — where a challenge was brought to its consentless processing of personal data by an individual in Spain — but while European courts established a right for individuals to ask search engines to remove inaccurate or outdated information about them (balanced against a public interest test), Google’s processing of personal data in that context (Internet search) did not get struck down by EU regulators over the lawfulness of processing point — seemingly on the grounds that it was providing a public utility. But also, ultimately, because Google ended up providing rights of erasure and rectification to EU data subjects.
“Large language models don’t offer those remedies and it’s not entirely clear they would, could or what the consequences would be,” Edwards added — suggesting that enforced retaining of models may be one potential fix.
Or, well, that technologies like ChatGPT may simply have broken data protection law…
This report was updated with additional comment
Italy orders ChatGPT blocked citing data protection concerns by Natasha Lomas originally published on TechCrunch
