Case ico-snap-my-ai-preliminary-noticeChatbot harmUnited Kingdom
UK ICO issues preliminary enforcement notice to Snap over My AI's privacy risks to children
The Information Commissioner's Office provisionally found that Snap had not adequately assessed the data protection risks of its generative AI chatbot My AI, especially for children, before launch. A final notice could have forced Snap to stop offering My AI in the UK.
HighHarm caused by AIStatus: ResolvedVerified case file
Open in the interactive mapCite this case
- Reported
- Oct 6, 2023
- Location
- United Kingdom
- Target
- UK Snapchat users of My AI, including children aged 13 to 17
- Severity
- High
- Verification
- Confirmed by victim, vendor or authority
- Triggered by
- Snap Inc. (My AI, built on OpenAI models)
- Actor type
- AI product design / safeguards
- Sector
- Individuals & consumers
- Reported loss
- None reported
- CVE
- None
What happened
The Information Commissioner's Office provisionally found that Snap had not adequately assessed the data protection risks of its generative AI chatbot My AI, especially for children, before launch. A final notice could have forced Snap to stop offering My AI in the UK.
Impact: Preliminary enforcement notice; the ICO's first action against a generative AI product
Who triggered it
Actor: Snap Inc. (My AI, built on OpenAI models)
Attribution: UK Information Commissioner's Office
Motive: None; regulator action over product risk assessment
How it happened
Attack vector: Chatbot pinned to the top of Snapchat users' chat feeds, used by millions of UK teens
Root cause: Inadequate pre-launch data protection risk assessment
Timeline
- My AI rolled out to all UK Snapchat users
- ICO issues preliminary enforcement notice
- ICO concludes investigation without a final notice
Response
Snap said it was reviewing the findings. In May 2024 the ICO closed the case without a final notice after Snap completed a compliant risk assessment.
Current status: Resolved
Research warnings this case matches
- Seen in real incidents People's trust in AI companions turns harmful · CAIS
- Early signs Relying on AI companions harms some users · International AI Safety Report
- Early signs Many people lean on AI for emotional support · UK AISI
- Seen in real incidents Human-computer interaction · MIT AI Risk Repository
Compared with published AI risk research from several institutes on the Research vs reality page.
Sources (open in a new tab)
- LBC lbc.co.uk
- Addleshaw Goddard addleshawgoddard.com
- ICO via Wired-Gov wired-gov.net
Cite this case
UK ICO issues preliminary enforcement notice to Snap over My AI's privacy risks to children. AI Threat Watch, case ico-snap-my-ai-preliminary-notice, Oct 6, 2023. https://aithreat.io/cases/ico-snap-my-ai-preliminary-notice/
Permanent link: https://aithreat.io/cases/ico-snap-my-ai-preliminary-notice/
Related cases
The same type of incident, most recent first.