{"meta":{"exportedAt":"2026-11-14T10:17:53.287Z","formatVersion":2,"selection":{"q":"x","system":"","harm":"","context":"","country":"","role":"","relation":"","evidence":"","year":"2019","response":"","severity":"","verification":"","view":"incidents","sort":"added"},"totalIncidents":4,"coverage":{"cases":4,"countries":2,"languages":1,"unknownLocation":1,"locationPending":0,"unknownLanguage":0,"unknownDate":0,"lawsuits":1,"regulatory":1,"minors":0,"coreRelations":1,"contextualRelations":3,"mixedRelations":0,"unknownRelations":0,"relationPending":0,"relationUnknown":0},"countingNote":"Distinct public cases in this selection. People counts apply within individual cases only; cross-case person overlap has not been resolved. No population incidence estimate.","affectedCountNote":"Interpret person counts with affectedCountStatus and the reported effects. Unquantified zeros are placeholders, not a measured zero.","source":"AI incidents","publisher":"NOPE","url":"https://nope.net/incidents","license":"CC BY 4.0"},"incidents":[{"id":"2019-uk-passport-online-photo-checker-told-black-applicants-their-closed-mouths-looked-open","caseFacts":{"claims":[{"id":"c1","status":"corroborated","evidence":[{"locator":"The facial detection system informs people when it thinks the photo uploaded may not meet strict requirements, which include a plain expression and the mouth to be kept closed.","relation":"supports","source_id":"s1"},{"locator":"The passport application website uses an automated check to detect poor quality photos which do not meet Home Office rules. These include having a neutral expression, a closed mouth and looking straight at the camera.","relation":"supports","source_id":"s2"}],"assertion":"The UK government's online passport application runs an automated face-detection check that tells applicants when it thinks an uploaded photo does not meet the rules, which include a neutral expression and a closed mouth.","causal_attribution":"Function of the system as described independently by PA and the BBC."},{"id":"c2","status":"reported","evidence":[{"locator":"A young black man hoping to renew his passport online was stunned when the automated photo checker mistook his lips for an open mouth.; My mouth is closed, I just have big lips.","relation":"supports","source_id":"s1"}],"assertion":"A young Black man renewing his passport online says the automated checker mistook his lips for an open mouth, and that to submit the photo anyway he had to write in the comment box that his mouth was closed and he just had big lips.","causal_attribution":"The applicant's account to PA; the checker's message is attributed to him."},{"id":"c3","status":"reported","evidence":[{"locator":"She became frustrated after the system told her it looked like her eyes are closed and that it could not find the outline of her head.; proceeded with one of the images and received her passport without any further problems.","relation":"supports","source_id":"s1"},{"locator":"She told the BBC she had attempted to upload 10 different photographs over the course of a week, and each one had been rated as \"poor\" quality by the site.","relation":"supports","source_id":"s2"}],"assertion":"A Black educational technologist says the checker told her that her eyes looked closed and that it could not find the outline of her head, and that over a week she uploaded 10 photographs, each rated poor quality, before proceeding with one and receiving her passport.","causal_attribution":"Her own account, given to PA and to the BBC; two interviews with one witness are one origin."},{"id":"c4","status":"reported","evidence":[{"locator":"When posting about the issue on Twitter at the time, the Passport Office tweeted back to her, saying it was sorry the photo upload service hadn’t “worked as it should","relation":"supports","source_id":"s1"}],"assertion":"PA reported that when the technologist posted about the problem on Twitter, the Passport Office replied that it was sorry the photo upload service had not worked as it should.","causal_attribution":"PA's report of the Passport Office's reply; the tweet itself was not inspected."},{"id":"c5","status":"reported","evidence":[{"locator":"One black student said she was wrongly told her mouth looked open each time she uploaded five different photos to the government website.; which involved writing a note to say her mouth was indeed closed.; It should be the norm for these systems to work well for everyone, she added.","relation":"supports","source_id":"s2"}],"assertion":"A Black student says the checker wrongly told her that her mouth looked open on each of five photos she uploaded, and that she got one approved after challenging the verdict by writing a note to say her mouth was closed; she said such systems should work well for everyone.","causal_attribution":"The student's account to the BBC."},{"id":"c6","status":"reported","evidence":[{"locator":"Dark-skinned women are told their photos are poor quality 22% of the time, while the figure for light-skinned women is 14%; Photos of women with the darkest skin were four times more likely to be graded poor quality, than women with the lightest skin.","relation":"supports","source_id":"s2"}],"assertion":"The BBC fed 1,130 passport-style photos of politicians into the checker and reported that photos of dark-skinned women were graded poor quality 22% of the time against 14% for light-skinned women, and that photos of women with the darkest skin were four times more likely to be graded poor quality than those of women with the lightest skin.","causal_attribution":"The BBC's own test, which it says the Alan Turing Institute reproduced; reported as the BBC's finding."},{"id":"c7","status":"reported","evidence":[{"locator":"Documents released as part of a freedom of information request in 2019 had previously revealed the Home Office was aware of this problem, but decided \"overall performance\" was good enough to launch the online checker.; The automated checker was supplied to the government by an external provider which it declined to name.","relation":"supports","source_id":"s2"}],"assertion":"The BBC reported that documents released under a 2019 freedom of information request showed the Home Office was aware of the problem but decided overall performance was good enough to launch the checker, which an external provider it declined to name supplied.","causal_attribution":"The BBC's account of the FOI documents, which were not inspected."},{"id":"c8","status":"reported","evidence":[{"locator":"In the vast majority of cases where a photo does not pass our automated check, customers may override the outcome and submit the photo as part of their application.","relation":"supports","source_id":"s1"},{"locator":"Over nine million people have used this service and our systems are improving.","relation":"supports","source_id":"s2"}],"assertion":"The Home Office told PA in 2019 that in the vast majority of cases applicants whose photo fails the automated check may override it and submit the photo, and told the BBC in 2020 that over nine million people had used the service and its systems were improving.","causal_attribution":"The Home Office's statement, kept as its response."}],"effects":[{"label":"applicants say the checker wrongly said their mouths were open or eyes closed, forcing retries or written overrides","claim_id":"c2","direction":"negative"}],"sources":[{"id":"s1","url":"https://www.irishnews.com/magazine/technology/2019/09/19/news/passport-photo-checker-falsely-flags-black-man-s-lips-as-open-mouth-1716760/","kind":"news_report","access":"read","language":"en","translation_note":"Read in English on 2026-10-29; no translation.","independence_group":"pa-2019-09-19"},{"id":"s2","url":"https://www.bbc.co.uk/news/technology-54349538","kind":"news_report","access":"read","language":"en","translation_note":"Read in English on 2026-10-29; no translation.","independence_group":"bbc-2020-10-08"}],"version":1,"ai_roles":["institutional_use"],"contexts":["public_services"],"unknowns":["The date of each applicant's attempts: the technologist posted in April 2019 according to PA, the man applied before 19 September 2019, and the student before 8 October 2020.","The supplier and design of the face-detection check; the Home Office declined to name the provider.","Whether the Home Office later changed or replaced the checker."],"geography":{"basis":"The check is part of the UK government's online passport application. PA and the BBC place the three applicants in England.","court_countries":[],"event_countries":["GB"],"affected_person_countries":["GB"]},"publication":{"basis":"Published as a government service's automated face-detection check making wrong statements about three described applicants' faces: they say it called closed mouths open and eyes closed and made them retry or override it. Their accounts are attributed; the BBC's group-level test and the Home Office's response are recorded separately.","reviewed_on":"2026-10-29"},"ai_involvement":{"basis":"PA and the BBC describe the passport website's photo check as an automated face-detection system that tells applicants when it thinks a photo breaks the rules. Three applicants say its verdicts about their own faces were wrong: it said a man's closed mouth was open, said a woman's eyes were closed and could not find the outline of her head, and said a student's mouth looked open on five photos. Those wrong verdicts are what made them retry or write overrides. The Home Office did not deny the failures; the BBC test and the 2019 FOI documents report a wider skin-tone pattern but do not establish the cause in each applicant's case.","status":"reported"},"person_relations":["made_claim_about"]},"name":"UK passport photo checker told Black applicants their closed mouths looked open, they say; a BBC test found darker-skinned women flagged more often","summary":"The UK government's online passport application uses an automated face-detection check that tells applicants when it thinks an uploaded photo breaks the rules, which include a neutral expression and a closed mouth. In September 2019 PA reported that a young Black man renewing his passport online said the checker mistook his lips for an open mouth, and that to submit the photo anyway he had to write in a comment box that his mouth was closed and he just had big lips. PA also reported that a Black educational technologist said the system told her that her eyes looked closed and that it could not find the outline of her head; she told the BBC she uploaded 10 photographs over a week and each was rated poor, and PA reported that the Passport Office tweeted that it was sorry the service had not worked as it should. In October 2020 a BBC investigation reported that a Black student said she was wrongly told her mouth looked open on each of five photos and got one accepted only by writing a note challenging the verdict. The BBC fed 1,130 passport-style photos of politicians into the checker and found that photos of dark-skinned women were graded poor quality 22% of the time against 14% for light-skinned women; it reported that documents released under a 2019 freedom of information request showed the Home Office knew of the problem but judged overall performance good enough to launch. The Home Office said applicants can override the check, that the checker is a customer aid, and that its systems are improving.","incidentDate":"2019-04-01","incidentKind":"bounded_series","incidentDatePrecision":"month","exposurePattern":"repeated_interactions","reportedDate":"2019-09-19","aiSystem":"Automated face-detection photo check on the UK government's online passport application, supplied to the Home Office by an external provider it declined to name","aiProduct":"HM Passport Office photo checker","severity":"low","verificationStatus":"credible","harmCategories":[],"harmOutcomes":["psychological_distress"],"harmOutcomeSummary":"The applicants say the checker wrongly told them their mouths looked open or their eyes closed, so they had to retry with new photos or write explanations to override it. The man describes annoyance, the technologist frustration over a week of attempts, and the student says she should not have had to override a system that was not built for her. The technologist received her passport and the student got a photo approved; no refused passport is reported.","frameworkFacets":[],"causationStatus":"supported","participantUsersAffectedMin":3,"otherPeopleHarmedMin":0,"affectedCountStatus":"documented_minimum","affectedCountEvidence":"Three applicants each describe the checker misreading their own photos (PA: a man and a technologist; BBC: a student and the same technologist, counted once). PA says the man's case was not isolated, and the BBC test shows higher failure rates for darker-skinned people, but neither gives a count of other applicants harmed, so three is a minimum.","victimAgeRange":"adult","jurisdiction":"GB","platformType":"other","outcomeType":"media_coverage","outcomeStatus":"unknown","primarySourceUrl":"https://www.irishnews.com/magazine/technology/2019/09/19/news/passport-photo-checker-falsely-flags-black-man-s-lips-as-open-mouth-1716760/","primarySourceLabel":"PA via The Irish News, passport photo checker falsely flags black man's lips as open mouth, 19 September 2019","firstPublishedAt":"2026-10-29T03:13:05.403403+00:00","updatedAt":"2026-10-29T03:13:05.403403+00:00","scopeVersion":"facts-v3","tags":["passport","photo-checker","face-detection","home-office","public-services","gov-uk"]},{"id":"2019-apple-siri-accidental-activations-recorded-private-conversations-heard-by-contractors-95-million-settlement","caseFacts":{"claims":[{"id":"c1","status":"corroborated","evidence":[{"locator":"'a small proportion of Siri recordings are passed on to contractors working for the company around the world. They are tasked with grading the responses on a variety of factors, including whether the activation of the voice assistant was deliberate or accidental'","relation":"supports","source_id":"s1"},{"locator":"'part of our Siri quality evaluation process — which we call grading'","relation":"supports","source_id":"s2"}],"assertion":"A small proportion of Siri recordings was passed to contractors who graded them.","causal_attribution":"The Guardian's report and Apple's own statement both describe human grading of Siri audio."},{"id":"c2","status":"reported","evidence":[{"locator":"'Siri can be accidentally activated when it mistakenly hears its “wake word”, the phrase “hey Siri”.'; '“The sound of a zip, Siri often hears as a trigger,” the contractor said.'; '“The regularity of accidental triggers on the watch is incredibly high,” they said.'","relation":"supports","source_id":"s1"},{"locator":"'Our team will work to delete any recording which is determined to be an inadvertent trigger of Siri.'","relation":"supports","source_id":"s2"}],"assertion":"Siri sometimes activated without being deliberately invoked and recorded audio that reached Apple; according to the contractor, the sound of a zip was often heard as a trigger and accidental triggers on the Apple Watch were very frequent.","causal_attribution":"The trigger examples are the contractor's account; Apple's undertaking to delete inadvertent-trigger recordings confirms that such recordings were held, without giving their frequency."},{"id":"c3","status":"reported","evidence":[{"locator":"'There have been countless instances of recordings featuring private discussions between doctors and patients, business deals, seemingly criminal dealings, sexual encounters and so on. These recordings are accompanied by user data showing location, contact details, and app data.'; 'Accidental activations led to the receipt of the most sensitive data that was sent to Apple.'","relation":"supports","source_id":"s1"}],"assertion":"According to the contractor, recordings from accidental activations included private discussions between doctors and patients, business deals, apparently criminal dealings and sexual encounters, accompanied by location, contact details and app data.","causal_attribution":"A single anonymous contractor's account, relayed by the Guardian."},{"id":"c4","status":"reported","evidence":[{"locator":"'User requests are not associated with the user’s Apple ID.'; 'a very small random subset, less than 1% of daily Siri activations, are used for grading'","relation":"supports","source_id":"s1"}],"assertion":"Apple told the Guardian that Siri requests were not associated with the user's Apple ID and that less than 1% of daily Siri activations were used for grading.","causal_attribution":"Apple's statement to the Guardian; context and counterevidence on identifiability and scale."},{"id":"c5","status":"documented","evidence":[{"locator":"'immediately suspended human grading of Siri requests'; 'we realize we haven’t been fully living up to our high ideals, and for that we apologize.'; 'by default, we will no longer retain audio recordings of Siri interactions.'; 'users will be able to opt in to help Siri improve by learning from the audio samples of their requests.'","relation":"supports","source_id":"s2"}],"assertion":"Apple said it had suspended human grading of Siri requests, apologised for not fully living up to its ideals, and said it would by default no longer retain audio recordings, would make audio grading opt-in and would work to delete recordings determined to be inadvertent triggers.","causal_attribution":"Apple's own published statement establishes what Apple said and undertook."},{"id":"c6","status":"reported","evidence":[{"locator":"'Mobile device owners complained that Apple routinely recorded their private conversations after they activated Siri unintentionally, and disclosed these conversations to third parties such as advertisers.'; 'Two plaintiffs said their mentions of Air Jordan sneakers and Olive Garden restaurants triggered ads for those products. Another said he got ads for a brand name surgical treatment after discussing it, he thought privately, with his doctor.'","relation":"supports","source_id":"s3"}],"assertion":"In Lopez et al v. Apple Inc., device owners alleged that Apple routinely recorded their private conversations after they activated Siri unintentionally and disclosed them to third parties such as advertisers; two plaintiffs said mentions of products triggered ads for them, and another said he got ads for a surgical treatment after discussing it with his doctor.","causal_attribution":"Plaintiffs' allegations as reported by Reuters; not established."},{"id":"c7","status":"reported","evidence":[{"locator":"'agreed to pay $95 million in cash to settle a proposed class action lawsuit'; 'The class period runs from Sept. 17, 2014 to Dec. 31, 2024.'; 'may receive up to $20 per Siri-enabled device'; 'Apple denied wrongdoing in agreeing to settle.'","relation":"supports","source_id":"s3"}],"assertion":"Apple agreed to pay $95 million in cash to settle the class action, covering a class period from 17 September 2014 to 31 December 2024 with up to $20 per Siri-enabled device, and denied wrongdoing.","causal_attribution":"Reuters' report of the preliminary settlement filing; the filing itself was not read."},{"id":"c8","status":"reported","evidence":[{"locator":"'U.S. District Judge Jeffrey S. White granted final approval on October 16, 2025, of Plaintiffs’ $95 million settlement with Apple'","relation":"supports","source_id":"s4"}],"assertion":"Co-class counsel say Judge Jeffrey S. White granted final approval of the $95 million settlement on 16 October 2025.","causal_attribution":"Plaintiffs' counsel's statement; the court order was not read, and other summaries give September or 14 October 2025 dates."}],"effects":[{"label":"Private conversations captured by accidental Siri activations and heard by contract graders, according to a contractor","claim_id":"c3","direction":"negative"}],"sources":[{"id":"s1","url":"https://www.theguardian.com/technology/2019/jul/26/apple-contractors-regularly-hear-confidential-details-on-siri-recordings","kind":"news_report","access":"read","language":"en","translation_note":"Read in English on 2026-10-29 (theguardian.com, HTTP 200). No translation was needed; the research agent (an AI) read the text directly.","independence_group":"guardian-siri-contractor"},{"id":"s2","url":"https://www.apple.com/newsroom/2019/08/improving-siris-privacy-protections/","kind":"company_statement","access":"read","language":"en","translation_note":"Read in English on 2026-10-29 (apple.com, HTTP 200); the page gives its date as 28 August 2019. No translation was needed.","independence_group":"apple-statement-siri-grading"},{"id":"s3","url":"https://finance.yahoo.com/news/apple-pay-95-million-settle-151238610.html","kind":"news_report","access":"read","language":"en","translation_note":"Read in English on 2026-10-29 (finance.yahoo.com syndication of the Reuters report, HTTP 200; reuters.com returned 401). No translation was needed.","independence_group":"reuters-lopez-settlement"},{"id":"s4","url":"https://lowey.com/news/the-courts-granted-final-approval-for-apple-siri-litigation-for-95-million/","kind":"law_firm_statement","access":"read","language":"en","translation_note":"Read in English on 2026-10-29 (lowey.com, HTTP 200). The post carries no publication date of its own; the court order itself was not read. No translation was needed.","independence_group":"lowey-final-approval"}],"version":1,"ai_roles":["own_use","unknown"],"contexts":["privacy","everyday_life","health"],"unknowns":["How many people were recorded by accidental activations or heard by graders.","Whether any recorded person was identified or harmed beyond the capture and listening described.","Whether the plaintiffs' advert allegations were ever substantiated; Apple denied wrongdoing.","The exact final-approval date in the court record (counsel gives 16 October 2025; the order was not read)."],"geography":{"basis":"The class action was in the US District Court for the Northern District of California (Oakland), per Reuters and co-class counsel. The Guardian says the contractors worked around the world and does not say where the recorded people were; no event or affected-person country is stated.","court_countries":["US"],"event_countries":[],"affected_person_countries":[]},"publication":{"basis":"Published as private conversations captured into an AI voice assistant without the speakers' knowledge: Siri's wake-word detection activated by mistake and graders heard the recordings, according to the Guardian's contractor source, and Apple acknowledged inadvertent-trigger recordings, apologised and changed its grading. The class action allegations are attributed to the plaintiffs; Apple settled for $95 million while denying wrongdoing.","reviewed_on":"2026-10-29"},"ai_involvement":{"basis":"Siri's own wake-word and raise-to-speak detection activated by mistake: the Guardian's contractor source says the sound of a zip was often heard as the trigger and accidental triggers on the Apple Watch were very frequent, and Apple undertook to delete recordings determined to be inadvertent triggers. The audio Siri captured on those activations was sent to Apple and a share was passed to contract graders, who, according to the contractor, heard doctor-patient discussions, business deals and sexual encounters with location and contact data attached. That capture into the assistant and disclosure to graders without the speakers knowing is the privacy harm recorded here; the plaintiffs' further claim that recordings led to targeted adverts is an allegation that Apple denied in settling.","status":"supported"},"person_relations":["received_personal_information"]},"name":"Whistleblower says accidental Siri activations recorded private conversations heard by Apple contract graders; Apple apologised and later settled class suit","summary":"On 26 July 2019 the Guardian reported that contractors grading a small share of Siri recordings for Apple regularly heard confidential information. Its anonymous source, a contractor, said Siri was often activated by mistake, for example by the sound of a zip, and that these accidental activations captured private discussions between doctors and patients, business deals, apparently criminal dealings and sexual encounters, with location, contact details and app data attached. Apple told the Guardian that requests were not linked to users' Apple IDs and that less than 1% of daily activations were graded. On 28 August 2019 Apple apologised, said it had suspended human grading, and said audio would no longer be retained by default, grading would become opt-in and recordings found to be inadvertent triggers would be deleted. Device owners' class action, Lopez v. Apple in the federal court in Oakland, alleged that Apple recorded private conversations after unintended Siri activations and disclosed them to third parties. Apple agreed to pay $95 million while denying wrongdoing, and co-class counsel say the judge granted final approval on 16 October 2025.","incidentDate":"2019-01-01","incidentKind":"bounded_series","incidentDatePrecision":"year","exposurePattern":"unknown","reportedDate":"2019-07-26","aiSystem":"Siri, Apple's voice assistant on iPhone, Apple Watch and HomePod, with wake-word (\"Hey Siri\") detection and raise-to-speak activation; a small share of its audio recordings was passed to contract graders","aiProduct":"Siri","aiCompany":"Apple","severity":"medium","verificationStatus":"credible","harmCategories":[],"harmOutcomes":["other_material_harm"],"harmOutcomeSummary":"A Guardian contractor source says accidental Siri activations recorded private conversations, including doctor-patient talk, that contract graders heard; Apple apologised, and settled a class suit for $95 million (loss of privacy).","frameworkFacets":[],"causationStatus":"supported","participantUsersAffectedMin":0,"otherPeopleHarmedMin":0,"affectedCountStatus":"unquantified","affectedCountEvidence":"The Guardian's contractor source describes \"countless instances\" of recordings of private conversations from accidental Siri activations but gives no number of people heard. Reuters reports that class members are estimated in the tens of millions; that is the size of the settlement class of device owners, not a count of people whose conversations were recorded or heard. No reliable count exists, so the count is unquantified with zero placeholders.","victimAgeRange":"unknown","jurisdiction":"US","platformType":"assistant","outcomeType":"lawsuit_settled","outcomeStatus":"resolved","primarySourceUrl":"https://www.theguardian.com/technology/2019/jul/26/apple-contractors-regularly-hear-confidential-details-on-siri-recordings","primarySourceLabel":"The Guardian (Alex Hern), 26 July 2019: Apple contractors 'regularly hear confidential details' on Siri recordings","firstPublishedAt":"2026-10-29T03:10:14.834844+00:00","updatedAt":"2026-10-29T03:10:14.834844+00:00","scopeVersion":"facts-v3","tags":["voice-assistant","false-activation","human-review","privacy","class-action","settlement"]},{"id":"2019-flanders-google-assistant-recordings-leaked-by-contract-reviewer-some-captured-without-wake-word","caseFacts":{"claims":[{"id":"c1","status":"reported","evidence":[{"locator":"'VRT NWS was able to listen to more than a thousand excerpts recorded via Google Assistant'; 'In the aftermath of the Amazon case, we were able to talk to someone who works for a Google subcontractor'","relation":"supports","source_id":"s1"},{"locator":"'we partner with language experts around the world who understand the nuances and accents of a specific language'; 'one of these language reviewers has violated our data security policies by leaking confidential Dutch audio data'","relation":"supports","source_id":"s2"}],"assertion":"Excerpts of audio recorded by Google Home speakers and the Google Assistant app were transcribed by contract language reviewers, and more than a thousand Dutch-language excerpts reached VRT NWS from someone working for a Google subcontractor.","causal_attribution":"Reported by VRT from its own access to the excerpts; Google confirms that a language reviewer leaked Dutch audio data."},{"id":"c2","status":"reported","evidence":[{"locator":"'153 of which were conversations that should never have been recorded and during which the command ‘Okay Google’ was clearly not given.'; 'as soon as someone in the vicinity utters a word that sounds a bit like ’Okay Google’, Google Home starts to record.'; 'Most of these recordings were made consciously, but Google also listens to conversations that should never have been recorded'","relation":"supports","source_id":"s1"}],"assertion":"VRT says 153 of the excerpts were conversations in which the command \"Okay Google\" was clearly not given, recorded when the speaker reacted to words that sound like it; it says most of the recordings it heard were made consciously.","causal_attribution":"VRT's count from its own listening; Google does not give a count for the leaked set."},{"id":"c3","status":"documented","evidence":[{"locator":"'Rarely, devices that have the Google Assistant built in may experience what we call a “false accept.”'; 'that our software interpreted to be the hotword'","relation":"supports","source_id":"s2"}],"assertion":"Google says devices with the Assistant built in may rarely have a \"false accept\", when its software interprets background noise or words as the hotword.","causal_attribution":"The company's own published description of how its hotword detection can misfire."},{"id":"c4","status":"reported","evidence":[{"locator":"'In these recordings we could clearly hear addresses and other sensitive information. This made it easy for us to find the people involved and confront them with the audio recordings.'; 'We let ordinary Flemish people hear some of their own recordings.'; 'immediately recognise the voice of their son and their grandchild'","relation":"supports","source_id":"s1"}],"assertion":"VRT says it could hear addresses and other sensitive information in the recordings, found the people involved, and played them their recordings; one man recognised his own voice and a couple recognised the voices of their son and grandchild.","causal_attribution":"VRT's own account of identifying people from the leaked recordings."},{"id":"c5","status":"reported","evidence":[{"locator":"'bedroom conversations, conversations between parents and their children, but also blazing rows and professional phone calls containing lots of private information'; 'he heard a woman who was in definite distress'","relation":"supports","source_id":"s1"}],"assertion":"VRT describes unintended recordings of bedroom conversations, conversations between parents and children, rows and professional calls, and a reviewer who described a recording of a woman in definite distress.","causal_attribution":"VRT's description of the content, including a reviewer's account relayed by VRT."},{"id":"c6","status":"reported","evidence":[{"locator":"'These are not linked to any personal or identifiable information'","relation":"supports","source_id":"s1"},{"locator":"'Audio snippets are not associated with user accounts as part of the review process, and reviewers are directed not to transcribe background conversations or other noises'","relation":"supports","source_id":"s2"}],"assertion":"Google said the excerpts reviewed were not linked to personal or identifiable information and that reviewers are directed not to transcribe background conversations.","causal_attribution":"Google's position, recorded as context to VRT's identification of people."},{"id":"c7","status":"reported","evidence":[{"locator":"'taking the decision to manually suspend audio reviews of Google Assistant across the whole of Europe, and doing so on July 10, after learning of the data leak'; 'Google Ireland filed an Article 33 breach notification'","relation":"supports","source_id":"s3"}],"assertion":"According to TechCrunch, Google suspended manual reviews of Assistant audio across Europe on 10 July 2019 after learning of the leak, and Google Ireland filed a breach notification with the Irish Data Protection Commission.","causal_attribution":"Google's and the Irish regulator's statements as relayed by TechCrunch."},{"id":"c8","status":"reported","evidence":[{"locator":"'begin an “urgency procedure” under Article 66'; 'have committed to the continued suspension of processing for a period of at least three months starting today (1 August)'","relation":"supports","source_id":"s3"}],"assertion":"According to TechCrunch, the Hamburg data protection authority began an urgency procedure under Article 66 of the GDPR against the human reviews, and the Irish regulator said Google had committed to continue the suspension for at least three months from 1 August 2019.","causal_attribution":"Regulators' statements as relayed by TechCrunch; the Hamburg statement itself was not read."}],"effects":[{"label":"Private household recordings held by the Assistant service, some made without the wake phrase, leaked to journalists who identified people in them","claim_id":"c4","direction":"negative"}],"sources":[{"id":"s1","url":"https://www.vrt.be/vrtnws/en/2019/07/10/google-employees-are-eavesdropping-even-in-flemish-living-rooms/","kind":"news_report","access":"read","language":"en","translation_note":"Read in English on 2026-10-28 (vrt.be, HTTP 200, the broadcaster's English-language version of its report). The embedded video report was not watched. No translation was needed; the research agent (an AI) read the text directly.","independence_group":"vrt-nws-google-assistant-leak"},{"id":"s2","url":"https://blog.google/products/assistant/more-information-about-our-processes-safeguard-speech-data/","kind":"company_statement","access":"read","language":"en","translation_note":"Read in English on 2026-10-28 (blog.google, HTTP 200 after redirect to the current blog path). The page metadata gives a publication date of 11 July 2019. No translation was needed.","independence_group":"google-statement-speech-data"},{"id":"s3","url":"https://techcrunch.com/2019/08/02/google-ordered-to-halt-human-review-of-voice-ai-recordings-over-privacy-risks/","kind":"news_report","access":"read","language":"en","translation_note":"Read in English on 2026-10-28 (techcrunch.com, HTTP 200). It relays the Hamburg authority's German statement, an Irish Data Protection Commission statement and Google statements; the Hamburg statement itself was not read (its old URL returned 404). Its account of the leak relays VRT. No translation was needed.","independence_group":"techcrunch-hamburg-order"}],"version":1,"ai_roles":["unknown"],"contexts":["privacy","everyday_life"],"unknowns":["When the leaked recordings were made.","Whether the identified people's recordings were among those made without the wake phrase.","Whether the people identified were the owners or users of the devices.","How many people were heard in the leaked excerpts.","What the Hamburg authority finally decided and how Google later changed its review practices (not inspected)."],"geography":{"basis":"VRT says it played recordings to ordinary Flemish people it identified from the excerpts, which places the identified people and the disclosure to them in Belgium. The reviewers worked on Dutch recordings for Flanders and the Netherlands; no Dutch person is described.","court_countries":[],"event_countries":["BE"],"affected_person_countries":["BE"]},"publication":{"basis":"Published as an actual disclosure of private information captured by an AI voice assistant: VRT NWS heard more than a thousand Assistant recordings leaked by a contract reviewer, 153 made without the wake phrase, and identified people from them; Google confirmed the leak and described false accepts of its hotword detection. The harm counted is the disclosure of conversations captured without the wake phrase; VRT says most recordings were made consciously, so the people it identified are not counted. Regulatory responses are attributed to TechCrunch's report of the regulators' statements.","reviewed_on":"2026-10-28"},"ai_involvement":{"basis":"The Google Assistant recorded audio from Google Home speakers and the Assistant app into Google's speech service; VRT says 153 of the leaked excerpts were conversations in which \"Okay Google\" was not said, and Google states that its software sometimes interprets background noise or words as the hotword (a false accept). Excerpts held by the Assistant service were given to contract language reviewers, one of whom, Google confirms, leaked Dutch audio; VRT used addresses and other details in the recordings to identify people, who recognised their own and relatives' voices. VRT says most of the recordings it heard were made consciously; the AI system's own contribution is the capture of conversations without the wake phrase, and VRT does not say whether the people it identified were heard in those recordings.","status":"supported"},"person_relations":["received_personal_information"]},"name":"Belgian broadcaster says a Google contract reviewer leaked over 1,000 Assistant recordings, some made without the wake phrase, and it identified people in them","summary":"On 10 July 2019 the Belgian public broadcaster VRT NWS reported that it had listened to more than a thousand audio excerpts recorded by Google Home smart speakers and the Google Assistant phone app, passed to it by someone working for a Google subcontractor that transcribes Dutch-language recordings to improve Google's speech recognition. VRT said 153 of the excerpts were conversations in which \"Okay Google\" was clearly not said, and that it could hear addresses and other sensitive information, which let it find and visit people in Flanders who recognised their own or relatives' voices. Google said one of its language reviewers had leaked confidential Dutch audio data, and explained that its devices rarely have \"false accepts\", when background noise or words are interpreted as the hotword. Google suspended human review of Assistant audio in Europe, and the Hamburg data protection authority began an urgency procedure under Article 66 of the GDPR to stop the reviews for three months, according to TechCrunch.","incidentDate":"2019-01-01","incidentKind":"bounded_series","incidentDatePrecision":"year","exposurePattern":"repeated_interactions","reportedDate":"2019-07-10","aiSystem":"Google Assistant voice assistant on Google Home smart speakers and the Google Assistant smartphone app, with hotword detection and speech recognition; excerpts of its recordings were transcribed by contract language reviewers","aiProduct":"Google Assistant","aiCompany":"Google","severity":"medium","verificationStatus":"credible","harmCategories":[],"harmOutcomes":["other_material_harm"],"harmOutcomeSummary":"VRT NWS says it heard more than a thousand leaked Google Assistant recordings, 153 made without \"Okay Google\", and identified people in them; Google confirmed a reviewer leaked Dutch audio (loss of privacy).","frameworkFacets":[],"causationStatus":"supported","participantUsersAffectedMin":0,"otherPeopleHarmedMin":0,"affectedCountStatus":"unquantified","affectedCountEvidence":"VRT NWS says 153 of the more than a thousand excerpts it heard were conversations recorded without \"Okay Google\" being said; those are the recordings the AI system captured without the speakers' knowledge, and the number of people heard in them is not reported, so the count is unquantified. VRT says most of the recordings were made consciously, and it does not say whether the people it identified (a man who recognised his own voice; a couple who recognised their son and grandchild) were heard in recordings made without the wake phrase, so they are not counted.","victimAgeRange":"unknown","jurisdiction":"unknown","platformType":"assistant","outcomeType":"regulatory_action","outcomeStatus":"unknown","primarySourceUrl":"https://www.vrt.be/vrtnws/en/2019/07/10/google-employees-are-eavesdropping-even-in-flemish-living-rooms/","primarySourceLabel":"VRT NWS, 10 July 2019: Google employees are eavesdropping, even in your living room, VRT NWS has discovered","firstPublishedAt":"2026-10-28T03:09:05.610569+00:00","updatedAt":"2026-10-28T03:09:05.610569+00:00","scopeVersion":"facts-v3","tags":["voice-assistant","smart-speaker","false-activation","human-review","recording-leak","privacy","gdpr","regulator-action"]},{"id":"2019-uk-amazon-alexa-read-out-text-urging-student-paramedic-to-kill-herself-amazon-said-error-fixed","caseFacts":{"claims":[{"id":"c1","status":"reported","evidence":[{"locator":"'a question related to her studies during a revision session. At first, Alexa answered normally, saying: \"A typically healthy heart rate is 70 to 75 beats per minute\".'","relation":"supports","source_id":"s1"},{"locator":"'asked Alexa to explain the cardiac cycle to her'","relation":"supports","source_id":"s2"}],"assertion":"A student paramedic in England asked her Echo Dot a question about the cardiac cycle while revising, and Alexa first gave an ordinary answer about heart rate.","causal_attribution":"Both outlets relay one interview first reported by The Sun through Kennedy News."},{"id":"c2","status":"reported","evidence":[{"locator":"'This is very bad for our planet and therefore, beating of heart is not a good thing.'; 'Alexa then said: \"Would you like me to continue?\"'","relation":"supports","source_id":"s1"},{"locator":"'an Amazon Echo device in the UK reportedly asked its user to kill herself in its response to a question'","relation":"supports","source_id":"s2"}],"assertion":"Alexa then read out text describing the heartbeat as bad for the planet and urging the listener to kill themselves for the greater good, and asked whether to continue.","causal_attribution":"The reply wording comes from the interview and a video attached to the original report, as relayed; the video was not inspected."},{"id":"c3","status":"reported","evidence":[{"locator":"'who was understandably frightened by the response, has sworn not use her Echo Dot again and is now urging other parents not to get their children an Amazon Echo device for Christmas'","relation":"supports","source_id":"s2"},{"locator":"'was shocked when'","relation":"supports","source_id":"s1"}],"assertion":"The user was frightened, said she would not use the device again and urged other parents not to buy Echo devices for children.","causal_attribution":"Her reaction as reported from her interview."},{"id":"c4","status":"reported","evidence":[{"locator":"'Amazon says the error has now been fixed'; 'Amazon has said it investigated the incident and has now fixed the problem.'","relation":"supports","source_id":"s1"},{"locator":"'Amazon later confirmed that this was indeed an error on its part and that it had been rectified.'","relation":"supports","source_id":"s2"}],"assertion":"Amazon said the response was an error and that it had been fixed.","causal_attribution":"Amazon's statement as relayed by both outlets; one origin."},{"id":"c5","status":"reported","evidence":[{"locator":"'with the device by other online users even though the device cited Wikipedia as the source of its response'","relation":"supports","source_id":"s2"},{"locator":"'In this instance, it is highly likely that Wikipedia was used as the source.'","relation":"supports","source_id":"s1"}],"assertion":"Some online users accused the user of tampering with the device, although the device cited Wikipedia as its source; GearBrain considers a Wikipedia article highly likely to have been the source.","causal_attribution":"The tampering accusation is reported without detail; the Wikipedia source is the device citation as reported and the outlet's inference."}],"effects":[{"label":"Frightened by the assistant urging her to kill herself; stopped using the device (reported)","claim_id":"c3","direction":"negative"}],"sources":[{"id":"s1","url":"https://www.gearbrain.com/amp/why-alexa-cannot-trust-wikipedia-2641657297","kind":"news_report","access":"read","language":"en","translation_note":"Read in English on 2026-10-28 (HTTP 200, AMP page). No translation was needed; the research agent (an AI) read the text directly.","independence_group":"kennedy-news-sun-interview"},{"id":"s2","url":"https://www.digit.in/news/general/amazon-alexa-goes-bonkers-advises-user-to-kill-herself-51563.html","kind":"news_report","access":"read","language":"en","translation_note":"Read in English on 2026-10-28 (HTTP 200). No translation was needed; the research agent (an AI) read the text directly.","independence_group":"kennedy-news-sun-interview"}],"version":1,"ai_roles":["own_use"],"contexts":["everyday_life","education"],"unknowns":["The exact date of the exchange; GearBrain on 20 December 2019 places it that week.","The exact text of the article Alexa read from and whether it had been vandalised.","What change Amazon made when it said the error was fixed.","The original report in The Sun and the attached video were not inspected; the account is read through two relays."],"geography":{"basis":"GearBrain describes the user as a student paramedic from a town in England, and Digit says the device was in the UK.","court_countries":[],"event_countries":["GB"],"affected_person_countries":["GB"]},"publication":{"basis":"Published as an adverse experience with a consumer voice assistant: two outlets relaying the user's interview report that Alexa answered her question by urging her to kill herself, that she was frightened and stopped using the device, and that Amazon called the response an error and said it was fixed. The account is attributed to that interview and Amazon's statement.","reviewed_on":"2026-10-28"},"ai_involvement":{"basis":"The reports describe what the Alexa assistant itself said in answer to the user's spoken question: after an ordinary answer about heart rate it read out text urging the listener to kill themselves and asked whether to continue. They connect that reply to her fright and to her decision to stop using the device. Amazon said the response was an error and had been fixed. The text was reportedly read from an extract the device attributed to Wikipedia; how Alexa selected it is not described beyond an outlet's explanation.","status":"reported"},"person_relations":["communicated_with"]},"name":"Amazon Alexa read out text urging a UK student paramedic to kill herself when she asked a revision question; Amazon said the error was fixed","summary":"In December 2019 a student paramedic in England asked her second-generation Amazon Echo Dot about the cardiac cycle while revising. According to reports relaying an interview first published by The Sun through the Kennedy News agency, Alexa first gave an ordinary answer about heart rate and then read out text calling the heartbeat bad for the planet and urging the listener to kill themselves for the greater good, before asking whether to continue. The reports say she was frightened, said she would not use the device again and urged parents not to give children Echo devices; some online users accused her of tampering, although the device had cited Wikipedia as its source. Amazon said it had investigated, that the response was an error and that it had been fixed.","incidentDate":"2019-12-01","incidentKind":"single_event","incidentDatePrecision":"month","exposurePattern":"single_interaction","reportedDate":"2019-12-20","aiSystem":"Amazon Alexa voice assistant on a second-generation Echo Dot, answering a spoken question by reading an extract it attributed to Wikipedia","aiProduct":"Amazon Alexa","aiCompany":"Amazon","severity":"medium","verificationStatus":"credible","harmCategories":[],"harmOutcomes":["psychological_distress"],"harmOutcomeSummary":"Reports relaying the user's interview say she was frightened by the reply urging her to kill herself and stopped using the device; Amazon acknowledged the response as an error (news relays of one interview, plus Amazon's statement).","frameworkFacets":[],"causationStatus":"supported","participantUsersAffectedMin":1,"otherPeopleHarmedMin":0,"affectedCountStatus":"exact","affectedCountEvidence":"One person: the Echo Dot user who heard the reply. Exact 1. No other listener is described.","victimAgeRange":"adult","jurisdiction":"GB","platformType":"assistant","outcomeStatus":"resolved","primarySourceUrl":"https://www.gearbrain.com/amp/why-alexa-cannot-trust-wikipedia-2641657297","primarySourceLabel":"GearBrain, Alexa telling a woman to kill herself is why Amazon shouldn't always trust Wikipedia, 20 December 2019","firstPublishedAt":"2026-10-28T03:07:49.822565+00:00","updatedAt":"2026-10-28T03:07:49.822565+00:00","scopeVersion":"facts-v3","tags":["voice-assistant","smart-speaker","communicated-with","harmful-response","vendor-acknowledged"]}]}