{"meta":{"exportedAt":"2026-11-02T07:20:25.873Z","formatVersion":2,"selection":{"q":"medical-photos","system":"","harm":"","context":"","country":"","role":"","relation":"contextual","evidence":"","year":"","response":"","severity":"","verification":"","view":"incidents","sort":"added"},"totalIncidents":1,"coverage":{"cases":1,"countries":1,"languages":1,"unknownLocation":0,"locationPending":0,"unknownLanguage":0,"unknownDate":0,"lawsuits":0,"regulatory":1,"minors":0,"coreRelations":0,"contextualRelations":1,"mixedRelations":0,"unknownRelations":0,"relationPending":0,"relationUnknown":0},"countingNote":"Distinct public cases in this selection. People counts apply within individual cases only; cross-case person overlap has not been resolved. No population incidence estimate.","affectedCountNote":"Interpret person counts with affectedCountStatus and the reported effects. Unquantified zeros are placeholders, not a measured zero.","source":"AI incidents","publisher":"NOPE","url":"https://nope.net/incidents","license":"CC BY 4.0"},"incidents":[{"id":"2021-us-google-ai-flagged-fathers-medical-photos-of-toddlers-as-child-abuse-accounts-disabled-police-cleared","caseFacts":{"claims":[{"id":"c1","status":"reported","evidence":[{"locator":"'It was a Friday night in February 2021.'; 'The nurse said to send photos so the doctor could review them in advance.'; 'the doctor diagnosed the issue and prescribed antibiotics'","relation":"supports","source_id":"s1"},{"locator":"'The doctor used that image to diagnose'","relation":"supports","source_id":"s3"}],"assertion":"In February 2021 a San Francisco father photographed his toddler son's medical condition after a nurse asked for photos before a video consultation, and the doctor used them to diagnose the problem and prescribe antibiotics.","causal_attribution":"Background facts from the father's account to the Times."},{"id":"c2","status":"reported","evidence":[{"locator":"'Google developed an artificially intelligent tool that could recognize never-before-seen exploitative images of children'; 'this technology flagged them'; 'A human content moderator for Google would have reviewed the photos after they were flagged by the artificial intelligence'","relation":"supports","source_id":"s1"},{"locator":"'automatically uploaded to the cloud, Google’s system identified them as CSAM'","relation":"supports","source_id":"s3"}],"assertion":"When the photos were backed up to Google's servers, Google's AI tool for recognising never-before-seen child abuse images flagged them; a human moderator reviews flagged photos before Google locks an account and reports it.","causal_attribution":"The Times attributes the flag to Google's AI tool; the human review step is described as the general process."},{"id":"c3","status":"reported","evidence":[{"locator":"'His account had been disabled because of “harmful content” that was “a severe violation of Google’s policies and might be illegal.”'; 'he had to get a new phone number with another carrier'","relation":"supports","source_id":"s1"},{"locator":"'ended up losing access to his emails, contacts, photos, and even his phone number'","relation":"supports","source_id":"s2"}],"assertion":"Two days after the photos were taken his Google account was disabled for harmful content; he lost email, contacts and photos, and his Google Fi phone number.","causal_attribution":"The account loss followed directly from Google's flag and decision."},{"id":"c4","status":"reported","evidence":[{"locator":"'copies of the search warrants served on Google and his internet service provider'; '“I determined that the incident did not meet the elements of a crime and that no crime occurred,”'","relation":"supports","source_id":"s1"},{"locator":"'the San Francisco police department opened an investigation into him'","relation":"supports","source_id":"s3"}],"assertion":"San Francisco police obtained search warrants for his Google account and internet provider, and the investigator concluded that no crime had occurred.","causal_attribution":"The Times quotes the investigator's report; the report itself was not inspected."},{"id":"c5","status":"reported","evidence":[{"locator":"'Google responded that it would not reinstate the account, with no further explanation.'; 'appealed his case to Google again, providing the police report, but to no avail'; 'his account was being permanently deleted'","relation":"supports","source_id":"s1"}],"assertion":"Google rejected his appeals, including one with the police report, and later notified him that the account was being permanently deleted.","causal_attribution":"Reported from the father's account to the Times."},{"id":"c6","status":"reported","evidence":[{"locator":"'the same scenario was playing out in Texas'; 'which made the broker suspicious until'; 'A detective from the Houston Police department called in the fall of 2021, asking him to come into the station.'; 'But he, too, was unable to get his decade-old Google account back'","relation":"supports","source_id":"s1"}],"assertion":"A Houston father whose photos of his toddler, taken at a paediatrician's request, were flagged the next day had his Gmail account disabled while buying a house; police called him in and cleared him, and he could not recover the account.","causal_attribution":"Reported by the Times from the second father's account."},{"id":"c7","status":"reported","evidence":[{"locator":"'A Google spokeswoman said the company stands by its decisions, even though law enforcement cleared the two men.'; 'reviewers had not detected a rash or redness in the photos he took'","relation":"supports","source_id":"s1"},{"locator":"'use a combination of hash matching technology and artificial intelligence to identify it and remove it from our platforms'; 'reviews flagged content for accuracy and consults with pediatricians'","relation":"supports","source_id":"s2"},{"locator":"'Google has said it will stand by its decision'","relation":"supports","source_id":"s3"}],"assertion":"Google says it uses hash matching and artificial intelligence to identify child sexual abuse material, that its child-safety team reviews flagged content and consults paediatricians, and that it stands by its decisions although law enforcement cleared both men; it told the Times its reviewers saw no rash or redness and found a further video it considered problematic.","causal_attribution":"Google's account disputes that the decisions were wrong."}],"effects":[{"label":"Google accounts disabled and police investigations after AI flagged medical photos as child abuse material","claim_id":"c3","direction":"negative"}],"sources":[{"id":"s1","url":"https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html","kind":"news_report","access":"read","language":"en","translation_note":"Read in English by the research agent (an AI) on 2026-11-02 through the Internet Archive capture web.archive.org/web/20221231230516id_/ of the article (HTTP 200; the live page returned 403); no translation and no human reviewer.","independence_group":"nyt-aug-2022-google-flagged-fathers-report"},{"id":"s2","url":"https://www.theverge.com/2022/8/21/23315513/google-photos-csam-scanning-account-deletion-investigation","kind":"news_report","access":"read","language":"en","translation_note":"Read in English by the research agent (an AI) on 2026-11-02 (HTTP 200); no translation. Relays the Times report; the Google statement it carries was emailed to The Verge.","independence_group":"nyt-aug-2022-google-flagged-fathers-report"},{"id":"s3","url":"https://www.theguardian.com/technology/2022/aug/22/google-csam-account-blocked","kind":"news_report","access":"read","language":"en","translation_note":"Read in English by the research agent (an AI) on 2026-11-02 (HTTP 200); no translation. Relays the Times report and carries a Google spokesperson statement.","independence_group":"nyt-aug-2022-google-flagged-fathers-report"}],"version":1,"ai_roles":["institutional_use"],"contexts":["privacy","health","justice","everyday_life"],"unknowns":["Whether either account was ever reinstated or either father sued after August 2022.","What the human reviewer saw and decided in each case, beyond the general process the Times describes.","The content of the further video Google says it found, which the father does not remember."],"geography":{"basis":"The Times places the first father in San Francisco and his investigation with the San Francisco Police Department, and the second in Houston, Texas, questioned by Houston police.","court_countries":[],"event_countries":["US"],"affected_person_countries":["US"]},"publication":{"basis":"Google's AI classifier made a claim about each father's photos that informed Google's decision to disable his account and report him, after which police investigated and cleared him. The account rests on the New York Times report and Google's own statements, which dispute that the decisions were wrong.","reviewed_on":"2026-11-02"},"ai_involvement":{"basis":"The New York Times reports that Google's artificial-intelligence tool for recognising never-before-seen child abuse images flagged the fathers' medical photos when they were backed up to Google's servers. That flag led Google to disable their accounts and report them, which led to the police investigations. Google confirms it uses AI alongside hash matching and that human reviewers check flagged content, so the AI flag informed the human decision.","status":"reported"},"person_relations":["made_claim_about","informed_decision_about"]},"name":"Google's AI flagged two fathers' medical photos of their toddlers as child abuse; accounts disabled, police investigated then cleared them, NYT reports","summary":"In February 2021 a San Francisco father took photos of his toddler son's medical condition after a nurse asked for photos before a video consultation; the doctor used them to diagnose the problem and prescribe antibiotics. The New York Times reported in August 2022 that when the photos were backed up from his Android phone to Google's servers, Google's artificial-intelligence tool for recognising never-before-seen child abuse images flagged them. Two days after the photos were taken his Google account was disabled for 'harmful content', which cost him more than a decade of email, contacts and photos and, through Google Fi, his phone number. Google reported him to the National Center for Missing and Exploited Children, and the San Francisco police obtained search warrants for his account; the investigator concluded that no crime had occurred. Google refused to reinstate the account after two appeals, the second with the police report, and later notified him that it was being permanently deleted. The Times reported a second father in Houston whose photos of his toddler, taken at a paediatrician's request, were also flagged; his Gmail account was disabled in the middle of buying a house, Houston police questioned and cleared him, and he did not get the account back. Google said it stood by its decisions and that its reviewers had found a further video in the San Francisco father's account that it considered problematic.","incidentDate":"2021-02-01","incidentKind":"bounded_series","incidentDatePrecision":"month","exposurePattern":"single_interaction","reportedDate":"2022-08-21","aiSystem":"Google's machine-learning tool for recognising never-before-seen child sexual abuse images (used with hash matching), applied to photos backed up from Android phones to Google's servers, followed by human review","aiProduct":"Google CSAM image classifier","severity":"medium","verificationStatus":"credible","harmCategories":[],"harmOutcomes":["legal_harm","other_material_harm"],"harmOutcomeSummary":"The New York Times reports that Google's AI flagged two fathers' medical photos of their toddlers as child abuse material, after which Google disabled their accounts (losing years of email, photos and, for one, his phone number) and police investigated both before clearing them; Google refused to reinstate the accounts.","frameworkFacets":[],"causationStatus":"supported","participantUsersAffectedMin":2,"otherPeopleHarmedMin":0,"affectedCountStatus":"exact","affectedCountEvidence":"Two account holders: the San Francisco and Houston fathers whose accounts were disabled and who were investigated. Their sons, counted by the CyberTipline among potential new child victims, and the San Francisco father's wife are not counted as harmed people.","victimAgeRange":"adult","jurisdiction":"US","platformType":"other","outcomeType":"investigation_opened","outcomeStatus":"unknown","primarySourceUrl":"https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html","primarySourceLabel":"The New York Times, 21 Aug 2022: Google flagged a father as a criminal over medical photos of his toddler taken for the doctor","firstPublishedAt":"2026-11-02T03:05:51.915854+00:00","updatedAt":"2026-11-02T03:05:51.915854+00:00","scopeVersion":"facts-v3","tags":["google","child-safety-classifier","account-disabled","police-investigation","medical-photos","privacy","false-positive"]}]}