Google's AI flagged two fathers' medical photos of their toddlers as child abuse; accounts disabled, police investigated then cleared them, NYT reports

The photos. On a Friday night in February 2021 a San Francisco father photographed his toddler son's medical condition with his Android phone. A nurse at the family's health-care provider asked for photos so a doctor could review them before an emergency video consultation; the doctor diagnosed the problem and prescribed antibiotics, which cleared it.

The flag and the account. The New York Times reports that when the photos were backed up to Google's servers, the artificial-intelligence tool Google developed in 2018 to recognise never-before-seen child abuse images flagged them, and that a human content moderator would have reviewed flagged photos before Google locked the account and reported it to the National Center for Missing and Exploited Children. Two days after the photos were taken, his account was disabled for 'harmful content' described as a severe violation of Google's policies that might be illegal. He lost emails, contacts and documentation of his son's first years, and because his phone plan was Google Fi he also lost his phone number, which locked him out of other accounts that sent security codes. Google rejected his appeal without explanation.

The police investigation. Google's review team also flagged a video in his account. In December 2021 he received a letter from the San Francisco Police Department telling him he had been investigated and enclosing search warrants served on Google and his internet provider in February 2021. The investigator wrote that the incident did not meet the elements of a crime and that no crime had occurred. A second appeal to Google with the police report failed, and he was later notified that his account was being permanently deleted.

A second father. The Times reports that the same thing happened the next day to a father in Houston who photographed his toddler's infection at a paediatrician's request. His Gmail account was disabled while he was buying a house, which made his mortgage broker suspicious; Houston police called him in during the autumn of 2021 and quickly cleared him, and he could not recover his account.

Google's position. Google told the Times and The Verge that it uses hash matching and artificial intelligence to identify such material, that its child-safety team reviews flagged content and consults paediatricians, and that it stood by its decisions although law enforcement cleared both men. It told the Times that its reviewers had not detected a rash or redness in the San Francisco father's photos and that a further video found in his account was also considered problematic; he said he did not remember it.

Limits. Both fathers are identified in the reporting only by first name. The photos and Google's notices were not inspected. Whether the human reviewer's decision added anything to the classifier's flag in these two cases is not stated beyond the Times' description of the general process.

The New York Times reports that Google's AI flagged two fathers' medical photos of their toddlers as child abuse material, after which Google disabled their accounts (losing years of email, photos and, for one, his phone number) and police investigated both before clearing them; Google refused to reinstate the accounts.

San Francisco police obtained search warrants for the father's Google account and his internet provider; the investigator wrote that the incident did not meet the elements of a crime. Houston police questioned the second father and cleared him. Google refused to reinstate either account and told the Times it stood by its decisions; the San Francisco father was notified that his account would be permanently deleted and had spoken with a lawyer about suing. No later reinstatement or lawsuit is reported in the sources read.