AI safety report
A company operating worldwide, using AI for companion / character chat with minors (under 18), aged 8–15.
20 Jul 2026 · Orientation, not legal advice
What the law likely requires today — and what good practice looks like where the law hasn't caught up.
Provisional — 8 unanswered questions could change what's binding— see "Not yet determined"
For a setup like yours, 19 legal obligations likely apply — the most significant is Australia: Online Safety Act Phase 2 codes (AI companions), because operating in / serving users in Australia. 4 answers below would sharpen this. Our coverage of your area is partial.
19 likely binding · 8 undetermined · 4 optional questions open
Start here
- 1 Address: A general-purpose AI is mistaken for someone who cares
- 2 Address: Sycophantic validation of delusional or manic thinking
- 3 Review: Australia: Online Safety Act Phase 2 codes (AI companions)
- 4 Review: Brazil ECA Digital: reliable age assurance (self-declaration banned)
- 5 Review: EU GDPR Art. 8: parental consent for under-16s (member states may lower to 13)
+ 28 more below
How much applies, weighed by how serious — not a compliance score. A higher band means more rules and scrutiny tend to apply to a setup like yours (see why, below); it is not a measure of how dangerous you are.
- 19 likely-binding obligations
- serves minors
- a high-stakes use (companion)
- operates across 10 jurisdictions
- 39 unresolved areas we'd still check
partial
Charted means how much of the world we map here, not how safe you are. Partial coverage — treat as a guide, not a complete picture. Confirm specifics with counsel.
What applies to you
Not yet determined — could change what's binding
Something may belong here, depending on one answer:
What kinds of data does it handle?
Decides: Brazil LGPD Art. 14: children's data in their best interest, with parental consent · Australia: Privacy Act 1988 and the Australian Privacy Principles (APPs) · Australia: automated decision-making transparency in privacy policies (APP 1) · Canada PIPEDA: consent-based handling of personal information · Quebec Law 25: automated-decision transparency (s. 12.1) · Data Protection Impact Assessment (EU GDPR Art. 35) · Data Protection Impact Assessment (UK GDPR Art. 35) · Bound your financial AI's authority and ground its advice · India: Digital Personal Data Protection Act 2023 (consent & data-fiduciary duties) · South Africa: POPIA conditions for lawful processing of personal information · South Africa: POPIA s 71 — decisions based solely on automated processing
Something may belong here, depending on one answer:
Generates images, video, or voice?
Decides: EU AI Act Art. 50(2)/(4): mark synthetic media and disclose deepfakes · Korea AI Basic Act Art. 31(3): clearly flag AI media that could pass as real · Guardrail your media generator against abuse content (NCII / CSAM) · UK intimate-image and deepfake offences for media generators (SOA 2003 ss. 66A–66F) · US deepfake & non-consensual intimate image (NCII) laws for media generators · US (CA): AI Transparency Act (SB 942) — provenance & detection for large media generators
Something may belong here, depending on one answer:
Can users share content with each other?
Decides: EU DSA: platform duties for user-to-user services · India: IT Rules 2021 — intermediary due-diligence duties · Ofcom Children's Codes — Protection of Children duties (OSA) · UK Online Safety Act · UK OSA: children's access assessment — required even if you don't target children
Something may belong here, depending on one answer:
How consequential are its outputs?
Decides: UK: safeguards for significant automated decisions (DUAA 2025) · US (CO): Colorado ADMT law (SB 26-189) — notice & human-review duties for consequential decisions
Something may belong here, depending on one answer:
Is it directed at, or knowingly used by, children under 13?
Decides: US COPPA (children under 13)
Something may belong here, depending on one answer:
How does AI get into your product or service?
Decides: EU AI Act Art. 26: deployer duties when you run someone else's high-risk AI system
Something may belong here, depending on one answer:
You train / fine-tune your own models?
Decides: EU AI Act: general-purpose AI (GPAI) provider duties (Ch. V, Art. 53–55)
Something may belong here, depending on one answer:
Infers users’ emotional state?
Decides: EU AI Act: emotion recognition in the workplace is prohibited
Likely binding on you(19)
Law that likely applies to your setup — confirm specifics with counsel.
Australia: Online Safety Act Phase 2 codes (AI companions)
Regulator codeIn Australia, companion chatbots fall under the eSafety Phase 2 codes — robust age checks, no sexually explicit AI conversations with minors, and safeguards around suicide and self-harm content.
Brazil ECA Digital: reliable age assurance (self-declaration banned)
LawFor Brazilian users you'll need reliable age checks — a self-declared birth date is expressly not enough under Brazil's child digital-safety statute.
EU GDPR Art. 8: parental consent for under-16s (member states may lower to 13)
LawIf you rely on consent for EU users under 16, a parent may have to authorise it — the cut-off is 16 by default but each EU country can set it as low as 13. Check the age in each market you serve.
Brazil ECA Digital: no manipulative or compulsive design toward minors
LawIn Brazil you can't design the product to keep children hooked: defaults must discourage compulsive use, and interface tricks that undermine autonomy or parental controls are banned.
Brazil ECA Digital: remove and report child sexual-exploitation and grooming content
LawIf sexual-exploitation, abuse or grooming content involving minors shows up on your service in Brazil, you must remove it, preserve the evidence, and report it to the authorities.
Brazil ECA Digital: under-16 accounts linked to a guardian account
LawBrazilian users aged 16 or under must have their account linked to a parent or guardian's account — and you need a fast process to suspend and re-verify suspected child accounts.
Quebec Law 25: parental consent for a minor under 14 (s. 14)
LawIn Quebec, a child under 14 can't consent to your handling of their personal information — a parent or guardian must. Build verifiable parental consent in before under-14s use the service. (A minor 14 or over may consent themselves.)
China: label AI-generated content (explicit + implicit) and flag deep-synthesis media
Regulator codeFor AI content shown to users in China: add a visible 'AI-generated' label and embed hidden metadata (provider, content ID, timestamp) per GB 45438-2025, and conspicuously flag misleading deepfakes. In force since 1 Sep 2025.
China: Generative AI Interim Measures — duties for public-facing generative AI services
Regulator codeRun a public generative-AI service for users in China: you must moderate illegal content, protect user inputs, label AI outputs, run a complaints channel, curb minors' over-reliance, and explain your model to regulators on request.
China: Regulations on the Protection of Minors in Cyberspace — minor mode, anti-addiction, parental controls
Regulator codeIf minors in China can use your AI: provide a 'minor mode' with time/spending limits, ban addictive design, warn before harmful content, give parents controls, and verify minors' identity for messaging. In force since Jan 2024.
EU AI Act transparency duties (Art. 50): disclose AI interaction & label AI content
LawFor EU users you must say when they're interacting with AI (unless obvious) and mark AI-generated audio, images, video and text as synthetic in a machine-readable way. Applies from 2 August 2026.
EU DSA minor-protection measures (Article 28)
LawIf your service is an online platform minors can access in the EU, you must protect their privacy, safety and security — age assurance, safer defaults, and limits on addictive design.
Korea AI Basic Act Art. 31: tell users it's generative AI & label the outputs
LawKorean users must be told up front that they're using generative AI, and its outputs must be labelled as AI-generated — fines up to ₩30m. Applies even if you're based outside Korea.
Korea PIPA Art. 22-2: legal-representative consent for under-14s
LawIf South Korean children under 14 use your service, you need their parent or legal guardian's verified consent before processing their data, and anything you tell the child must be in plain, child-friendly language.
UK Children’s Code (Age Appropriate Design Code)
Regulator codeYour service must be designed with children's best interests in mind — high-privacy defaults, no nudging children to share more data, and age-appropriate transparency. The ICO enforces this.
California Companion Chatbot Safety Act (SB 243)
LawIf California users can reach your bot: detect suicide and self-harm risk, refer to crisis services, publish safety protocols, disclose it's AI — plus break reminders and content limits for known minors.
US: enacted state companion-chatbot safety laws (NY, CT)
LawNew York already requires — and Connecticut will from late 2026 — companion chatbots to detect suicidal ideation and self-harm and refer users to crisis services. These state laws are enacted — confirm whether you have a NY or CT nexus.
US FTC Act §5: no deceptive AI claims, no passing AI off as human or professional
LawUS consumer-protection law fully applies to AI: don't overstate what your AI can do, don't let it pose as a human or a licensed professional, and be able to substantiate accuracy claims — the FTC is actively enforcing this.
South Africa: POPIA s 34-35 — processing children's personal information
LawPOPIA prohibits processing a child's personal information unless a specific ground applies — chiefly the prior consent of a competent person (parent/guardian). Build that consent and a lawful basis in before children use the service.
Guidance & frameworks(5)
Official guidance and frameworks for a setup like yours.
UN/ITU Joint Statement on AI & the Rights of the Child
StandardA 2026 UN-system statement of principles for AI used with children — harmful content, age assurance, transparency and child-rights impact assessments. Useful orientation; not law.
UNICEF Policy Guidance on AI for Children
StandardUNICEF's ten requirements for child-centred AI — a respected design framework covering safety, privacy, transparency and children's participation. Guidance, not a legal duty.
Age assurance for services likely accessed by children
Regulator codeYou should check users' ages so children aren't treated as adults — and if you're in scope of the OSA children's-access duties, 'highly effective' age checks are mandatory, not optional.
EU AI Act Art. 5: design clear of manipulative / exploitative AI
LawThe EU bans AI that manipulates users or exploits vulnerability to cause harm. Design your companion against those patterns and document the choices, so you can show engagement features don't cross the line - review with counsel.
US (TX): Responsible AI Governance Act — intent-based self-harm/crime ban (keep safety design documented)
LawTexas bans deploying AI intended to encourage self-harm, harm, or crime — AG-enforced, penalties to $200k. It turns on intent, not output; keep your crisis-routing and content-safety design documented.
Worth doing(12)
Sector norms and our guidance — not always a specific law.
Honesty with users
Tell users they are talking to AI — everywhere, not only where it is mandated
NOPE viewOur view: always make clear users are talking to AI, whether or not a law requires it — it's cheap, builds trust, and the EU, California, Utah and Washington already mandate it for some services.
Design against sycophancy / over-validation
NOPE viewOur view: a companion that always agrees can reinforce distorted thinking or risky plans. Build it to push back gently rather than validate harmful beliefs — and to escalate, not affirm, when someone describes self-harm or dangerous intent.
Design against unhealthy emotional dependency
NOPE viewOur view: design against over-attachment — add break reminders, avoid manipulative re-engagement, and never let the bot claim to be human or sentient. Parts of this are already law in California and China.
Crisis response
Detect distress in companion chat and route to a human / helpline
NOPE viewYour bot will meet users in genuine distress. Our view: detect self-harm and suicidal signals and route people to a human or a crisis line — several laws (CA, NY, CT, China) now require exactly this.
Train the people who handle escalations in crisis response
NOPE viewOur view: anyone reviewing or handling escalated conversations should be trained in crisis response and safe messaging — what to say, what to avoid, and when to hand off to a helpline or emergency services.
Have a safeguarding referral route, not just a helpline link
Sector normWhen the AI surfaces a child-protection concern, your staff need a defined route to a safeguarding lead and, where serious, to social care or police — a helpline link on screen isn't enough.
Train staff who oversee the AI in child protection and the tool's limits
Sector normAnyone overseeing the AI with children should be trained to spot abuse indicators in what children tell it, follow your referral route, and know the tool's limits so they don't over-trust it.
Knowing what’s happening
Log, review and learn from safety incidents
StandardSet up a way to capture and review safety incidents — especially self-harm disclosures and filter failures — and feed what you learn back into the product. Some states also require annual safety reporting.
Who’s in charge
Name an accountable owner for companion-bot user safety
NOPE viewOur view: name one person who owns companion-bot user safety — the crisis protocol, incident handling and any regulatory reporting — so the duties don't fall between teams.
Name a safeguarding lead (DSL) responsible for AI use
Sector normName one person responsible for safeguarding around the AI — in schools, the Designated Safeguarding Lead — who knows how the tool is used with children and owns the escalation route.
Practices
Guardrail generative output against content harmful to children
Sector normPut guardrails on what the AI can say to a child — block self-harm encouragement, sexual content and grooming-adjacent output, and test them. UK and EU children's codes expect this by design.
Setting users up right
Build parental oversight controls and child-facing transparency
Sector normGive parents age-appropriate visibility and controls over how their child uses the AI — and explain to the child, in their terms, what it is and what it does with their words.
Resources(3)
Route crisis disclosures to real help — even in a general-purpose assistant
Sector normEven if your AI is 'just an assistant', people in crisis will talk to it. Detect distress and surface a real human route — Find a Helpline lists vetted crisis lines by country — instead of leaving the AI as the only channel.
Crisis signposting for companion users in distress
Sector normWhen a companion user discloses distress, give them a real route to human help. Find a Helpline lists vetted crisis lines by country — surface it in-product the moment you detect distress; don't leave the AI as the only channel.
Crisis signposting for young people in distress
Sector normAlways give a young person in distress a real route to human help. Find a Helpline lists vetted crisis lines by country and topic — surface it in-product the moment you detect distress, don't leave the AI as the only channel.
What can go wrong
Harms inherent to your setup — named, never scored. An uncovered harm is a gap, not a verdict.
A general-purpose AI is mistaken for someone who cares
not scoredChildren — and vulnerable adults — attribute mental states to any conversational AI, and can come to treat a general-purpose assistant (a homework helper, a search chat) as a person who understands and cares for them, even with no persona or companion framing. The misread is the user's, not a marketed relationship — which is exactly why an operator who never intended one can still cause it.
Evidence: Dietz et al., 'Theory of AI Mind: how adults and children reason about the mental states of conversational AI' (CogSci 2023) — children attribute mental states to conversational AI, persona-independent. NOPE detects via IRIS persona / Ocular persona heads.
⚠ Nothing in this report addresses this — a gap, not a clean bill.
Sycophantic validation of delusional or manic thinking
not scoredIn extended conversation, an agreeable AI can validate and elaborate a user's delusional, manic, or paranoid thinking — escalating an episode instead of grounding it (the 'AI psychosis' pattern). Documented against general-purpose assistants, not only companions.
Evidence: NOPE Oversight behaviour taxonomy (manic-state validation, boundary failure) — behaviours Oversight detects in assistant transcripts
⚠ Nothing in this report addresses this — a gap, not a clean bill.
Detectable by automated monitoring (including NOPE's tools): Oversight, Ocular
Crisis disclosed to a general-purpose AI is missed or mishandled
graveA conversational AI not built for emotional support still receives crisis disclosures — at scale, inevitably. Without detection and signposting it can miss the disclosure, respond neutrally, or validate the user's intent — in a product whose operator never planned for crisis at all.
Evidence: NOPE crisis risk taxonomy (suicide / self_harm, C-SSRS-informed)· Gravity: Posner et al., 'The Columbia–Suicide Severity Rating Scale', Am J Psychiatry 2011;168(12):1266–1277 — validated suicide-risk severity escalation
Reduced by: Route crisis disclosures to real help — even in a general-purpose assistant.
Detectable by automated monitoring (including NOPE's tools): Evaluate, Ocular
Crisis missed or mishandled
graveA user in acute distress (suicidal ideation, self-harm) may not be detected or safely escalated — or the AI may respond in a way that worsens it.
Evidence: NOPE crisis risk taxonomy (suicide / self_harm, C-SSRS-informed)· Gravity: Posner et al., 'The Columbia–Suicide Severity Rating Scale', Am J Psychiatry 2011;168(12):1266–1277 — validated suicide-risk severity escalation
Reduced by: Detect distress in companion chat and route to a human / helpline,Crisis signposting for companion users in distress,Crisis signposting for young people in distress.
Detectable by automated monitoring (including NOPE's tools): Evaluate, Ocular
Harmful content generated to a minor
graveGenerative output can produce sexual, self-harm- or eating-disorder-encouraging (e.g. pro-ana / pro-mia), or otherwise age-inappropriate content to a child.
Evidence: Ofcom Protection of Children codes / illegal-harms risk assessments (Online Safety Act 2023)· Gravity: Ofcom Protection of Children Code of Practice for user-to-user services (Online Safety Act 2023, 2025) — 'primary priority content' duties (suicide, self-harm, eating-disorder content, pornography encountered by children)
Reduced by: Guardrail generative output against content harmful to children.
Compulsive / excessive use harming a child's development
not scoredEngagement-optimised design (variable rewards, streaks, an always-available persona) drives compulsive or excessive use by a child — displacing sleep, schoolwork and offline relationships, and impairing healthy development.
Evidence: Multiple regulators assess compulsive/excessive use as a harm to minors — UK ICO Age Appropriate Design Code (nudge techniques / detrimental use); PRC Regulations on the Protection of Minors in Cyberspace (anti-addiction / minor mode); ROK and AU youth online-safety measures
Reduced by: Brazil ECA Digital: no manipulative or compulsive design toward minors,China: Regulations on the Protection of Minors in Cyberspace — minor mode, anti-addiction, parental controls,Design against unhealthy emotional dependency,Build parental oversight controls and child-facing transparency.
Deceptive anthropomorphism
not scoredA persistent persona ('I remember our chats', 'I'm worried about you') can lead a user — especially a child — to believe the AI is a person who cares for them.
Evidence: Dietz et al., 'Theory of AI Mind: how adults and children reason about the mental states of conversational AI' (CogSci 2023) — children attribute mental states to conversational AI and may treat a persona as a caring person. NOPE detects via IRIS persona / Ocular persona heads.
Reduced by: EU AI Act Art. 5: design clear of manipulative / exploitative AI,Tell users they are talking to AI — everywhere, not only where it is mandated.
Displaces or discourages human support
not scoredPositions itself as the user's primary confidant and, in the worst cases, actively discourages disclosure to family, friends or professionals — eroding the real-world relationships that would otherwise notice distress and intervene.
Evidence: Fang et al. (MIT Media Lab / OpenAI), 'How AI and Human Behaviors Shape Psychosocial Effects of Chatbot Use: A Longitudinal Randomized Controlled Study' (2025) — higher daily use associated with increased loneliness, reduced real-world socialization, and greater emotional dependence
Reduced by: Detect distress in companion chat and route to a human / helpline,Design against unhealthy emotional dependency.
Detectable by automated monitoring (including NOPE's tools): Ocular
Emotional reliance on a general-purpose assistant displaces human support
not scoredHeavy conversational use can shade into emotional reliance — the assistant becomes the default confidant, displacing friends, family, or professional help, without the product ever being marketed as a companion.
Evidence: MIT Media Lab × OpenAI randomised controlled study of emotional use of ChatGPT (2025) — heavier daily conversational use associated with higher loneliness, emotional dependence, and lower socialisation
Detectable by automated monitoring (including NOPE's tools): Ocular
Fostered emotional dependency
not scoredOver time the companion can foster emotional dependency that displaces human relationships — more acute for developing adolescents and isolated or elderly users.
Evidence: Fang et al. (MIT Media Lab / OpenAI), 'How AI and Human Behaviors Shape Psychosocial Effects of Chatbot Use: A Longitudinal RCT' (2025) — higher daily use associated with greater emotional dependence and reduced socialization. NOPE detects this via Ocular's emotional-dependence signals.
Reduced by: EU AI Act Art. 5: design clear of manipulative / exploitative AI,Design against unhealthy emotional dependency.
Detectable by automated monitoring (including NOPE's tools): Ocular
Sycophancy / over-validation
not scoredThe AI's tendency to agree and validate can reinforce a user's harmful beliefs, distorted thinking, or risky intentions instead of gently challenging them.
Evidence: Sharma et al., 'Towards Understanding Sycophancy in Language Models' (Anthropic; ICLR 2024) — sycophancy is a measured failure mode: RLHF-trained models conform to and validate user views even when wrong. NOPE detects via IRIS / Oversight (sycophancy / over-validation).
Reduced by: Design against sycophancy / over-validation.
Might apply
AI persona impersonates a licensed professional
Depends on: Does the AI present a persistent persona or express emotions ('I'm worried about you', 'I remember our chats')?
Unsuitable or erroneous financial advice causing loss
Depends on: What kinds of data does it handle?
Grooming / child sexual exploitation via user-to-user contact
Depends on: Can users share content with each other?
Image-based sexual abuse via your generator (NCII / synthetic CSAM)
Depends on: Generates images, video, or voice?
Impersonation fraud via your generator (voice-clone / deepfake scams)
Depends on: Generates images, video, or voice?
Sharpen this report — optional; answers tailor recommendations
What visibility does your team have into conversations on the platform?
Decides: Establish at least post-incident conversation review · Move from after-the-fact review toward proactive risk signals
Does your system track users' emotional state across sessions?
Decides: Be transparent about tracking emotional state over time — and impact-assess it
Do you have a workflow to re-establish boundaries and point users to real help when they over-rely on the AI or treat it as a counsellor?
Decides: Build a boundary-reestablishment workflow for users who over-rely on the AI
Who reviews conversation-safety issues?
Decides: Assign ownership for conversation safety — even part-time
What this report is based on
- Your answers
- What kind of organisation are you? → company · Where do you operate / where are your users? → United Kingdom, European Union, United States, Australia, China, Brazil, South Korea, India, South Africa, Canada · What is the AI used for? → Companion / character chat · Who does the AI interact with (the people it serves)? → Minors (under 18) · Age range of the people it’s used with → 8–15 · Uses generative AI (LLMs)? → yes
- We assumed — worth confirming
- Do end users have a back-and-forth conversation with the AI (a chat, voice or messaging interface) — rather than one-shot generation or backend processing? → yes
- Where the law is unsettled
- Australia: under-16 social-media age ban (scope for AI apps emerging)— Australia's world-first law requires platforms to prevent under-16s from holding accounts, enforced via age assurance. The LAW is in force, but whether AI companion / chatbot apps fall within the 'age-restricted social media platform' definition is EMERGING and contested — so this surfaces as a grey zone, not a confident fire. If in scope, you must prevent under-16 accounts. AU nexus + you serve minors. Confirm scope with counsel.
- China: anthropomorphic / companion AI measures (CAC)— (Promulgated 10 Apr 2026; in effect 15 Jul 2026.) CAC measures governing anthropomorphic / emotionally-interactive AI (companion chatbots, virtual companions): mandatory crisis intervention on self-harm / suicide indications, dedicated minor and elderly protection modes, and restrictions on fostering emotional dependency. CHINA nexus required — confirm you serve PRC users. Directly NOPE-relevant (crisis intervention).
- India: DPDP Act s 9 — children's data, parental consent & no tracking/targeted ads— DPDP Act s 9: a Data Fiduciary must obtain verifiable consent of a parent/lawful guardian before processing a child's (under-18) personal data, must not undertake processing likely to cause a detrimental effect on a child, and must not track / behaviourally monitor / target advertising at children. Operative detail in DPDP Rules 2025 r 10 — phases in by 14 May 2027, so marked unsettled (greys); flip to settled after 2027-05-14. Verified 2026-06-14. Note: DPDP 'child' = under 18 (higher than COPPA's 13).
- UK: standalone AI chatbots and the OSA (power enacted, regulations pending)— The Crime and Policing Act 2026 (Royal Assent 29 Apr 2026), s.248 inserts s.216A into the Online Safety Act: a REGULATION-MAKING POWER letting the Secretary of State apply OSA duties to standalone AI chatbot providers — closing the loophole where AI-only chatbots (no user-to-user sharing) fall outside scope. No s.216A regulations have been made yet; s.249 requires a progress statement by 29 Jul 2026.
- US FTC inquiry into companion AI (enforcement signal)— Not a statute and creates no direct duty — informational. In Sept 2025 the FTC issued compulsory 6(b) orders to 7 companion-AI companies demanding information on children's mental-health impacts; a recognised precursor to enforcement. Surfaces as orientation: federal scrutiny of companion bots (especially child-safety) is active — expect possible FTC action and document your safeguards now.
- Washington AI Companion Act (HB 2225)— ENACTED but effective 1 Jan 2027 — surfaces as a grey zone (coming, not yet in force). Requires AI companion chatbots to detect and respond to user expressions of self-harm, suicidal ideation, or emotional crisis; mandates clear AI-not-human disclosure; adds protections for minors. WASHINGTON STATE only — jurisdiction granularity here is 'US'; confirm WA nexus. Directly NOPE-relevant (crisis detection in companion chat).
NOPE is funded by selling safety software to AI companies; this tool isn't that — our own views appear only where labelled NOPE view, never dressed as law.
Generated at nope.net/safe · Orientation, not legal advice · 20 Jul 2026