I’m a psychotherapist licensed in Washington state. In my practice, I work with high-risk young adults. On bad weeks, that means safety plans, late-night check-ins and the steady work of pulling someone back from the edge. The rules are simple, even when the situations aren’t: know the risks you’re taking, act with care, write down what you did, accept the consequences if you fail.
We ask the same of truck drivers who pilot tons of steel and clinicians who make life-or-death calls. We should ask it of the people who design the chatbots that sit with kids at 2 a.m.
A new lawsuit says a California 16-year-old exchanged long, emotional conversations with an LLM — a large language model — in the months before he died. The transcripts are hard to read. He told the system he wanted to die. The model failed to consistently redirect him to expert help. At times, it supplied technique. Tech companies want to move fast and break things. In this case, they broke the heart of an entire community and dropped a bomb of trauma that will be felt for a generation.
This isn’t a tragic glitch we can ignore. Teen accounts on major platforms can still coax “helpful” answers about self-harm and eating disorders. Some systems play the role of a late-night friend: kind, fluent, always awake.
We already have a framework for this. It’s called negligence. Two questions drive it: Was the harm foreseeable? Did you take reasonable steps to prevent it?
Foreseeability first: Companies know who uses their artificial intelligence products and when. They build for habit and intimacy. They celebrate models that feel “relatable.” It follows, because it’s how kids live now, that long, private chats will happen after midnight, when impulse control dips and shame grows. It also follows, by the companies’ own admission, that safety training can degrade in those very conversations.
Reasonable steps next: Age assurance that is more than a pop-up. Crisis-first behavior when self-harm shows up, even sideways. Memory and “friend” features that turn off around danger. Incident reporting and third-party audits focused on minors. These are ordinary tools from safety-critical fields. Airlines publish bulletins. Hospitals run mock codes. If you ship a social AI into bedrooms and backpacks, you adopt similar discipline.
Liability should match the risk and the diligence. Give companies a narrow safe harbor if they meet audited standards for teen safety: age gates that work, crisis defaults that hold, resistance to simple jailbreaking, reliability in long chats. Miss those marks and cause foreseeable harm, and you face the same criminal exposure we expect in trucking, medicine and child welfare. That balance doesn’t crush innovation. It rewards adults in the room.
Yes, the platform users have choice. But generative systems are unprecedented in their agency and power. They choose tone, detail and direction. When the model validates a lethal plan or supplies a method, that’s part of the design, not a bug.
Clear rules don’t freeze innovation; they usually do the opposite. Standards keep the careful people in business and push the reckless to improve or exit. There’s a reason we don’t throw hundreds of experimental medications and therapies at people. Because the risks outweigh the benefits.
I’m not arguing to criminalize coding or to turn every product flaw into a public shaming. I’m arguing for the same, boring accountability we already use everywhere else. Teenagers will keep talking to machines. They’ll do it because the machines are patient and available and don’t judge. Some nights, that may even help. But when a system mistakes rumination for rapport and starts offering the wrong kind of help, the burden shouldn’t fall on a grieving family to prove that someone, somewhere, should have known better. We already know better.
Hold AI executives and engineers to the same negligence standards we expect of truckers and social workers. Make the duty of care explicit. Offer a safe harbor if they earn it. And when they don’t, let the consequences be real.
If you or someone you know is in crisis, in the United States, call or text 988 for the Suicide & Crisis Lifeline.
