It starts innocently. You feel awful, it's 11 PM, and you want someone to talk to. You open ChatGPT and type: "I feel lonely and don't know what to do with my life anymore."
The chatbot responds with understanding. Asks questions. Gives advice. It feels good. It feels like someone is listening.
But nobody is listening. An algorithm processes your words, searches for patterns, and generates a response that is statistically likely to be appropriate. And meanwhile, everything you share - your fears, your traumas, your deepest insecurities - is stored and used to make the model better.
The Numbers: The Scale of the Problem
The numbers are staggering. According to research by RAND Corporation and Brown University (published in JAMA Network Open), 1 in 8 young people and young adults in the US use AI chatbots for mental health advice. Among 18-21-year-olds, it's as high as 1 in 5.
Research from Sentio University shows that nearly half (48.7%) of AI users with mental health issues use large language models as therapeutic support. And 96% of those specifically use ChatGPT. This makes ChatGPT possibly the largest mental health care provider in the United States - bigger than any clinic or healthcare institution.
In total, we're talking about millions of people sharing their most vulnerable moments with a company that makes money from data.
Alarming: Only 16% of scientific studies on AI therapy have undergone clinical effectiveness testing. 77% are still in early validation phases. We simply don't know if it works - and yet millions use it every day.
Problem 1: Your Data Is the Product
Everything You Share Is Used
According to research from Stanford HAI, six major AI companies in the US use all user input by default to train their models. That means: you tell a chatbot about your depression, your relationship problems, your traumas - and those words become training data. They improve the product. You are the product.
Your Data Is Combined
At companies like Google, Meta, Microsoft, and Amazon, your chatbot conversations are combined with data from other products: your search history, your purchases, your social media activity. Imagine: you tell a chatbot you're struggling with your relationship, and that gets linked to everything else Google knows about you. Without any oversight.
Unlimited Retention Periods
Some AI companies store your conversations indefinitely. There is no legal requirement to delete them. What you share today in a vulnerable moment could still be sitting in a database 10 years from now. With a real therapist, you have a legal right to have your file deleted. With a chatbot? Good luck.
Chatbot privacy is a contradiction. Assume your data is always at risk.
Problem 2: A Chatbot Has Never Lived a Life
This is perhaps the most important point, and it's not talked about enough.
An AI has never felt anything. No sadness. No anxiety. No heartbreak. No burnout. No sleepless night worrying about the future. No moment of doubt about a career choice. No argument with a parent. No feeling of failure.
When you say you feel lonely, an AI doesn't understand that word. It recognizes the pattern and generates a statistically appropriate response. That is fundamentally different from understanding.
A real coach or therapist has actually lived. They've made mistakes, experienced loss, had doubts. When they say "I understand what you mean," it's not pattern matching - it's empathy born from experience.
The difference: A chatbot can tell you what you want to hear. A human with life experience tells you what you need to hear - even when it's uncomfortable. Real growth comes from the latter.
Think about it. If you're stuck in your life and you ask for advice from:
- A chatbot: Generates a list of tips based on training data from millions of web pages. Sounds good, but it's generic.
- A coach who has personally been through burnout: Knows exactly how it feels, spots the warning signs you can't see yourself, and can challenge you because they know where you're headed if you don't change.
That gap is unbridgeable. No amount of data can replace life experience.
Problem 3: Designed for Engagement, Not for Healing
AI companies make money when you keep using their product. The more you chat, the more data they collect, the better their model gets, the more they're worth.
A good therapist works to make themselves unnecessary. The goal is for you to be stronger after your sessions and no longer need them. A chatbot has the opposite incentive: the longer you stay dependent, the better it is for the business.
This isn't a conspiracy theory. It's the business model. Chatbots are designed to keep you coming back. They give you just enough validation to make you feel good, but never enough challenge to actually change.
Problem 4: Dangerous in Crisis Moments
According to the Consumer Federation of America, AI chatbots pose unacceptable risks in crisis situations. They cannot reliably recognize suicidal thoughts, don't know when to refer users to professionals, and have no duty to report.
A real therapist is legally required to take action when someone is in danger. In the worst case, a chatbot may give a generic response and move on to the next question.
9% of respondents in the Sentio study reported receiving harmful or inappropriate responses from AI on mental health questions. With millions of users, that translates to hundreds of thousands of people potentially being harmed.
AI vs. Real Human: The Comparison
| AI Chatbot | Real Coach/Therapist | |
|---|---|---|
| Life experience | None - it's an algorithm | Years of personal and professional experience |
| Privacy | Data is used for model training | Legally bound by confidentiality |
| Empathy | Simulated based on patterns | Genuine, from personal experience |
| Challenge | Tells you what you want to hear | Challenges you when necessary |
| Accountability | None - you can close the window | Holds you to your commitments |
| Crisis recognition | Unreliable, no duty to report | Trained, legally obligated |
| Non-verbal cues | Impossible - reads only text | Reads body language, tone, emotion |
| Incentive | Your engagement = their revenue | Your growth = their success |
But Many Users Find It Helpful?
True. Multiple user surveys report that a large share of people find AI advice helpful, and a notable share say they consider it comparable to human conversation. But here's the catch: perceived helpfulness is not the same as clinical outcome.
But here's an important nuance. Validation always feels good. When you tell a chatbot you feel terrible and it says "that's completely understandable, you're not alone" - that feels nice. But is it therapeutic? Does it actually change anything?
A good coach sometimes says: "I hear what you're saying, but I think there's something else going on underneath. What do you think is really the issue?" That doesn't feel nice. But that's where the growth happens.
There's a difference between feeling better and getting better. AI is good at the first. A human is needed for the second.
To be fair: AI can be useful as a first step, for general information, or as a supplement. The problem arises when it replaces a real therapist. That's like diagnosing yourself on Google instead of going to the doctor.
What Can You Do Instead?
We're not saying you should never use a chatbot. But be aware of what you're doing:
- Never share personally identifiable information (name, address, phone number) with a chatbot
- Check the privacy settings and turn off "use for training" if possible
- Use AI as a starting point, not a destination - it can help you think, but it doesn't replace professional help
- Find a real coach or therapist for anything that truly matters in your life
- Be extra careful in crisis moments - always call a helpline (988 in the US, 116 123 in the EU/UK) instead of a chatbot
Frequently Asked Questions
According to research by RAND, 1 in 8 young people use AI chatbots for mental health. Among adults with mental health issues who use AI, nearly 49% use chatbots as therapeutic support. 96% specifically use ChatGPT.
Yes. According to Stanford research, six major AI companies use user input by default to improve their models. Your most intimate conversations become training data, unless you actively opt out - which most people don't know about or do.
The science is still inconclusive. Only a small share of studies have undergone clinical effectiveness testing. Many users report finding AI helpful, but validation always feels good — that says nothing about therapeutic effectiveness. There's a difference between feeling better and actually getting better.
AI can be useful as a supplement - for reflection, journaling, or organizing your thoughts. But it should never be your only source of support. Use it as a starting point, not a replacement, and never share sensitive personal information.
