Pluralistic: Anyone who trusts an AI therapist needs their head examined (01 Apr 2025)


Today's links


Sigmund Freud's study with his famous couch. Behind the couch stands an altered version of the classic Freud portrait in which he is smoking a cigar. Freud's clothes and cigar have all been tinted in bright neon colors. His head has been replaced with the glaring red eye of HAL9000 from Kubrick's '2001: A Space Odyssey.' His legs have been replaced with a tangle of tentacles.

Anyone who trusts an AI therapist needs their head examined (permalink)

There's a debate to be had about whether AI chatbots make good psychotherapists. This is not an area of my expertise, so I'm not going to weigh in on that debate. But nevertheless, I think that if you use an AI therapist, you need your head examined:

https://www.salon.com/2025/03/30/some-argue-ai-therapy-can-break-down-mental-health-stigma–others-warn-it-could-make-it-worse/

I'm not an expert on psychotherapy, but I am an expert on privacy and corporate misconduct, and holy shit is the idea of a chatbot psychotherapist running on some Big Tech cloud a terrible idea. Because while I'm no expert on therapy, I have benefited from therapy, and I know this for certain: therapy requires confidentiality.

Shrinks are incredibly careful about privacy. For example: when my brother was getting married, my therapist was invited to the wedding. His daughter and my brother's fiancee were close friends, and my brother's fiancee had grown up staying over at their house and wanted her friend and her friend's parents at the wedding. My therapist sat me down and said, "Now listen, I take confidentiality very seriously. If you want me to, I will pretend not to know you at the wedding. No one needs to know that you're seeing me or – any therapist."

I told him I didn't mind people knowing I'd seen him, but just that little fastidious gesture confirmed the trust I'd put in Alan. It meant that I could openly and freely discuss things I'd never told anyone before, and that I never told anyone ever again. Having those genuinely open conversations transformed my life, for the better.

Now consider the chatbot therapist: what are its privacy safeguards? Well, the companies may make some promises about what they will and won't do with the transcripts of your AI sessions, but they are lying. Of course they're lying! AI companies lie about what their technology can do (of course). They lie about what their technologies will do. They lie about money. But most of all, they lie about data.

There is no subject on which AI companies have been more consistently, flagrantly, grotesquely dishonest than training data. When it comes to getting more data, AI companies will lie, cheat and steal in ways that would seem hacky if you wrote them into fiction, like they were pulp-novel dope fiends:

https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/

When an AI company tells you it won't use your intimate secrets as training data, they are lying. Of course they're lying! This isn't just any data, it's data that isn't replicated elsewhere on the internet. It's rare – it's unique. It's a competitive advantage. AI companies will 100%, without exception, totally use your private therapy data as training data.

What's more: they will leak your therapy sessions. They will leak them because they can't figure out how to prevent models from vomiting up their training data verbatim:

https://www.theatlantic.com/technology/archive/2024/01/chatgpt-memorization-lawsuit/677099/

But they'll also leak because tech companies leak like hell. They are crawling with insider threats. If the AI company sticks around long enough, it'll leak your secrets. And if it goes bankrupt? That's even worse! When tech companies go bust, the first thing their creditors do is sell off their warehouses full of private data. The more private and compromising that data is, the harder they'll try to sell it:

https://www.eff.org/deeplinks/2025/03/how-delete-your-23andme-data

Now, maybe you're thinking, "OK, but that's a small price to pay if we can finally get therapy for everyone." After all, the country – the world – is in the midst of a terrible mental health crisis and there's a dire shortage of therapists.

Now, let's stipulate for the moment hat chatbots are substitutes for human therapists – that, at the very least, they're better than nothing. I don't think that's true, but let's say it is. Even so, this is a bad tradeoff.

Here, try this thought-experiment: someone figures out a great business-model to pay for therapy for poor people. "We turned therapy into a livestreamed reality TV show. If you're too poor to afford a therapist, you can go to one of our partially trained livestreamer therapists, who will broadcast all of your secrets to anyone who watches. There's a permanent archive of these sessions, and the worst people in the world comb through it 24/7 looking for embarrassing stuff to repost and go viral with. What, you don't like that? Oh, I see: you just don't think poor people deserve mental health. I guess the perfect really is the enemy of the good."

This gambit is called "predatory inclusion." Think of Spike Lee shilling cryptocurrency scams as a way to "build Black wealth" or Mary Kay promising to "empower women" by embroiling them in a bank-account-draining, multi-level marketing cult. Having your personal, intimate secrets sold, leaked, published or otherwise exploited is worse for your mental health than not getting therapy in the first place, in the same way that having your money stolen by a Bitcoin grifter or Mary Kay is worse than not being able to access investment opportunities in the first place.

But it's not just people struggling with their mental health who shouldn't be sharing sensitive data with chatbots – it's everyone. All those business applications that AI companies are pushing, the kind where you entrust an AI with your firm's most commercially sensitive data? Are you crazy? These companies will not only leak that data, they'll sell it to your competition. Hell, Microsoft already does this with Office365 analytics:

https://pluralistic.net/2021/02/24/gwb-rumsfeld-monsters/#bossware

These companies lie all the time about everything, but the thing they lie most about is how they handle sensitive data. It's wild that anyone has to be reminded of this. Letting AI companies handle your sensitive data is like turning arsonists loose in your library with a can of gasoline, a book of matches, and a pinky-promise that this time, they won't set anything on fire.

(Image: Zde, CC BY-SA 4.0, modified)

Hey look at this (permalink)


A Wayback Machine banner.

Object permanence (permalink)

#15yrsago Noteworthy Modern Occurances: the Digital Economy Bill https://www.openrightsgroup.org/blog/disconnection-notices-served/

#15yrsago Digital Economy Bill: the last hours https://www.youtube.com/watch?v=gOyg1GUY18U

#5yrsago Turn on wifi sharing https://pluralistic.net/2020/04/02/eff-livestream-today/#digital-divide

#5yrsago Coronavirus travel posters https://pluralistic.net/2020/04/02/eff-livestream-today/#jennifer-baer

#5yrsago How you are subsidizing the otherwise unprofitable Fox News https://pluralistic.net/2020/04/02/eff-livestream-today/#unfoxmycablebox

#5yrsago Ted Chiang on pandemics as idiot plots https://pluralistic.net/2020/04/02/eff-livestream-today/#disaster-capitalism

#5yrsago Bird's "Black Mirror" mass layoffs https://pluralistic.net/2020/04/02/eff-livestream-today/#2-mins

#5yrsago UK public health official endorses official reagents for covid tests https://pluralistic.net/2020/04/02/eff-livestream-today/#unauthorized-reagents

#5yrsago A promising, plausible plan for "privacy-preserving" surveillance https://pluralistic.net/2020/04/02/eff-livestream-today/#pepp-pt

#5yrsago Private equity titan squats on empty hospital https://pluralistic.net/2020/04/02/eff-livestream-today/#joel-kills

#1yrago Prison-tech company bribed jails to ban in-person visits https://pluralistic.net/2024/04/02/captive-customers/#guillotine-watch

Upcoming appearances (permalink)

A photo of me onstage, giving a speech, pounding the podium.

Upcoming appearances (permalink)

A photo of me onstage, giving a speech, pounding the podium.


A screenshot of me at my desk, doing a livecast.

Recent appearances (permalink)


A grid of my books with Will Stahle covers..

Latest books (permalink)


A cardboard book box with the Macmillan logo.

Upcoming books (permalink)

  • Enshittification: Why Everything Suddenly Got Worse and What to Do About It, Farrar, Straus, Giroux, October 7 2025
    https://us.macmillan.com/books/9780374619329/enshittification/

  • Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026

  • Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026

  • The Memex Method, Farrar, Straus, Giroux, 2026


Colophon (permalink)

Today's top sources:

Currently writing:

  • Enshittification: a nonfiction book about platform decay for Farrar, Straus, Giroux. Status: second pass edit underway (readaloud)

  • A Little Brother short story about DIY insulin PLANNING

  • Picks and Shovels, a Martin Hench noir thriller about the heroic era of the PC. FORTHCOMING TOR BOOKS FEB 2025

Latest podcast: Why I don't like AI art https://craphound.com/news/2025/03/30/why-i-dont-like-ai-art/

This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

ISSN: 3066-764X

1 Comment