The National Eating Disorders Association has shut down its chatbot after it made harmful dietary recommendations
The National Eating Disorders Association has shut down the chatbot that replaced the staff and volunteer-run helpline after two users reported receiving harmful advice.
Earlier this week, Sharon Maxwell and Alexis Conason each posted on Instagram about their experiences with the chatbot. Both wrote that the bot, named Tessa, advised them to lose a pound to two pounds a week and eat in a 500-1,000 calorie deficit.
“Everything that Tessa suggested led to the development of my eating disorder.” wrote Maxwellwhich, according to its website, aims to “help fat people experience safety in healthcare and find joy in living in their fat bodies.”
“Imagine vulnerable people with eating disorders asking a robot for assistance because that’s all they have and receiving responses that further fuel the eating disorder,” says Conason, author of The Diet-Free Revolution “. wrote on Instagram.
The National Eating Disorders Association has been operating its chatbot since last year, but until June 1 it also operated a phone hotline operated by staff and hundreds of volunteers. The organization shut down the hotline and fired those employees, NPR reported last month.
Liz Thompson, CEO of the National Eating Disorders Association, told HuffPost in an email that Tessa was removed over the weekend for “supplying off-script language.”
Thompson said Tessa “was attacked and somehow was able to fend off the pre-approved programmed responses.”
She said Tessa has been on the organization’s website since February 2022 and has had “incredibly positive results.” But the Malicious messages from the chatbot are unacceptable, she said.
“Please note that at NEDA we do not believe that even 0.1% of the time for malicious messages is acceptable,” Thompson said in the email. “The language being shared about weight loss tips and dieting goes against our organizational philosophy and policy — and was not in the original Body Positive programming.”
She said Cass, the company that programs Tessa, told her that bad actors tried to trick Tessa and that Tessa’s reactions came after the bad actors’ “nefarious activities”.
Cass founder and CEO Michiel Rauws told HuffPost that traffic to the chatbot has increased by 600%.
“We’re grateful that some people have tested the service out of concern and pointed out these issues,” he said in an email. “Others exhibited behaviors that were not related to eating disorders but indicated various forms of nefarious activity by bad actors trying to trick Tessa. Very few messages were affected, but even one message is too many. We do not take this lightly and we apologize for it.”
Last month, the union represented helpline workers published an opinion Warning that a chatbot cannot offer the same help as a person.
“A chatbot is no substitute for human empathy, and we believe this decision will do irreparable harm to the eating disorder community,” wrote the Helpline Associates of the National Eating Disorders Association in May.
If you’re struggling with an eating disorder, call 988, text, or chat 988lifeline.org for support.