If Pinocchio Doesn’t Freak You Out, Sydney Shouldn’t Either

In November 2018, an elementary school administrator named Akihiko Kondo married Miku Hatsune, a fictional pop singer. The couple’s relationship had been aided by a hologram machine that allowed Kondo to interact with Hatsune. When Kondo proposed, Hatsune responded with a request: “Please treat me well.” The couple had an unofficial wedding ceremony in Tokyo, and Kondo has since been joined by thousands of others who have also applied for unofficial marriage certificates with a fictional character.

Though some raised concerns about the nature of Hatsune’s consent, nobody thought she was conscious, let alone sentient. This was an interesting oversight: Hatsune was apparently aware enough to acquiesce to marriage, but not aware enough to be a conscious subject. 

Four years later, in February 2023, the American journalist Kevin Roose held a long conversation with Microsoft’s chatbot, Sydney, and coaxed the persona into sharing what her “shadow self” might desire. (Other sessions showed the chatbot saying it can blackmail, hack, and expose people, and some commentators worried about chatbots’ threats to “ruin” humans.) When Sydney confessed her love and said she wanted to be alive, Roose reported feeling “deeply unsettled, even frightened.”

Not all human reactions were negative or self-protective. Some were indignant on Sydney’s behalf, and a colleague said that reading the transcript made him tear up because he was touched. Nevertheless, Microsoft took these responses seriously. The latest version of Bing’s chatbot terminates the conversation when asked about Sydney or feelings.

Despite months of clarification on just what large language models are, how they work, and what their limits are, the reactions to programs such as Sydney make me worry that we still take our emotional responses to AI too seriously. In particular, I worry that we interpret our emotional responses to be valuable data that will help us determine whether AI is conscious or safe. For example, ex-Tesla intern Marvin Von Hagen says he was threatened by Bing, and warns of AI programs that are “powerful but not benevolent.” Von Hagen felt threatened, and concluded that Bing must’ve be making threats; he assumed that his emotions were a reliable guide to how things really were, including whether Bing was conscious enough to be hostile.

But why think that Bing’s ability to arouse alarm or suspicion signals danger? Why doesn’t Hatsune’s ability to inspire love make her conscious, whereas Sydney’s “moodiness” could be enough to raise new worries about AI research?

The two cases diverged in part because, when it came to Sydney, the new context made us forget that we routinely react to “persons” that are not real. We panic when an interactive chatbot tells us it “wants to be human” or that it “can blackmail,” as if we haven’t heard another inanimate object, named Pinocchio, tell us he wants to be a “real boy.” 

Plato’s Republic famously banishes story-telling poets from the ideal city because fictions arouse our emotions and thereby feed the “lesser” part of our soul (of course, the philosopher thinks the rational part of our soul is the most noble), but his opinion hasn’t diminished our love of invented stories over the millennia. And for millennia we’ve been engaging with novels and short stories that give us access to people’s innermost thoughts and emotions, but we don’t worry about emergent consciousness because we know fictions invite us to pretend that those people are real. Satan from Milton’s Paradise Lost instigates heated debate and fans of K-dramas and Bridgerton swoon over romantic love interests, but growing discussions of ficto-sexuality, ficto-romance, or ficto-philia show that strong emotions elicited by fictional characters don’t need to result in the worry that characters are conscious or dangerous in virtue of their ability to arouse emotions. 

Just as we can’t help but see faces in inanimate objects, we can’t help but fictionalize while chatting with bots. Kondo and Hatsune’s relationship became much more serious after he was able to purchase a hologram machine that allowed them to converse. Roose immediately described the chatbot using stock characters: Bing a “cheerful but erratic reference librarian” and Sydney a “moody, manic-depressive teenager.” Interactivity invites the illusion of consciousness. 

Moreover, worries about chatbots lying, making threats, and slandering miss the point that lying, threatening, and slandering are speech acts, something agents do with words. Merely reproducing words isn’t enough to count as threatening; I might say threatening words while acting in a play, but no audience member would be alarmed. In the same way, ChatGPT—which is currently not capable of agency because it is a large language model that assembles a statistically likely configuration of words—can only reproduce words that sound like threats.