Chatbots work best when you speak to them with formal language

Talking to an AI chatbot in less formal language, as many people do, reduces the accuracy of its responses – suggesting that either we need to be linguistically stricter when using a chatbot, or that the AIs need to be trained to better adapt to informality.

Fulei Zhang and Zhou Yu at Amazon looked at how people begin conversations with human agents compared with a chatbot assistant powered by a large language model (LLM). They used the Claude 3.5 Sonnet model to score the conversations on a range of factors and found that people interacting with chatbots used less accurate grammar and were less polite than they were when addressing humans. They also used a slightly narrower range of vocabulary.

For example, human-to-human interaction was 14.5 per cent more polite and formal than conversations with chatbots, 5.3 per cent more fluent and 1.4 per cent more lexically diverse, according to the Claude-derived scores.

“Users adapt their linguistic style in human-LLM conversations, producing messages that are shorter, more direct, less formal, and grammatically simpler,” the authors, who didn’t respond to an interview request, write in a paper about the work. “This behaviour is likely shaped by users’ mental models of LLM chatbot[s] as less socially sensitive or less capable of nuanced interpretation.”

But it turns out this informality has a downside. In a second assessment, the researchers trained an AI model called Mistral 7B on 13,000 real-world human-to-human conversations and used it to interpret 1357 real-world messages sent to AI chatbots. They annotated each conversation within both datasets with an “intent” drawn from a limited list, summarising what the user was trying to do in each case. But because the Mistral AI had been trained on human-to-human conversations, the pair found that the AI struggled to correctly label intent for the chatbot conversations.

Zhang and Yu then tried various strategies to improve the Mistral AI’s understanding. First, they used the Claude AI  to rewrite users’ more terse missives into human-like prose and used them to fine-tune the Mistral model. This reduced the accuracy of its intent labels by 1.9 per cent compared to its default responses.

Next, they used Claude to provide a “minimal” rewrite, which was shorter and more blunt (for instance, “paris next month. flights hotels?” to ask about travel and accommodation options for an upcoming trip), but this reduced Mistral’s accuracy by 2.6 per cent. An alternative, “enriched” rewrite with more formal and varied language also saw accuracy drop by 1.8 per cent. It was only by training the Mistral model on both minimal and enriched rewrites that they saw improved performance, by 2.9 per cent.

Noah Giansiracusa at Bentley University in Massachusetts says he isn’t surprised that people talk differently to bots than they do to humans, but it isn’t necessarily something to be avoided.

“The finding that people communicate differently with chatbots than with other humans is temptingly framed as a shortcoming of the chatbot – but I’d argue that it’s not, that it’s good when people know they are talking with bots and adapt their behaviour accordingly,” says Giansiracusa. “I think that’s healthier than obsessively trying to eliminate the gap between human and bot.”