New research shows when AI boosts service, and when it backfires.
Think about the last time you contacted customer support. Did you start with a chatbot? If it failed to resolve your problem, how did you feel when transferred to a human agent? This dynamic defines our expectations of the modern customer service experience: the struggle to balance the cold speed of automation with the warm necessity of human empathy. However, in 鈥�,鈥� D^3 Frontier Firm affiliate Shunyuan Zhang and Das Narayandas explain how the results from a year-long experiment involving 138 customer service agents and over 250,000 conversations are far more complex than the typical assumption.
Key Insight: AI Assistance Isn鈥檛 Just Faster, It鈥檚 More Human
鈥淲e posit that AI enables agents to handle conversations more efficiently, thus encouraging more responses from customers, leading to deeper back-and-forth interactions between them.鈥� [1]
The prevailing fear in customer service is that introducing AI will turn human interactions into robotic, assembly-line exchanges. Yet, when agents received real-time AI-generated reply suggestions, they didn鈥檛 just respond 22% faster to customer messages, they actually sent more messages and saw a measurable boost in the 鈥渉uman鈥� quality of the chats. AI freed agents from the cognitive burden of composing responses, allowing them to engage customers more deeply. The conversations became richer, not shallower: using large language models to categorize agent messages, the researchers found that AI-assisted responses scored higher in the key aspects of empathy, information, and solution, with the largest jump in empathy.
Key Insight: The Experience Equalizer
鈥淪pecifically, for a hypothetical brand new agent, AI would lead to a remarkable reduction in agent response time of approximately 70.3%.鈥� [2]
One of the most business-relevant findings in the study was that AI assistance didn鈥檛 benefit everyone equally. When the researchers examined how agent tenure moderated AI鈥檚 effects, they found that less-experienced agents gained far more from AI suggestions than their veteran counterparts. Essentially, the AI 鈥渄ownloaded鈥� institutional knowledge into the workflow of new employees: having access to these real-time suggestions was the functional equivalent of nearly five months of experience. This has profound implications for industries with high turnover, suggesting that AI can serve as a stabilized bridge, ensuring that a customer鈥檚 experience doesn鈥檛 suffer just because they happened to be connected to a trainee.
Key Insight: Not All Conversations Are Created Equal
鈥淒ifferent customer intents shape the context and dynamics of conversations, and if AI fails to adapt to these nuances, it may provide misleading suggestions, potentially harming interactions.鈥� [3]
The AI algorithm鈥檚 impact varied depending on why customers were reaching out in the first place. For example, when customers wanted to cancel subscriptions鈥攖raditionally difficult conversations鈥擜I helped agents identify underlying reasons and recommend alternative options, leading to notable improvements in customer sentiment. But repeat complaints told a different story. Although AI helped agents respond quickly in these scenarios, customer sentiment barely improved. These complaints stemmed from systematic operational issues, like recurring delivery problems, that no amount of empathetic, information-rich messaging could solve. The AI could help agents communicate better about problems, but it couldn鈥檛 actually fix them.
Perhaps the most counterintuitive finding emerged from examining what happened in the handoff from a bot to a human agent. Many companies use a 鈥渃hatbot first鈥� approach, where a fully automated bot tries to solve the problem before transferring the customer to a human. As we鈥檝e seen, AI-assisted agents are able to respond more quickly, and if the AI-assisted agent responded too quickly, customers suspected that they were still talking to a bot. The response speed that might normally delight customers became a liability, triggering what the researchers term a negative 鈥渟pillover鈥� from the initial bot failure. In these contexts, the study found that increasing the delay in human responses actually helped rebuild trust and improve sentiment.
Why This Matters
For executives deploying AI in customer-facing operations, this research delivers three strategic imperatives. First, resist the temptation to replace human agents entirely: augmentation delivers better outcomes than automation alone, particularly for handling nuanced, emotionally charged interactions. Second, deploy AI with precision: it鈥檚 most valuable in specific conversation types (like retention scenarios). Third, manage your AI ecosystem holistically. If you鈥檙e using multiple AI systems in sequence, recognize that they鈥檙e not independent. The companies that will win with AI aren鈥檛 those that deploy the most LLMs, they鈥檙e those that understand how these systems interact across the entire customer ecosystem and adapt their implementation accordingly.
Bonus
When emotions are involved, who people think is responding can shape outcomes as much as what is said. For another angle on AI and human emotion, check out It Feels Like AI Understands, But Do We Care? New Research on Empathy.
References
[1] Zhang, Shunyuan, and Das Narayandas, 鈥淓ngaging Customers with AI in Online Chats: Evidence from a Randomized Field Experiment.鈥� Management Science 72 (1) (2025): 84.
[2] Zhang and Narayandas, 鈥淓ngaging Customers with AI in Online Chats,鈥� 84.
[3] Zhang and Narayandas, 鈥淓ngaging Customers with AI in Online Chats,鈥� 75-76.
Meet the Authors
is Associate Professor of Business Administration at 性视界 Business School. She and other HBS faculty contribute to the D^3 Frontier Firm Initiative.
is Edsel Bryant Ford Professor of Business Administration at 性视界 Business School.