LLM-based Smart Reply (LSR): Enhancing Collaborative Performance with ChatGPT-mediated Smart Reply System

Interactive user interfaces have increasingly explored AI's role in enhancingcommunication efficiency and productivity in collaborative tasks. AI tools suchas chatbots and smart replies aim to enhance conversation quality and improveteam performance. Early AI assistants, were limited by predefined knowledgebases and decision trees. However, the advent of Large Language Models (LLMs)such as ChatGPT has revolutionized AI assistants, employing advanced deeplearning architecture to generate context-aware, coherent, and personalizedresponses. Consequently, ChatGPT-based AI assistants provide a more natural andefficient user experience across various tasks and domains. In this paper, westudy how LLM models such as ChatGPT can be used to improve work efficiency incollaborative workplaces. Specifically, we present an LLM-based Smart Reply(LSR) system utilizing the ChatGPT to generate personalized responses in dailycollaborative scenarios, while adapting to context and communication stylebased on prior responses. Our two-step process involves generating apreliminary response type (e.g., Agree, Disagree) to provide a generalizeddirection for message generation, thus reducing response drafting time. Weconducted an experiment in which participants completed simulated work tasks,involving a Dual N-back test and subtask scheduling through Google Calendarwhile interacting with researchers posing as co-workers. Our findings indicatethat the proposed LSR reduces overall workload, as measured by the NASA TLX,and improves work performance and productivity in the N-back task. We alsoprovide qualitative feedback on participants' experiences as well as designrecommendations so as to provide future directions for the design of thesetechnologies.