English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

As conversational models become increasingly available to the general public,users are engaging with this technology in social interactions. Suchunprecedented interaction experiences may pose considerable social andpsychological risks to the users unless the technology is properly controlled.This highlights the need for scalable and robust evaluation metrics forconversational chatbots. Existing evaluation metrics aim to automate offlineuser evaluation and approximate human judgment of pre-curated dialogs. However,they are limited in their ability to capture subjective perceptions of userswho actually interact with the bots and might not generalize to real-worldsettings. To address this limitation, we propose an approach to approximateonline human evaluation leveraging large language models (LLMs) from the GPTfamily. We introduce a new Dialog system Evaluation framework based onPrompting (DEP), which enables a fully automatic evaluation pipeline thatreplicates live user studies and achieves an impressive correlation with humanjudgment (up to Pearson r=0.95 on a system level). The DEP approach involvescollecting synthetic chat logs of evaluated bots with an LLM in the other-playsetting, where the LLM is carefully conditioned to follow a specific scenario.We further explore different prompting approaches to produce evaluation scoreswith the same LLM. The best performing prompts, which contain few-shotdemonstrations and instructions, show outstanding performance on the testeddataset and demonstrate the ability to generalize to other dialog corpora.

Approximating Online Human Evaluation of Social Chatbots with Prompting