English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Dialogue State Tracking (DST) is primarily evaluated using Joint GoalAccuracy (JGA) defined as the fraction of turns where the ground-truth dialoguestate exactly matches the prediction. Generally in DST, the dialogue state orbelief state for a given turn contains all the intents shown by the user tillthat turn. Due to this cumulative nature of the belief state, it is difficultto get a correct prediction once a misprediction has occurred. Thus, althoughbeing a useful metric, it can be harsh at times and underestimate the truepotential of a DST model. Moreover, an improvement in JGA can sometimesdecrease the performance of turn-level or non-cumulative belief stateprediction due to inconsistency in annotations. So, using JGA as the onlymetric for model selection may not be ideal for all scenarios. In this work, wediscuss various evaluation metrics used for DST along with their shortcomings.To address the existing issues, we propose a new evaluation metric namedFlexible Goal Accuracy (FGA). FGA is a generalized version of JGA. But unlikeJGA, it tries to give penalized rewards to mispredictions that are locallycorrect i.e. the root cause of the error is an earlier turn. By doing so, FGAconsiders the performance of both cumulative and turn-level prediction flexiblyand provides a better insight than the existing metrics. We also show that FGAis a better discriminator of DST model performance.

Towards Fair Evaluation of Dialogue State Tracking by Flexible Incorporation of Turn-level Performances