English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In Natural Language Processing (NLP), binary classification algorithms areoften evaluated using the F1 score. Because the sample F1 score is an estimateof the population F1 score, it is not sufficient to report the sample F1 scorewithout an indication of how accurate it is. Confidence intervals are anindication of how accurate the sample F1 score is. However, most studies eitherdo not report them or report them using methods that demonstrate poorstatistical properties. In the present study, I review current analyticalmethods (i.e., Clopper-Pearson method and Wald method) to construct confidenceintervals for the population F1 score, propose two new analytical methods(i.e., Wilson direct method and Wilson indirect method) to do so, and comparethese methods based on their coverage probabilities and interval lengths, aswell as whether these methods suffer from overshoot and degeneracy. Theoreticalresults demonstrate that both proposed methods do not suffer from overshoot anddegeneracy. Experimental results suggest that both proposed methods performbetter, as compared to current methods, in terms of coverage probabilities andinterval lengths. I illustrate both current and proposed methods on twosuggestion mining tasks. I discuss the practical implications of these results,and suggest areas for future research.

Confidence Intervals for the F1 Score: A Comparison of Four Methods