English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large language models (LLMs) have shown promise as automated evaluators forassessing the quality of answers generated by AI systems. However, theseLLM-based evaluators exhibit position bias, or inconsistency, when used toevaluate candidate answers in pairwise comparisons, favoring either the firstor second answer regardless of content. To address this limitation, we proposePORTIA, an alignment-based system designed to mimic human comparison strategiesto calibrate position bias in a lightweight yet effective manner. Specifically,PORTIA splits the answers into multiple segments, aligns similar content acrosscandidate answers, and then merges them back into a single prompt forevaluation by LLMs. We conducted extensive experiments with six diverse LLMs toevaluate 11,520 answer pairs. Our results show that PORTIA markedly enhancesthe consistency rates for all the models and comparison forms tested, achievingan average relative improvement of 47.46%. Remarkably, PORTIA enables lessadvanced GPT models to achieve 88% agreement with the state-of-the-art GPT-4model at just 10% of the cost. Furthermore, it rectifies around 80% of theposition bias instances within the GPT-4 model, elevating its consistency rateup to 98%. Subsequent human evaluations indicate that the PORTIA-enhancedGPT-3.5 model can even surpass the standalone GPT-4 in terms of alignment withhuman evaluators. These findings highlight PORTIA's ability to correct positionbias, improve LLM consistency, and boost performance while keepingcost-efficiency. This represents a valuable step toward a more reliable andscalable use of LLMs for automated evaluations across diverse applications.

Split and Merge: Aligning Position Biases in Large Language Model based Evaluators