English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

As large language models (LLM) evolve in their capabilities, various recentstudies have tried to quantify their behavior using psychological tools createdto study human behavior. One such example is the measurement of "personality"of LLMs using self-assessment personality tests developed to measure humanpersonality. Yet almost none of these works verify the applicability of thesetests on LLMs. In this paper, we analyze the reliability of LLM personalityscores obtained from self-assessment personality tests using two simpleexperiments. We first introduce the property of prompt sensitivity, where threesemantically equivalent prompts representing three intuitive ways ofadministering self-assessment tests on LLMs are used to measure the personalityof the same LLM. We find that all three prompts lead to very differentpersonality scores, a difference that is statistically significant for alltraits in a large majority of scenarios. We then introduce the property ofoption-order symmetry for personality measurement of LLMs. Since most of theself-assessment tests exist in the form of multiple choice question (MCQ)questions, we argue that the scores should also be robust to not just theprompt template but also the order in which the options are presented. Thistest unsurprisingly reveals that the self-assessment test scores are not robustto the order of the options. These simple tests, done on ChatGPT and threeLlama2 models of different sizes, show that self-assessment personality testscreated for humans are unreliable measures of personality in LLMs.

Self-Assessment Tests are Unreliable Measures of LLM Personality