English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

As large language models (LLMs) gain popularity among speakers of diverselanguages, we believe that it is crucial to benchmark them to better understandmodel behaviors, failures, and limitations in languages beyond English. In thiswork, we evaluate LLM APIs (ChatGPT, GPT-3, and GPT-4) on the Japanese nationalmedical licensing examinations from the past five years, including the currentyear. Our team comprises native Japanese-speaking NLP researchers and apracticing cardiologist based in Japan. Our experiments show that GPT-4outperforms ChatGPT and GPT-3 and passes all six years of the exams,highlighting LLMs' potential in a language that is typologically distant fromEnglish. However, our evaluation also exposes critical limitations of thecurrent LLM APIs. First, LLMs sometimes select prohibited choices that shouldbe strictly avoided in medical practice in Japan, such as suggestingeuthanasia. Further, our analysis shows that the API costs are generally higherand the maximum context size is smaller for Japanese because of the waynon-Latin scripts are currently tokenized in the pipeline. We release ourbenchmark as Igaku QA as well as all model outputs and exam metadata. We hopethat our results and benchmark will spur progress on more diverse applicationsof LLMs. Our benchmark is available at https://github.com/jungokasai/IgakuQA.

Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations