English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Natural Language to SQL systems (NL-to-SQL) have recently shown a significantincrease in accuracy for natural language to SQL query translation. Thisimprovement is due to the emergence of transformer-based language models, andthe popularity of the Spider benchmark - the de-facto standard for evaluatingNL-to-SQL systems. The top NL-to-SQL systems reach accuracies of up to 85\%.However, Spider mainly contains simple databases with few tables, columns, andentries, which does not reflect a realistic setting. Moreover, complexreal-world databases with domain-specific content have little to no trainingdata available in the form of NL/SQL-pairs leading to poor performance ofexisting NL-to-SQL systems.  In this paper, we introduce ScienceBenchmark, a new complex NL-to-SQLbenchmark for three real-world, highly domain-specific databases. For this newbenchmark, SQL experts and domain experts created high-quality NL/SQL-pairs foreach domain. To garner more data, we extended the small amount ofhuman-generated data with synthetic data generated using GPT-3. We show thatour benchmark is highly challenging, as the top performing systems on Spiderachieve a very low performance on our benchmark. Thus, the challenge ismany-fold: creating NL-to-SQL systems for highly complex domains with a smallamount of hand-made training data augmented with synthetic data. To ourknowledge, ScienceBenchmark is the first NL-to-SQL benchmark designed withcomplex real-world scientific databases, containing challenging training andtest data carefully validated by domain experts.

ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems