Enhancing Text-to-SQL Translation for Financial System Design

Text-to-SQL, the task of translating natural language questions into SQLqueries, is part of various business processes. Its automation, which is anemerging challenge, will empower software practitioners to seamlessly interactwith relational databases using natural language, thereby bridging the gapbetween business needs and software capabilities. In this paper, we considerLarge Language Models (LLMs), which have achieved state of the art for variousNLP tasks. Specifically, we benchmark Text-to-SQL performance, the evaluationmethodologies, as well as input optimization (e.g., prompting). In light of theempirical observations that we have made, we propose two novel metrics thatwere designed to adequately measure the similarity between SQL queries.Overall, we share with the community various findings, notably on how to selectthe right LLM on Text-to-SQL tasks. We further demonstrate that a tree-basededit distance constitutes a reliable metric for assessing the similaritybetween generated SQL queries and the oracle for benchmarking Text2SQLapproaches. This metric is important as it relieves researchers from the needto perform computationally expensive experiments such as executing generatedqueries as done in prior works. Our work implements financial domain use casesand, therefore contributes to the advancement of Text2SQL systems and theirpractical adoption in this domain.