gitwtfhub

wtf is llm-text2sql-taxonomy?

suyiyun/llm-text2sql-taxonomy — explained in plain English

Analysis updated 2026-05-18

15PythonAudience · researcherComplexity · 4/5Setup · hard

TL;DR

Official code for a research paper that ranks Text-to-SQL AI systems by how autonomous they are and includes a case study fine-tuning models with chain-of-thought reasoning.

Mindmap

mindmap
  root((llm-text2sql-taxonomy))
    What it does
      Ranks text-to-SQL systems
      Autonomy based scale
      Spider benchmark case study
    Tech stack
      Python
      LoRA fine tuning
      Qwen3 and LLaMA models
    Use cases
      Study text-to-SQL research
      Reproduce case study results
      Read companion taxonomy paper
    Audience
      Researchers
      NLP students
      Text-to-SQL builders

Code map

Detail Auto

An interactive map of this repo's files and how they connect — its source is parsed live in your browser. Click Visualize to build it.

filefunction / class

Why would anyone build with this?

REASON 1

Read the paper and code to understand how text-to-SQL systems are classified by autonomy level.

REASON 2

Reproduce the Spider benchmark case study comparing chain-of-thought and non-chain-of-thought fine-tuning.

REASON 3

Use the case_study_spider_cot pipeline as a template for a similar fine-tuning experiment.

REASON 4

Reference the leaderboard aggregation approach when surveying existing text-to-SQL systems.

What's in the stack?

PythonLoRAQwen3-8BLLaMA-3.1-8BDeepSeek V3

How it stacks up

suyiyun/llm-text2sql-taxonomy0pen-sourcer/hearth13127905/deep-learning-based-air-gesture-text-recognition-
Stars151515
LanguagePythonPythonPython
Setup difficultyhardmoderatemoderate
Complexity4/53/53/5
Audienceresearchervibe coderdeveloper

Figures from each repo's GitHub metadata at analysis time.

How do you spin it up?

Difficulty · hard Time to first run · 1day+

Reproducing the case study requires GPU fine-tuning of Qwen3-8B/LLaMA-3.1-8B and DeepSeek V3 access for CoT generation.

No license information is stated in the README, so terms of use are unclear.

Wtf does this do

This repository is the official code release for a research paper called "Agentic-SQL Revisited: Autonomy-Based Taxonomy and Empirical Benchmark Analysis for LLM Text-to-SQL." It supports academic work on text-to-SQL, meaning systems that turn a plain English question into a database query written in SQL. The paper's core idea is to treat evaluation of these systems as a leaderboard problem. Instead of running every system itself, it collects results that other researchers have already reported and organizes them along a single scale of how much the system acts on its own, running from constrained systems, through those that use examples in context, to iterative systems, then agentic systems, and finally systems where reasoning is built directly into the model. The repository also includes a focused case study built on the Spider benchmark, a widely used dataset for testing text-to-SQL systems, to test this scale with real numbers. The code is organized into four folders. The leaderboard folder is meant to hold a Python tool for gathering and organizing the reported results, marked in the README as a placeholder still to be filled in. The case_study_spider_cot folder contains the full pipeline for the empirical case study: preparing the Spider data, generating step by step reasoning, called chain of thought, with DeepSeek V3, fine tuning smaller models, Qwen3-8B and LLaMA-3.1-8B, using a technique called LoRA, comparing versions with and without the reasoning step, and comparing all of that against DeepSeek V3 and GLM-4 used with three examples. A docs folder holds supporting material such as dataset statistics and a paper outline, and a papers folder has the PDF of this paper plus a companion paper introducing the taxonomy this work builds on. For instructions on running the case study itself, from extracting database schemas to generating reasoning data, building fine tuning data, training, and evaluating results, the README points to a separate README inside the case_study_spider_cot folder. This is a research artifact aimed at people working on natural language to database query systems, not a general purpose tool.

Yoink these prompts

Prompt 1
Explain the autonomy-based taxonomy for text-to-SQL systems described in this repo's paper.
Prompt 2
Walk me through setting up the case_study_spider_cot pipeline on the Spider dataset.
Prompt 3
Summarize how this repo fine-tunes Qwen3-8B with chain-of-thought data using LoRA.
Prompt 4
Compare the CoT and No-CoT ablation results described in this repository's case study.
Prompt 5
Help me adapt this repo's leaderboard aggregation approach to a different benchmark.

Frequently asked questions

wtf is llm-text2sql-taxonomy?

Official code for a research paper that ranks Text-to-SQL AI systems by how autonomous they are and includes a case study fine-tuning models with chain-of-thought reasoning.

What language is llm-text2sql-taxonomy written in?

Mainly Python. The stack also includes Python, LoRA, Qwen3-8B.

What license does llm-text2sql-taxonomy use?

No license information is stated in the README, so terms of use are unclear.

How hard is llm-text2sql-taxonomy to set up?

Setup difficulty is rated hard, with roughly 1day+ to a first successful run.

Who is llm-text2sql-taxonomy for?

Mainly researcher.

View the repo → Decode another repo

This repo across BitVibe Labs

Don't trust strangers blindly. Verify against the repo.