SQLora:Llama-3.2-3B Text-to-SQL Fine-Tune
QLoRA fine-tune of Llama-3.2-3B on 78K+ Text-to-SQL samples with end-to-end training, execution accuracy evaluation via Modal/SQLite, vLLM inference server, and automated HuggingFace Hub deployment pipeline.
Core Impact
“95.9% Execution Accuracy and 81.3% perplexity drop (25.84→4.82) on Text-to-SQL generation by fine-tuning only 0.67% of Llama-3.2-3B parameters via QLoRA.”
95.9%
Execution accuracy on held-out SQL
81.3%
Perplexity drop (25.84 to 4.82)
0.67%
Of model parameters trained
The Problem
Text-to-SQL models usually get judged on string similarity, which happily rewards a query that looks correct but returns the wrong rows. I wanted to fine-tune a small model that produces queries that actually run and return the right data, and I wanted to measure it the honest way, by executing the SQL.
The Approach
Why execution accuracy over ROUGE
String metrics miss logic errors, so I built an eval that runs both the generated SQL and the reference SQL against a real SQLite database and compares the result sets. That's how I can say it's 95.9% correct instead of 95.9% similar, which are very different claims.
Why QLoRA on a 3B model
I used 4-bit NF4 with LoRA adapters on all seven projections, around 20M trainable parameters. Small enough to fine-tune on a single GPU, and it dropped perplexity 81.3% across 78k Text-to-SQL examples.
Why Modal and vLLM for serving
I deployed the adapter behind vLLM on Modal for fast inference, then ran the execution eval against that live endpoint. The accuracy number comes from the real serving path, not a one-off local run that nobody else could reproduce.
Why automate Hub publishing
The publish script writes the real eval metrics into the model card before upload, so what's posted on HuggingFace always matches what was actually measured. Same discipline I hold across all my fine-tuning projects.
Technical Deep Dive
Fine-tuned Llama-3.2-3B on 78K+ Text-to-SQL samples (b-mc2/sql-create-context) via QLoRA (4-bit NF4 + double quantization, LoRA r=16 α=32 across all 7 attention/MLP projections). Perplexity dropped 81.3% (25.84→4.82) while training only 0.67% of model weights (~20M of 3B params).
Achieved 95.9% Execution Accuracy (47/49) on held-out samples evaluated via Modal inference. It runs model-generated SQL and reference SQL against in-memory SQLite populated with synthetic data, comparing result sets to catch logic errors string-similarity metrics miss.
ROUGE-L improved 0.259→0.353 (+36.3%) on 200-sample held-out test; built full inference stack with CLI REPL, base model comparison mode, and remote adapter loading from HuggingFace Hub.
Deployed production inference server on Modal using vLLM for optimized throughput; built Next.js frontend for live query generation demos.
Automated Hub publishing pipeline that injects real eval metrics into model card README before upload, ensuring published numbers always match actual results.
Systems Analysis Concluded