Hey HN! We just open-sourced – our text to SQL LLM that outperforms OpenAI's gpt-3.5-turbo on out-of-training-set schemas, and matches gpt-4 when trained on a single business's schema.
SQLCoder is a fine-tuned variant of StarCoder, supplemented with a lot of hand-curated of data and slightly novel fine-tuning techniques.
We are also open-sourcing our framework for evaluating whether LLM-generated SQL is correct. SQL is tricky to evaluate. Two very different SQL queries can both be "correct". For the question, "who are the 10 most recent users from Toronto", both of the following are correct in their own ways – so we had to build a new framework algorithm to evaluate query correctness.
Query 1: ```sql
SELECT userid, username
from users
where city='Toronto'
order by created_at DESC LIMIT 10;
```
Query 2: ```sql
SELECT userid, firstname || ' ' || lastname
from users
where city='Toronto'
order by created_at DESC LIMIT 10;
```
The model is small enough to run on a single A100 40GB with weights in 16 bit floats, or on a single high-end consumer GPU (like RTX 3090/4090) with 8bit quantization. We will also release a ggml-based quantized version soon, and you should soon be able to run it on most M1 or M2 Macbooks with 32GB of RAM.
The model weights have a CC BY-SA 4.0 license. You can use and modify the model for any purpose – including commercial use. However, if you modify the weights (for example, by fine-tuning), you must open-source your modified weights under the same license terms.
Our evaluation framework is at https://defog.ai/blog/open-sourcing-sqleval/
Interactive demo: https://defog.ai/sqlcoder-demo/
Would love for you to give it a spin, and let us know what you think!