Re:Any text-to-SQL benchmark should address difficulties of ...
by mmasu on Wednesday July 22, 2026 @08:52PM
(#49015499)
also, the same metric can be used in different ways and for different purposes across a business, so you might need different validation rules and definitions.
Maybe one separate semantics validation layer could help, but costs 2x or possibly Nx if you need to recover and turn a wrong query into a correct one
Re:Any text-to-SQL benchmark should address difficulties of ...
by nchammas on Wednesday July 22, 2026 @11:56PM
(#49016751)
I recently shared a similar rationale for why a custom query language can help users even in the age of LLMs.
https://nchammas.com/writing/custom-query-language-design#wh...
Users can't "own" queries they don't understand. An LLM helping a user avoid annoying syntax errors is very useful; but helping a user write a query they ultimately cannot understand is fraught with the problems you describe.
Re:Any text-to-SQL benchmark should address difficulties of ...
by efromvt on Thursday July 23, 2026 @06:57AM
(#49019578)
This was a very enjoyable read! Constraining the language surface is helpful, but the lost expressiveness can bite unless you’re in a constrained domain - which this seems like it was!
Re:Any text-to-SQL benchmark should address difficulties of ...
by rgavuliak on Thursday July 23, 2026 @07:36AM
(#49019933)
Fair point, but before business people relied on other people to pull it for them. Sure, it was checked by human, but a technical human often lacked the business context and there's been plenty of instances, where the business person knowing their revenue and sales numbers challenged the analysis of a technical person.
Re:Any text-to-SQL benchmark should address difficulties of ...
by efromvt on Thursday July 23, 2026 @08:29AM
(#49020522)
Yeah with self-serve analytics all the rage (for good reason) for a bit, the bar from some places I've worked wouldn't be "does the agent beat a good analyst" it's "does the agent beat the a business person with SQL access who might have an overworked analyst glance over it", which is a much more tractable break-even point - and as you said, the business person might be even better at sanity checking numbers at that point.
That is the crux of the issue. No amount of semantic layers and context will help this until an LLM can read the user’s mind to remove ambiguity in the prompt.
I see most of the benefits of LLMs to be used by analysts who know SQL to work more productivly.