Re:Replacing a SQL analyst with 26 recursive GPT prompts
by arrosenberg on Wednesday January 25, 2023 @05:25PM
(#34525019)
Thats not really fair to the human. The time to learn basic SQL may be longer, but the algorithm will never develop intuition or independence, while many junior analysts will eventually operate effectively on their own.
It’s a neat tool for analysts as a query generator - I would use it in situations where I’m not familiar with the schema, but it would become less useful as I learn.
Re:Replacing a SQL analyst with 26 recursive GPT prompts
by mrbungie on Wednesday January 25, 2023 @05:34PM
(#34525100)
Yeah, maybe I should've said "first-weeks/months" level junior data analyst.
But hell, as an analyst I would've paid a lot for a tool that searched intelligently through giant datawarehouses (or whatever consultants call them now) and at least gave you probable matches.
Now that same thing exists and you can even finetune its "DSL" towards your own organization.
As part of a POC I made, I built a similar bot without recursion for debugging and iterative query building though. It does the following:
- It predicts most probable entities from the question. - Searches AWS Glue Data Catalog for the most probable/useful tables. - It builds an Athena SQL Query from N most useful tables.
It obviously get it catastrophically wrong sometimes, but hell, it was a 3 hour POC. If you can make better indices that map entity->table relationships it should get better at searching tables. Add this kind of recursive/iterative debugging of queries, and you get at least something near a junior-level SQL Analyst.
These kind of bots are analogous to Stable Diffusion, they DO need a good prompter/puppeteer/solution-verifier. Most non-senior Data Analysts also need one anyways.