Show HN: YoBulk – Open Source GPT powered CSV importer[Flatfile.com alternative]

Re:Show HN: YoBulk – Open Source GPT powered CSV importer[Fl... by nerdponx on Tuesday February 21, 2023 @10:12AM (#34882029)

This is a really interesting use of AI, and I think this has been a sought-after use case for a while. I recall the wave of "ML APIs" and auto-ML frameworks a few years ago that promised to use an ML model to automatically perform feature engineering, hyperparameter optimization, data cleaning, etc., but never caught on as tools the hands of non-experts.

However I'm surprised that this works in completely automated fashion. Given the fundamentally nondeterministic nature of language models, how do you ensure that the output is correct? Do you have a set of assertions that must become true about the data before the result is returned? How do you prevent the model from being too clever with your assertions, and replacing the data with all 0s or something similarly à la Asimov's Three Laws of Robotics (see eg https://en.wikipedia.org/wiki/Runaround_(story))?

[ Parent | on HN ]

Re:Show HN: YoBulk – Open Source GPT powered CSV importer[Fl... by yosai on Tuesday February 21, 2023 @02:14PM (#34885524)

@nerdponx This is really a great question. We are currently using AI for schema generation as well as column matching. The column matching is done with the Dice's Coefficient in yobulk system. But with chatgpt's column matcher, we are leveraging the model of chatgpt to match the columns. Further, there is a roadmap for auto cleaning by keeping the historical records and bulding a model to sense the data type entered into the csv for the specific organization.We give the user final power to decide which if the GPT output is correct or not.Happy to engage with you on this topic.

[ Parent | on HN ]