Show HN: Work with CSV files using SQL. For data scientists and engineers

Re:Show HN: Work with CSV files using SQL. For data scientis... by tanin on Sunday July 18, 2021 @01:43PM (#27874815)

Hi! I'm the OP here.

The performance gain is mainly from reading/writing files in C and/or by Sqlite (which is also C).

Any other tool that use dynamic languages like python will immediately takes minutes to handle any GB-sized file (initializing string is already slow).

The main difference is essentially GUI vs command-line. Datasette is the closest but it seems to require command line (based on the usage page). I haven't tried it out yet, so I may be wrong here.

OctoSQL looks very interesting.

Initially, I was trying to use DuckDB but couldn't make it work on Windows.

Sqlite's SQL dialect is somewhat lacking (hello WITH RECURSIVE). It is one of the gaps I would like to solve.

[ Parent | on HN ]

Re:Show HN: Work with CSV files using SQL. For data scientis... by legg0myegg0 on Sunday July 18, 2021 @09:42PM (#27878396)

Odd that DuckDB didn't work for you on Windows! I only use it on Windows and love it!

[ Parent | on HN ]

Re:Show HN: Work with CSV files using SQL. For data scientis... by tanin on Sunday July 18, 2021 @10:08PM (#27878542)

My use case is a bit more complex. I have to compile it with an electron app and etc.

I filed a few GitHub issues with them as well.

This has great potential. The dialect is much more comprehensive than the Sqlite one.

[ Parent | on HN ]