Show HN: Datasette Publish – Turn CSVs into a SQLite-Backed JSON API

Re:Show HN: Datasette Publish – Turn CSVs into a SQLite-Back... by simonw on Wednesday January 17, 2018 @05:55PM (#16172867)

Nope, it's much more simple than that. Each CSV upload is converted into a SQLite database. Then I deploy a static, read-only copy of that database to a Zeit Now hosting account and fire up my "datasette" command-line tool (written in Python 3) which opens the SQLite database and starts serving up queries from it.

https://github.com/simonw/datasette

The magic here is that the SQLite database is opened in read-only mode, which means I don't have to worry about concurrent access or write activity. SQLite is screamingly fast if you use it in this context.

Because the data is read-only, if you ever DO need to scale to handle more traffic you can do so by firing up additional instances, each with their own copy of the SQLite database file.

[ Parent | on HN ]

Re:Show HN: Datasette Publish – Turn CSVs into a SQLite-Back... by brandonlipman on Thursday January 18, 2018 @06:24PM (#16182589)

This is awesome. Could this be extended so that new CSV uploads can be added at a later date and deduplicate?

[ Parent | on HN ]

Re:Show HN: Datasette Publish – Turn CSVs into a SQLite-Back... by simonw on Thursday January 18, 2018 @10:05PM (#16183849)

Zeit Now deployments are immutable, but I can have an "edit" feature which retrieves the data used for a deployment, lets you modify it (add new CSV files for example) and then deploys a brand new instance with the new changes. Zeit supports URL aliases (which I need to expose in Datasette Publish) so you can point an alias at the fresh deployment.

Stuff like CSV de-deplication is out of scope for Datasette Publish - it's something you would need to do with other tools (e.g. Excel) before uploading the data.

[ Parent | on HN ]