Thanks for making Datasette! I've been using it to build a project to make US ranked choice election results more accessible. It's very intuitive and easy to use. The main difficulty for me is how to get around the limitation that the sqlite files have to be colocated on the same server. These datasets can get pretty large, and I can't host them on Github, and since I can't put the datasette db on an S3 bucket, I've been exploring mounting AWS Elastic File System to the Docker container. Is there a better way?
Google Cloud Run works well with database files up to a couple of GBs. Beyond that the best solution at the moment is to run a VPS - I've had good results with DigitalOcean droplets.
Re:Serverless SQLite
by justsomeuser on Wednesday January 20, 2021 @08:52AM
(#25846408)
Why don't you place them onto an AWS disk and attach it to the VM instance running Datasette?
Yep, that's probably what I'll end up doing.
Hi Picardo!
We are toying around with the idea of launching a Cloud SQLite product targeting this exact use case.
How large are your data sets?
Would you be interested in a quick call to discuss your needs in more detail?
FYI, I'm using Google Cloud Run for this at the moment. Scales to zero, issues the SSL cert and provides the HTTPS endpoint. That's what I'd be benchmarking your service against. Let me know if you actually do something in this space.
hey, thanks. That sounds interesting, but this is a volunteer project. I'm donating my time, so I don't think I can justify a service like that right now. The datasets are around 200MB each, just over the 100MB limit of Github.
Thanks for the details!
We would price such a service to be competitive with the obvious alternative of using a long-running VPS or ec2 instance.
I’m using the WASM SQLite (in the browser) for a BI product [1] I’m working on.
It’s been working extremely well and it runs fully sandboxed in a separate worker thread with zero network traffic after the source data has been transferred across. With computational notebooks running alongside it, you can do some pretty powerful analytics with very low latency and it’s great to be able to do inserts and complex queries entirely client-side.
SQLite is one of the greatest software projects I know.
My https://datasette.io/ project is built around a similar idea to this: the original inspiration for it was Zeit Now (now Vercel) and my realization that SQLite read-only workloads are an amazing fit for serverless providers, since you don't need to pay for an additional database server - you can literally bundle your data with the rest of your application code (the SQLite database is just a binary file).
If you want to serve a SQLite API in an edge-closest-to-the-user manner for much larger database files (100MB+) I've had some success running Datasette on https://fly.io/ - which runs Docker containers in a geo-load-balanced manner. I have a plugin for Datasette that can publish database files directly to Fly: https://docs.datasette.io/en/stable/publish.html#publishing-...