News for hackers. Stuff that matters.
Re:Show HN: SQLite-S3-query – Python function to query a SQL... by hk1337 on Saturday August 14, 2021 @08:12PM (#28185079)
Why not just use the SQL abilities already in S3? CSV, JSON, parquet, Athena, glue, query a specific object
Re:Show HN: SQLite-S3-query – Python function to query a SQL... by michalc on Sunday August 15, 2021 @01:43AM (#28186505)
There is certainly overlap with what's available in AWS/S3, so in many situations, if you're architecting a system, some of the things you suggest may be completely reasonable. In fact, in another project I'm involved in, we use S3's S3 Select to query CSVs extensively.

But I think there is a, or could be, a place for this:

- I don't think (but very happy to be corrected) that the things you mention can query SQLite files directly. So, there would have to be both a conversion of data, and then the query language used would not be SQLite's flavour of SQL. So, if you're using, or pro, using SQLite and wanting to keep the moving parts and languages down, then I _think_ there is a good argument for sqlite-s3-query to be used (with the caveat that it's a _very_ early version!)

- If you want, "light" dependency on AWS. There are a number of S3-compatible services that this could work on. Or even, the code could be tweaked to just use HTTP range headers, remove the dependency on versionId, which I think is only feature that makes this quite S3-specific. (Although, this would allow the case of errors/corruption if the database is replaced half way through a query...)

Re:Show HN: SQLite-S3-query – Python function to query a SQL... by psanford on Sunday August 15, 2021 @08:15AM (#28188134)
Athena is great for if you have _a lot_ of data. It isn't great for real time queries.

I use a SQLite database stored in S3 + SQLite's FTS5 for a really cheap full text search implementation.

Re:Show HN: SQLite-S3-query – Python function to query a SQL... by hk1337 on Tuesday August 24, 2021 @09:06PM (#28296970)
Yeah, Athena is great for doing queries across multiple S3 objects. I have also used sqlite recently too, usually either with pandas reading from S3 then dumping it to a sqlite database table. I don't really like pandas but it's useful for creating the table structure for me.
Re:Show HN: SQLite-S3-query – Python function to query a SQL... by gsempe on Tuesday August 17, 2021 @03:42AM (#28207233)
I'm really interested to read articles or how to around this solution. Do you have anything public that could help me?
Re:Show HN: SQLite-S3-query – Python function to query a SQL... by psanford on Tuesday August 17, 2021 @04:07PM (#28214258)
For accessing the database I use sqlite3vfshttp[0] with an s3 presigned url for authentication.

[0]: https://github.com/psanford/sqlite3vfshttp

Re:Show HN: SQLite-S3-query – Python function to query a SQL... by krat0sprakhar on Sunday August 15, 2021 @12:19AM (#28186232)
I'm hearing this for the first time. can you link to some documentation? sounds really interesting
Re:Show HN: SQLite-S3-query – Python function to query a SQL... by indrax on Sunday August 15, 2021 @12:27AM (#28186270)