News for hackers. Stuff that matters.
Re:Many small queries are efficient in SQLite by chrisweekly on Tuesday February 16, 2021 @01:23PM (#26157021)
+1, thanks for this and your previous comment. Have you written up your approach in any more detail? I'd love to reference it.
Re:Many small queries are efficient in SQLite by derekp7 on Tuesday February 16, 2021 @02:03PM (#26157468)
The code is on github.com/derekp7/snebu if you want to take a look. The next task on my list is to write up developer / contributor documentation on how the whole thing is put together along with various tricks (and simplified demonstration code).

But specifically, look in "snebu-main.c" that is where the opendb function is (so you can see the pragma statements), and there is a busy_retry function that gets referenced (all it does is sleep for .1 seconds). I believe that you don't need the busy-retry function, if you use the built-in busy handler, but I'm not really sure and don't want to take a chance and break working code.

For the temp tables, look in snebu-submitfiles.c -- the function at the top handles the DB operations, one towards the bottom handles the non-DB tar file consumption operations, and there is a circular queue in the middle to handle buffering so the data ingestion can keep going while the data is getting flushed (these three run as separate processes). I should learn threads, as there may be more flexibility in that, but not comfortable enough with thread programming yet.

Re:Many small queries are efficient in SQLite by chrisweekly on Wednesday February 17, 2021 @10:23PM (#26175216)
Thanks so much! The phrase "unreasonably helpful" comes to mind. Also, kudos for the extraordinarily high-quality documentation in snebu!