Notes on Writing Web Scrapers

Re:Notes on Writing Web Scrapers by muxator on Friday December 03, 2021 @05:45AM (#29428583)

You can't use SQLite from multiple threads if you are in single thread mode [0]. If you are in multi-thread mode you can use multiple connections to the DB from different threads, or if you want to share a connection then you have to serialize over it. Or you can use SERIALIZED mode.

This is if you do not care writing any logic. If you do, there are other possibilities, for example using a queue and a a dedicated thread for handling database access, but I personally do not think there are many advantages to this more complicated approach.

[0] https://www.sqlite.org/threadsafe.html

[ Parent | on HN ]

Re:Notes on Writing Web Scrapers by jmnicolas on Friday December 03, 2021 @09:59AM (#29430729)

Thanks, I wasn't aware of this. I'm not sure if it was old info or I "imagined" it but in any case it's good to know.

I hope it wasn't true in 2019, since I painstakingly wrote a multi-threaded app that was accessing SQLite on only 1 thread!

[ Parent | on HN ]