Re:Support for indexes that use deterministic expressions
by stevoski on Saturday September 05, 2015 @11:04AM
(#10174972)
If SQLite can handle your data handling needs, then almost by definition you are not doing "big data".
Quoting my friend Wikipedia: "Big data is a broad term for data sets so large or complex that traditional data processing applications are inadequate."
But yes, I agree that SQLite gets better all the time.
Re:Support for indexes that use deterministic expressions
by seppo0010 on Saturday September 05, 2015 @10:58AM
(#10174946)
Last time I check, multi-process byte-range file locking was a mess. I'm surprised by your statement about BerkeleyDB, because as far as I can tell it simply cannot be safely implemented.
This blog post (2010) explains what the status is http://0pointer.de/blog/projects/locking.html
If something has changed since then (or that article is incorrect), please let me know.
Re:Support for indexes that use deterministic expressions
by asutherland on Saturday September 05, 2015 @12:27PM
(#10175240)
SQLite addressed writers blocking readers in 3.7.0 with its Write-Ahead Log. See https://www.sqlite.org/wal.html for more details, but point 2 at the top is "WAL provides more concurrency as readers do not block writers and a writer does not block readers. Reading and writing can proceed concurrently."
(Writes will still block writes of course.)
Re:Support for indexes that use deterministic expressions
by coleifer on Sunday September 06, 2015 @12:21PM
(#10178060)
Berkeleydb comes with a Sqlite api-compatible library. Using this, you effectively get sqlite with page level locking as opposed to database. Instructions here for compiling:
http://charlesleifer.com/blog/building-the-python-sqlite-dri...
I use it in several of my personal and professional projects, especially in cases where data can be sharded well. The one thing that is a bit annoying though is the locking mechanism in SQLite, which prevents simultaneous writes to the database and locks it for reading while a write transaction is underway. If they could find a way to improve that behavior I think the use cases of SQLite would grow way beyond embedded application databases. Whether or not this is a good thing is of course a different question, but having a zero-configuration "plug-and-play" database engine could be a very attractive proposition to many people.
BerkeleyDB for example, which handles concurrent access much better than SQLite and also comes in a high-availability version is used in many production environments today, most notably as the backend of Amazon's DynamoDB key/value store.