News for hackers. Stuff that matters.
Re:Support for indexes that use deterministic expressions by ThePhysicist on Saturday September 05, 2015 @10:47AM (#10174922)
SQLite gets better and better all the time. Considering that hard disks and memories get faster and cheaper as well SQLite is even becoming a serious option for handling "big data".

I use it in several of my personal and professional projects, especially in cases where data can be sharded well. The one thing that is a bit annoying though is the locking mechanism in SQLite, which prevents simultaneous writes to the database and locks it for reading while a write transaction is underway. If they could find a way to improve that behavior I think the use cases of SQLite would grow way beyond embedded application databases. Whether or not this is a good thing is of course a different question, but having a zero-configuration "plug-and-play" database engine could be a very attractive proposition to many people.

BerkeleyDB for example, which handles concurrent access much better than SQLite and also comes in a high-availability version is used in many production environments today, most notably as the backend of Amazon's DynamoDB key/value store.

Re:Support for indexes that use deterministic expressions by stevoski on Saturday September 05, 2015 @11:04AM (#10174972)
If SQLite can handle your data handling needs, then almost by definition you are not doing "big data".

Quoting my friend Wikipedia: "Big data is a broad term for data sets so large or complex that traditional data processing applications are inadequate."

But yes, I agree that SQLite gets better all the time.

Re:Support for indexes that use deterministic expressions by seppo0010 on Saturday September 05, 2015 @10:58AM (#10174946)
Last time I check, multi-process byte-range file locking was a mess. I'm surprised by your statement about BerkeleyDB, because as far as I can tell it simply cannot be safely implemented.

This blog post (2010) explains what the status is http://0pointer.de/blog/projects/locking.html

If something has changed since then (or that article is incorrect), please let me know.

Re:Support for indexes that use deterministic expressions by asutherland on Saturday September 05, 2015 @12:27PM (#10175240)
SQLite addressed writers blocking readers in 3.7.0 with its Write-Ahead Log. See https://www.sqlite.org/wal.html for more details, but point 2 at the top is "WAL provides more concurrency as readers do not block writers and a writer does not block readers. Reading and writing can proceed concurrently."

(Writes will still block writes of course.)

Re:Support for indexes that use deterministic expressions by coleifer on Sunday September 06, 2015 @12:21PM (#10178060)
Berkeleydb comes with a Sqlite api-compatible library. Using this, you effectively get sqlite with page level locking as opposed to database. Instructions here for compiling:

http://charlesleifer.com/blog/building-the-python-sqlite-dri...