News for hackers. Stuff that matters.
Re:SirixDB – An Evolutionary, Temporal NoSQL Storage System by refset on Sunday May 05, 2019 @06:34PM (#19835505)
It would be interesting to hear about the use-cases that prompted the creation of Sirix. For example, we were inspired to build Crux[0], another bitemporal document database (although we opted for Datalog rather that XQuery), following our experiences of integrating timestamped data from multiple upstream systems, whilst coping with delays and ad-hoc corrections, and also maintaining efficient time-travel auditability.

[0] https://juxt.pro/crux

Re:SirixDB – An Evolutionary, Temporal NoSQL Storage System by lichtenberger on Sunday May 05, 2019 @06:48PM (#19835566)
Some of the use cases can be found here:

https://sirix.io/documentation.html

The main distinctive features are our main document store index, features inherited by ZFS (checksums in parent pointers, the main index, a trie, compression and hopefully soon encryption of page-fragments, always consistent on-disk, log-structured without the need of a WAL...), versioned user-defined indexes, highly concurrent data structures (every transaction has access to one snapshot and we only allow one read/write transaction, parallelization has to be done by the client code) and record-level versioning (also a novel sliding window algorithm -- whereas a slightly other implementation is patented by the founder Marc Kramis).

I might implement pointer swizzling for the upcoming 1.0.0 release, should speedup Sirix considerably :-)

https://sirix.io/features.html

I think XQuery is great for querying JSON data. That said in the future I'd also implement something based on Spark to distribute queries...

Re:SirixDB – An Evolutionary, Temporal NoSQL Storage System by refset on Sunday May 05, 2019 @07:22PM (#19835710)
Cool, thanks for the summary! The user-defined indexes sound particularly intriguing.

The documentation page alludes to payroll, audit and decision support applications -- have you implemented one or more of those already? I have been compiling a list of known uses for bitemporality in the Crux docs which you are more than welcome to borrow from:

https://juxt.pro/crux/docs/bitemp.html#_known_uses

It is great to see all this new enthusiasm for temporal databases now that they are finally viable, decades after all the major research happened :)

Re:SirixDB – An Evolutionary, Temporal NoSQL Storage System by lichtenberger on Sunday May 05, 2019 @07:28PM (#19835736)
Will have to sleep now... but yes, the ideas and the first prototype emerged already in I think 2006 from Marc Kramis (back then it internally was named Idefix, then Treetank... but I think Treetank is pretty strange ;-))
Re:SirixDB – An Evolutionary, Temporal NoSQL Storage System by juskrey on Monday May 06, 2019 @02:56AM (#19837352)
How is Crux different from Datomic?
Re:SirixDB – An Evolutionary, Temporal NoSQL Storage System by slifin on Monday May 06, 2019 @05:38AM (#19837965)
These are the differences I think I know about so far: - You can't lie to Datomic, it's immutable and has some really interesting performance characteristics because of that i.e. TTL = infinity, the downside from a legal pov is no information is really gone only redacted from now, I think the law should be updated so that redacted is enough (if not already, I'm not a lawyer) Crux can delete things not sure about performance ramifications - Datomic stores information as datoms which are single pieces of information which are easy to re-arrange at run time to be any shape of tree you like, Crux is a document store which is a small tree already, not sure what abilities there are to re-arrange the tree at query time

- storing business time in Datomic is inefficient because it has to be added to the transaction which I don't think is added to indexes, this is the primary use case of crux though

- cardinality is defined up front in Datomic which I suspect catches data integrity errors, I haven't seen anything in crux to support this

- crux doesn't support either the pull or entity syntax ( I don't know which one or why that's important, I'm a bit out of my depth now)

Re:SirixDB – An Evolutionary, Temporal NoSQL Storage System by lichtenberger on Monday May 06, 2019 @05:41AM (#19837979)
Wow, great summary, thanks from me, too :)