stuarthalloway

Thanks Kyle! It is evident that our docs were insufficient. Rich and I have written what we hope is clearer, more comprehensive documentation about Datomic’s transaction model. We hope that this can preempt common misconceptions and we welcome all feedback!

https://docs.datomic.com/transactions/model.html

When we designed Datomic (circa 2010), we were concerned that many languages had better support for lists than for sets, in particular list literals and no set literals.

Clojure of course had set literals from the beginning...

Formal verification is very powerful but still not full assurance. Fun fact: Testing and monitoring of Datomic has sometimes uncovered design flaws in underlying storages that formal verification missed.

It is worth noting here that Datomic's intra-transaction semantics are not a decision made in isolation, they emerge naturally from the information model.

Everything in a Datomic transaction happens atomically at a single point in time. Datomic transactions are totally ordered, and this ordering is visible via the time t shared by every datom in the transaction. These properties vastly simplify reasoning about time.

With this information model intermediate database states are inexpressible. Intermediate states cannot all have the same t, because they did not happen at the same time. And they cannot have different ts, as they are part the same transaction.

Ed Wible is a founder of Nubank and chose Datomic. He and Lucas Cavalcanti gave a talk on it at Clojure/conj 2014.

https://www.youtube.com/watch?v=7lm3K8zVOdY

What does the following expression return?

(let [x 1] [(inc x) (inc x)])

In Clojure the answer is [2 2]. A beginner might guess [2 2] or [2 3]. Both are reasonable guesses, so a beginner needs to be quite careful!

But that isn't particularly interesting, because beginners always have to be quite careful. When you are learning any technology, you are a beginner once and experienced ever after. Tool design should optimize for the experienced practitioner. Immutability removes an enormous source of complexity from programs, so where it is feasible it is often desirable.

Datomic software needed no changes as a result of Jepsen testing. All functionality in datomic.api is unchanged.

Mea culpa on the docs, mea culpa. Better now [1].

In order for user code to impose invariants over the entire transaction, it must have access to the entire transaction. Entity predicates have such access (they are passed the after db, which includes the pending transaction and all other transactions to boot). Transaction functions are unsuitable, as they have access only to the before db. [2]

Use entity predicates for arbitrary functional validations of the entire transaction.

[1] https://docs.datomic.com/transactions/transactions.html#tran...

[2] https://docs.datomic.com/transactions/transaction-functions....

As a proponent of just such tools I would say also that "enough rope to shoot(?) yourself" is inherent in tools powerful enough to get anything done, and is not a tradeoff encountered only when reaching for high power or low ceremony.

As Jepsen confirmed, Datomic’s mechanisms for enforcing invariants work as designed. What does this mean practically for users? Consider the following transactional pseudo-data:

[

[Stu favorite-number 41]

;; maybe more stuff

[Stu favorite-number 42]

]

An operational reading of this data would be that early in the transaction I liked 41, and that later in the transaction I liked 42. Observers after the end of the transaction would hopefully see only that I liked 42, and we would have to worry about the conditions under which observers might see that 41.

This operational reading of intra-transaction semantics is typical of many databases, but it presumes the existence of multiple time points inside a transaction, which Datomic neither has nor wants — we quite like not worrying about what happened “in the middle of” a transaction. All facts in a transaction take place at the same point in time, so in Datomic this transaction states that I started liking both numbers simultaneously.

If you incorrectly read Datomic transactions as composed of multiple operations, you can of course find all kinds of “invariant anomalies”. Conversely, you can find “invariant anomalies” in SQL by incorrectly imposing Datomic’s model on SQL transactions. Such potential misreadings emphasize the need for good documentation. To that end, we have worked with Jepsen to enhance our documentation [1], tightening up casual language in the hopes of preventing misconceptions. We also added a tech note [2] addressing this particular misconception directly.

[1] https://docs.datomic.com/transactions/transactions.html#tran...

[2] https://docs.datomic.com/tech-notes/comparison-with-updating...

Do you have a specific example? Datomic is great for evolving in prod (https://blog.datomic.com/2017/01/the-ten-rules-of-schema-gro...), and Clojure is a great language for generating data (or ETL) when you want to make radical changes in dev.

All Datomic consulting will continue.

And Fressian https://github.com/Datomic/fressian/wiki

That depends on what queries the peer runs. In both On-Prem and Cloud, Datomic maintains an in-memory LRU object cache. Also, you can send queries with different working sets to different processes without having to shard data on ingest. For example, some peers might handle queries related to user transactions while other peers handle analytic or batch work.

See also https://docs.datomic.com/cloud/operation/caching.html.

Hm, I should have said "value proposition", e.g. SSDs are low latency but can fail, S3 is high latency but "never" fails, etc.

Datomic is designed for data sets larger than memory, and also to do a good job caching working sets that do fit in memory.

Datomic keeps your database in the same memory space as your application code, backed by SSD, backed by EFS, backed by S3/DynamoDB. Each layer of this cache provides a different value, and the whole cache system is automatic and eternally consistent because the database is an immutable value.

You want the illusion that your database is entirely in memory, just as want the illusion that all your memory is in your L1 cache.

You don't have to be very far "in" AWS to try Cloud. The bastion server will let you connect from anywhere, so the idea is that the only infrastructure you need is a laptop and an internet connection.

That said, we understand the value of a local development mode and that is possible in the future.

Cloud uses clients, not peers, so those apps can be much more lightweight. I have tested down to 128M per Clojure (JVM) client, which is an order of magnitude less mem than a typical peer.

Datomic Cloud makes it much easier to get started on AWS. It takes only a few minutes to sign up, see https://www.datomic.com/videos.html for a walkthrough.

Cloud is definitely the preferred way to get started if you are on AWS. When running the On-Prem version you also have to manage (and pay for) storage. Cloud also puts less load on DynamoDB, which can dominate your spend.

That should fit easily in the Solo Topology at about $1/day.

Alex Miller is the person primarily responsible for triaging Clojure tickets. For a thoughtful, balanced assessment of the issues Ashton raises, read Alex's detailed comment on the original post: http://ashtonkemerling.com/blog/2016/06/11/my-increasing-fru...

Hi, I am a Clojure committer and I just closed http://dev.clojure.org/jira/browse/CLJ-1013. Even though this issue is not a bug, explaining why is not entirely trivial. I have already spent 15 minutes on it, and have not yet found the best place to update the docs so that developers wanting to consume Clojure from Java find the information they need.

If you have a chance, could you please take care of updating the docs? Thanks in advance! Let's make Clojure great together.

Or like leaving the house without a suit of plate mail.

We (Cognitect) appreciate the power of negation, and it has always been part of the plan. From a priority perspective, negation doesn't open categoric new possibilities for our customers in the way that excision, or the log API, or a new storage option does. But we will get there.

"The only way to have such a versioned history is ..." is incorrect, and trivially so.

Datomic provides one existence proof of this: Datomic's history data is kept in distinct data structures, so it is "pay as you go" -- querying history is more expensive, because there is more stuff. Querying the present is cheaper.

Datomic queries are datalog, and do not require writing map/reduce jobs.

Just downloading it will give you blisters.

Additional analyzers can be added. The vision is definitely not language-specific, and Rich has already done some design work for a Java analyzer.

It is important to understand that datalog is not a new query language. It is an old one, with a strong pedigree. It has power equivalent to the relational model plus recursion.

SPARQL is cool, and they did a good job making it look a little like datalog.