News for hackers. Stuff that matters.
Re:How we test at Nubank [video] by jonahbenton on Wednesday March 11, 2020 @03:11PM (#22549293)
I was at the talk. From having engaged with a lot of organizations working at scale, it was pretty clear they were near the top in terms of not being hampered by technical debt. They have the capability to move quickly and evolve the system in pragmatic ways to dial the knobs towards correctness or speed or functionality or whatever. A tremendous achievement and speaks volumes for the tooling.

There is no "Datomic scale" problem. Datomic transactors are just another singleton service to deploy with your microservice pod. The underlying storage can obviously be consolidated and scaled independently. I don't recall what they're using, would guess it is pg.

Re:How we test at Nubank [video] by Scarbutt on Wednesday March 11, 2020 @03:38PM (#22549640)
Multiple datomic transactors can't talk to the same database, even if they share the same storage.
Re:How we test at Nubank [video] by jonahbenton on Wednesday March 11, 2020 @04:01PM (#22549872)
Sure they can. Just give them different database names.
Re:How we test at Nubank [video] by Scarbutt on Wednesday March 11, 2020 @04:09PM (#22549930)
A different name is literally a different database.
Re:How we test at Nubank [video] by jonahbenton on Wednesday March 11, 2020 @05:12PM (#22550715)
A single postgres daemon- a unit of scaling- can support many datomic transactors, each talking to a named "database" hosted by that one daemon.

You don't have to scale your postgres daemons with your microservices- each of which have their own transactor- which would be painful and out of the ordinary for an ops team.

Scaling datomic on top of postgres is no different from scaling any other microservice.

That all said, the architectural point that did sound painful in the presentation- and which is a common pain point in microservice architectures, not unique to what NuBank is doing with Datomic- was having to maintain an ETL for analytics purposes, to pull together all of the distinct microservice-specific Datomic-hosted data sets into a single uniformly SQL-queryable data set. The details of that implementation, and whether it made use of the new SQL interface supported by Datomic, were not discussed. But it smelled brittle and fragile.

Re:How we test at Nubank [video] by Scarbutt on Wednesday March 11, 2020 @06:25PM (#22551478)
A single postgres daemon- a unit of scaling- can support many datomic transactors

yes

,each talking to a named "database" hosted by that one daemon.

No. That would imply distributed writes and break the single write serializability property of datomic. Think about it, transactors don't sync with each other(only one for HA but that's orthogonal).

To put it simply, multiple transactors can share the same storage but only one can write to a single database at a time.

Re:How we test at Nubank [video] by dustingetz on Wednesday March 11, 2020 @04:19PM (#22550034)
datomic peers can interleave N databases at query time, which is what you want right?
Re:How we test at Nubank [video] by Scarbutt on Wednesday March 11, 2020 @04:21PM (#22550054)
To a single transactor only.
Re:How we test at Nubank [video] by dustingetz on Wednesday March 11, 2020 @05:27PM (#22550903)
I don't think that's right
Re:How we test at Nubank [video] by Scarbutt on Wednesday March 11, 2020 @06:26PM (#22551487)
Are you implying that a single peer can be configured to use multiple transactors at the same time?
Re:How we test at Nubank [video] by dustingetz on Wednesday March 11, 2020 @06:37PM (#22551581)
yes i believe they simply compete for object cache
Re:How we test at Nubank [video] by Scarbutt on Wednesday March 11, 2020 @07:24PM (#22551886)
Ok, I never tried but I don't see any reason why it wouldn't work but also don't know how the peer library handles multiple connection objects. Anyway this is a different issue from having multiple transactors configured to write to the same logical database.
Re:How we test at Nubank [video] by dustingetz on Wednesday March 11, 2020 @10:25PM (#22553005)
it is interesting to consider because it enables a constellation architecture, where writes and reads are sharded independently. you could imagine a distributed computation like a social network where each person owns their own transactor but you can still query across the parts of the social graph you care about.