News for hackers. Stuff that matters.
Re:Nubank acquires Cognitect by andrenth on Thursday July 23, 2020 @10:07AM (#23927493)
Are there any technical posts from Nubank on how they manage such a large Clojure codebase (I guess it must be Clojure's largest user in the industry).

I'm looking for things like how to refactor safely in the absence of static typing, for example. Do they make use of spec?

I'm a big fan of static typing, but when using a dynamic language with immutable values and pattern matching, like Erlang, I don't miss it that much. Maybe with Clojure is the same?

Re:Nubank acquires Cognitect by hcarvalhoalves on Thursday July 23, 2020 @11:43AM (#23928523)
Not sure we have any talks about that specifically, but what I can say is that on the Clojure codebases:

- We rely heavily on testing (unit tests, property-based/generative and integration)

- We adopt a micro-services architecture, so most codebases are small enough that refactoring is easy, or sometimes we just replace the service altogether

- We leverage Schema (https://github.com/plumatic/schema) and/or clojure.spec (https://clojure.org/guides/spec) to annotate functions and data schemas, but it's opt-in

- We have static checking of data schemas across boundaries (e.g. checking data producers did not break consumers over HTTP or Kafka), which is where we found the most value (replaced complicated end-to-end tests)

Re:Nubank acquires Cognitect by stopachka on Thursday July 23, 2020 @02:35PM (#23930636)
Curious, did you really find that microservices helped?

The downsides can be so tremendous -- I wonder if nubank would have done better to services in the low digit range.

(context: my thoughts on it -> https://stopa.io/post/236)

Re:Nubank acquires Cognitect by hcarvalhoalves on Thursday July 23, 2020 @05:32PM (#23932510)
I believe micro-services change the trade-offs, and it's a matter of what you value more.

We certainly have to invest more on infra-structure, monitoring, standardisation, and we have been doing since day one - by now it's pretty robust, to the point our internal tooling allowed us to migrate from raw EC2 to kubernetes (https://www.cncf.io/case-study/nubank/) and it was a transparent change to all services and teams.

On the other hand, failures are more localised and we gain robustness and speed to do certain kind of changes.

Your point about getting domain boundaries wrong is true, but IMO it's a pain orthogonal to micro-services. The solution is the same as if you defined a sub-optimal class interface on a big mono-repo: relentless refactoring. We are often strangling (https://martinfowler.com/bliki/StranglerFigApplication.html) and replacing legacy services.

Re:Nubank acquires Cognitect by hestefisk on Friday July 24, 2020 @09:29AM (#23939208)
Questions: transactions are really important in banking systems. If you have 100s of Microservices how do you manage transactions across them? Or maybe all of your microservices run on top of a monolith database with transaction safety?
Re:Nubank acquires Cognitect by hcarvalhoalves on Friday July 24, 2020 @12:52PM (#23941716)
It's probably a misconception that database-level transactions are the primary way to implement business transactions - many business processes w/ customers or 3rd parties rely on ACKs and/or compensating transactions, escaping the scope of a single database-level transaction.

In practice, if the services are loosely-coupled enough, transactional boundaries will be local in scope and transactions spanning multiple services are unnecessary. Downstream services can be eventually consistent if idempotency is guaranteed and retries/replays are free.

Inside the scope of a single service, when needed, we use Datomic, which provides ACID guarantees for that (https://docs.datomic.com/on-prem/acid.html) and also helps w/ idempotency guarantees by allowing time-travel (https://docs.datomic.com/on-prem/getting-started/see-histori...).

Re:Nubank acquires Cognitect by andrenth on Thursday July 23, 2020 @12:26PM (#23929047)
Thanks! Can you talk a bit more about the static checking of cross-boundaries schemas? Any relation to tools that allow a schema to evolve without breaking compatibility, like protobuf?
Re:Nubank acquires Cognitect by hcarvalhoalves on Thursday July 23, 2020 @04:31PM (#23931907)
It's not based on protobuf. There's some in-house tooling that checks the schemas of producers against consumers against known versions (aka the versions in production), and we run it as part of the CI.

Since services implement standard components/protocols there are standardised hooks to extract these schemas definitions from each HTTP endpoint or Kafka consumer.

Re:Nubank acquires Cognitect by andrenth on Thursday July 23, 2020 @11:14PM (#23935333)
Does this rely on code generators or maybe a schema generator from code? I’m curious about how schema and code are kept in sync.
Re:Nubank acquires Cognitect by hcarvalhoalves on Friday July 24, 2020 @03:19AM (#23936658)
Yes. The schema definition is part of the code/data itself, so you can reference a particular schema by its git rev hash.
Re:Nubank acquires Cognitect by ballpark on Thursday July 23, 2020 @05:15PM (#23932332)
I've watched a number of Clojure centric talks from Nubank employees. Actually, I think it was one particular employee, but I don't remember his name. One was on how they do testing. The others were about Fulcro, which is a very interesting web frontend library/framework.
Re:Nubank acquires Cognitect by timgilbert on Thursday July 30, 2020 @02:35PM (#24001471)
Nubank did a presentation on their approach to testing at the Clojure/NYC meetup: https://www.youtube.com/watch?v=1dW4V3stFX0