News for hackers. Stuff that matters.
Re:Unison Cloud by ildjarn on Wednesday February 07, 2024 @04:19PM (#39294451)
Has anyone used Unison instead of Spark? Seems like a natural fit to me.
Re:Unison Cloud by stewoconnor on Wednesday February 07, 2024 @04:58PM (#39294839)
We recently did a series of blog posts exploring how our remote programming model makes us a good fit for writing distributed map - reduce like programs.

https://www.unison-lang.org/articles/distributed-datasets/

One of the real strengths of individual parts of your program being content addressed, and our ability system that lets us track side effects, we can have a programming model where you only need to talk abstractly about where your data is, and how you'd like to operate on it, and then we have the ability to have our cluster gossip about parts of your program that need to be shipped to where parts of your data is. One node can ask another node "please apply this function to the data you have" and the other node can gossip to get any missing definitions it needs for that function.

You never have to talk about serialization or network connections or software distribution, we know how to move data for you, move code for you, and in some cases even cash partial program results.

Re:Unison Cloud by rlmark on Wednesday February 07, 2024 @04:49PM (#39294751)
We wrote an article speaking to this use case here: https://www.unison-lang.org/articles/distributed-datasets/ It's a bit of a deep dive into some of the building blocks of the Remote ecosystem - so it talks quite a bit about how you'd implement something like Spark itself in Unison, but you can see how running data aggregations would work on the Cloud.